Private GCS bucket access through Google Cloud CDN
Google Cloud CDN recently added support for private origin authentication. This feature can now be leveraged to configure access to private GCS buckets. Previously, you had to use signed URLs when a GCS bucket was private. With the private origin authentication feature, no client side changes are needed to access the private GCS bucket.
Please note that with this method, we use internet network endpoint group which requires a backend service instead of a backend bucket. Customers will see additional internet egress charges on cache misses for cacheable content when setting the GCS origin as a backend service, compared to a GCS origin set as a backend bucket. Also, the GCS specialty pages features are not supported via internet network endpoint groups.
Steps for configuring access to private GCS bucket using HMAC keys
- Create a GCS bucket “private-gcs” and ensure it is private while creating the bucket -
2. Upload an image or any file into the bucket for testing the set up at the end. We have the below image -
3. Create a service account that we will use to access the GCS bucket and also associate with the HMAC keys we will create for private access to the GCS bucket -
a. Go to IAM & Admin -> Service Accounts
b. Click on “Create Service Account”
c. Provide a name and click done -
d. Go to the bucket you created in step 1 and configure GCS bucket access for this service account in the permissions section —
i. Click on the permissions tab for the bucket
ii. Click on Grant Access and find the service account you created in the previous step.
iii. Provide Storage Admin access to the service account
4. Create HMAC keys
a. Go to Cloud Storage -> Settings and switch to the “Interoperability” tab -
b. Under “Access Keys for service accounts” , click on “Create a key for another service account”
c. Choose the service account we created in Step 3 and select “Create key”
d. Make a note of the Access key and secret as we will need it later. You won’t be able to access it after closing the window as indicated by the warming message -
5. Create an Internet NEG to point to the GCS endpoint. Note that we will be using the virtual hosted-style Cloud Storage XML API and adding it as an internet NEG. More on this here.
You will be able to access the object using the below format -
BUCKET_NAME.storage.googleapis.com/OBJECT_NAME
a. Go to Compute Engine -> Network endpoint groups
b. Click on “Create Network Endpoint Group”
c. Provide a name for the NEG and choose Network endpoint group type as “Network endpoint group(internet)”
d. We will be using the port as 443 got HTTPS access to the backend.
e. We will use the “Add through” option as “Fully qualified domain name and port”. In our case the FQDN for the GCS storage will be — “private-gcs.storage.googleapis.com” [The name of our bucket is private-gcs, yours might be different if you have a different bucket name]
f. Click “Create”
6. Create the HTTP(S) Load Balancer -
a. Go to Network Services -> Load Balancing. Click on “Start Configuration” for HTTP(S) Load Balancing. Select the Global HTTP(S) Load Balancer (classic) option as we need support for internet NEG backends.
b. Complete the front end configuration. I will be using HTTP in my case, but you can use HTTPS by binding an SSL certificate to your front end. I am reserving a front end IP for my load balancer and using it in my front end configuration -
c. Click on create a backend service. For the backend type choose “Internet network endpoint group” and select the internet NEG we created in the previous step. Choose protocol as HTTPS.
d. When a GCS bucket is marked as private, GCS automatically sets the cache control header to Cache-Control: private, max-age=0. Thus, for Caching mode you can either -
i. Choose “Force cache all” and specify a TTL so that all content served from your GCS bucket is cached.
ii. Choose “Use origin setting” or “Cache static content” but set the cache control header explicitly in GCS. You can use the setmeta command to set the cache control setting and override the default cache control values for a private bucket.
e. Click on Advanced Configuration and under Custom request headers, add a host header and set the value to the GCS hostname -
f. Click through the remaining steps in the load balancer configuration and click on create /done.
7. Update backend service for private origin authentication -
This functionality is only available through the CLI and hence this step will be through the CLI -
a. Get the current backend service configuration by running the following command -
gcloud beta compute backend-services describe my-lb-bs --globalIf you have a different name for your backend service, remember to use that instead of “my-lb-bs”.
b. Copy over the output to a .yaml file. Mine is called cdn-private-origin.yaml. You can instead run a single command as below -
gcloud beta compute backend-services describe my-lb-bs --global > cdn-private-origin.yamlc. Add the origin authentication section. For more details, refer here.
The code snippet we will be using :
securitySettings:
awsV4Authentication:
accessKeyId: [the access key from the HMAC creation step - 4.d.]
accessKey: [the access key secret from the HMAC creation step - 4.d.]
accessKeyVersion: Optional
originRegion: Can be any string as GCS does not require this parameter for signaturesAlso note: Cloud CDN private origin feature does not rely on Secret Manager hence the accessKeyVersion is optional — this is different from the private origin setup in Media CDN.
Below is the snippet we will add-
securitySettings:
awsV4Authentication:
accessKeyId: [the access key you got from the HMAC creation step - 4.d.]
accessKey: [the access key secret from the HMAC creation step - 4.d.]
originRegion: us-east-2Remove the below lines from the yaml that you get from the output of running the command in (7b).
fingerprint: <value>
Id: <value>
d. Now update your backend service using the below command -
gcloud beta compute backend-services import my-lb-bs --source cdn-private-origin.yaml --globalNote: If you need to make changes to the non secret parts of the backend service, you don’t have to re-enter your secret access key(accessKey). Also, when you get the backend service configuration either via a gcloud describe command or the REST GET API, the secret accessKey field is omitted. To protect the security of the key, it isn’t included when you look up the current backend service configuration.
That’s it. You are all done. Now we will test the access.
8. Testing
a. Get the IP of your LB
b. Run the below command and you should get a 200. If you get a 403, something is wrong and we will need to debug.
curl -X GET -I "http://<YOUR_LB_IP>/image.png"
$ curl -X GET -I "http://<YOUR_LB_IP>/image.png"
HTTP/1.1 200 OK
X-Guploader-Uploadid: ADPycdsUZ08qBdKRpMDy-M1DuTYVjBeL310XJ1P0Ho13-ZWAZL0_gxm5v7zINjzieVcGcH4NgYUWg0dYLARi_yWkklEtWPQTBYa2
X-Goog-Generation: 1673999018715990
X-Goog-Metageneration: 1
X-Goog-Stored-Content-Encoding: identity
X-Goog-Stored-Content-Length: 2641
X-Goog-Hash: crc32c=cI/CNg==
X-Goog-Hash: md5=OD2c1sr7TEDhU1atVGidHw==
X-Goog-Storage-Class: STANDARD
Accept-Ranges: bytes
Content-Length: 2641
Server: UploadServer
Alt-Svc: h3=":443"; ma=2592000,h3–29=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
Via: 1.1 google
Date: Wed, 18 Jan 2023 19:43:16 GMT
Last-Modified: Tue, 17 Jan 2023 23:43:38 GMT
ETag: "383d9cd6cafb4c40e15356ad54689d1f"
Content-Type: image/png
Age: 16
Cache-Control: public,max-age=86400
cache-id: SFO-1d5601d9
origin_auth_mode: sigv4Troubleshooting steps
- Issue : 403 Access Denied.
Potential causes :
a. Service account does not have sufficient access to GCS bucket. Follow step 3.d to grant service account proper access to GCS bucket
b. Incorrect HMAC key and secret. Double check step 7.c and ensure the proper HMAC key and secret are used.
2. Issue : 404 Not Found
Error : “<Error><Code>NoSuchBucket</Code><Message>The specified bucket does not exist.</Message></Error>”
Potential cause :
Cloud CDN by default forwards the incoming host header to the backend, unless explicitly set to use a different host header value. Follow step — 6.e and make sure host header with value of [bucket_name].storage.googleapis.com has been added to the custom request header section
3. Issue : 400 Bad Request
Error : <?xml version=’1.0' encoding=’UTF-8'?><Error><Code>MalformedSecurityHeader</Code><Message>Your request has a malformed header.</Message><ParameterName>cookie</ParameterName><Details>Header was included in signedheaders, but not in the request.</Details></Error>
Potential cause :
When Cloud CDN generates the v4 signature, it includes the Cookie header as part of the signedheaders field, but when the request arrives in GCS, the cookies are stripped. This causes the authentication to fail as there is a mismatch in the headers included in the request to GCS and the headers included in signedheaders. Ensure there is no cookie being passed in the client request. If setting a cookie at the client is inevitable, configure the load balancer(Cloud CDN) to set the cookie value to an empty string with Classic HTTP Load Balancer or remove the header entirely with Application Load Balancer (headerAction : requestHeadersToRemove) .
4. Issue : 403 Access Denied
<Error>
<Code>SignatureDoesNotMatch</Code>
<Message>Access denied.</Message>
<Details>The request signature we calculated does not match the
signature you provided. Check your Google secret key and signing
method.</Details>
Potential cause:
Due to a difference in how S3 and GCS handle RFC3986 reserved characters, customers must follow these url encoding rules, which are slightly different from the RFC. This is a known issue and a fix is in progress.
Solution:
- Percent encode all bytes except the unreserved characters: ‘A’-’Z’, ‘a’-’z’, ‘0’-’9', ‘-’, ‘.’, ‘_’, and ‘~’.
- Encode the space ‘ ‘ character as ‘%20’, not ‘+’.
- Do not encode the forward slash ‘/’ character.
- Example: “test-bucket/foo/foo ?#[]@!$&’()*+,;=.txt” should be url encoded as “test-bucket/foo/foo%20%3F%23%5B%5D%40%21%24%26%27%28%29%2A%2B%2C%3B%3D.txt”.
