Open in app

Sign in

Write

Sign in

Cloud Native Compliance as Code

Evaluating your GCP resource realtime

How to build a service to validate GCP resource from CAI Feed

Edi Wiraya
Google Cloud - Community
Published in
7 min readMay 3, 2023

--

There are many ways to enforce cloud infrastructure compliance, Preventive Control targets Infrastructure as Code (IaC) CI/CD pipeline before it gets applied, this is good to stop upfront misconfigurations. Open Policy Agent Evaluate Infrastructure Score is a good example.

Detective Control validates the resource after the changes are made. Why do we need this? Because there are situations when manual or external changes remain undetected if we only scan the IaC.

This article focuses on building Resource Validation service, a realtime Detective Control validator using Go library Config Validator on Cloud Run.

Cloud Asset Inventory Feed (CAI Feed) is used to detect resource changes and send them to Resource Validation service via Pub/Sub. The following diagram shows the high level flow:

If you just need to evaluate your existing GCP resources once, no continuous detection is required, consider a batch detective control approach from this article Evaluating your existing GCP resources.

Detecting resource change

CAI Feed is an asset monitoring with feature to notify GCP resource change to Pub/Sub.

Before configuring CAI Feed, create a Pub/Sub Topic first, you can create its Push Subscription later when you have the Resource Validation service on Cloud Run is ready.

Plan for what type of asset you want to monitor, generally for policy validation you want to monitor all assets. To create feed for all asset types:

gcloud asset feeds create cai_feed_all \
  --project=${PROJECT_ID} \
  --content-type=resource \
  --asset-types=".*.googleapis.com.*" \
  --pubsub-topic="projects/${PROJECT_ID}/topics/${PUBSUB_TOPIC}"

Replace ${…} with your Project ID, and Pub/Sub Topic you created ealier. At this stage, Pub/Sub Topic will receive the information when there is GCP resource change. Let’s create the validation service.

Resource Validation service

The purpose of this service is to evaluate your assets against Policy Library. Reuse your Policy Library if you already implemented it for Preventive Control.

This service is designed to validate asset change detected by CAI Feed, and you can expand it for bulk assets validation but we won’t cover that in this article.

If you just want to test the service without going through the nitty gritty, skip to Deploying to Cloud Run or Testing on Local Machine or simply checking out the code here.

Design overview

Service initialization is started with a typical HTTP server and configured to listen to port 8080 as default Cloud Run serving port. Policy Library which consists of policy constraint, template, and rego files is loaded to memory during this initialization.

When the service is ready, it routes HTTP POST requests receving Pub/Sub Message format from the Push Subscription. Data payload in Pub/Sub Message is base64 encoded, and the structure is CAI Temporal Asset. It contains Asset and PriorAsset, we only need Asset which has the latest resource attributes to be evaluated.

Config Validator is Go library to evaluates the GCP resources against the Policy Library, it accepts Asset as input and Asset Violations as output. Unfortunately the Asset required for Config Validator input, is not the same as CAI Asset, therefore we need to map necessary attributes before we can execute the evaluation.

At the of the process, you can choose to stream asset violations to console or persist it somewhere else eg. database.

Coding — step by step

Let’s deep dive into the code, starting with go.mod in your main directory.

go.mod

module gcp-resource-validator

go 1.20

// Prevent otel dependencies from getting out of sync.
// Cannot be upgraded until k8s.io/component-base uses a more recent version of
// opentelemetry.
replace (
    cloud.google.com/go/asset => cloud.google.com/go/asset v1.11.0
    go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp => go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.20.0
    go.opentelemetry.io/otel => go.opentelemetry.io/otel v0.20.0
    go.opentelemetry.io/otel/metric => go.opentelemetry.io/otel/metric v0.20.0
    go.opentelemetry.io/otel/sdk => go.opentelemetry.io/otel/sdk v0.20.0
    go.opentelemetry.io/otel/trace => go.opentelemetry.io/otel/trace v0.20.0
    go.opentelemetry.io/proto/otlp => go.opentelemetry.io/proto/otlp v0.7.0
)

replace (
    cloud.google.com/go/asset => cloud.google.com/go/asset v1.11.0
)

require (
    cloud.google.com/go/asset v1.12.0
    cloud.google.com/go/pubsub v1.30.0
    github.com/GoogleCloudPlatform/config-validator v0.0.0-20230328162739-ff3a6b2846d9
    github.com/gin-gonic/gin v1.9.0
    google.golang.org/protobuf v1.30.0
)

Note: as of April 2023 Config Validator must use some older opentelemetry and cloud asset dependencies. To overcome this, use ‘replace’ in your go.mod

We have go.mod ready, to download dependencies:

go mod download

I use gin, you can use your choice of http server. First initialize the server to listen to port 8080 and a POST route /validator. Create a new Validator to load Policy Library files. It is hardcoded and will be statically built to the container image, you should load the files from Cloud Storage buckets for production implementation.

main.go

package main

import (
        "log"
        "github.com/gin-gonic/gin"
)

func main() {
        router := gin.Default()

        validator, err := NewValidator()
        if err != nil {
                log.Fatal(err)
        }

        router.POST("/validator")  // No handler yet
        router.Run(":8080")
}

validator.go

package main

import (
        "log"

        "github.com/GoogleCloudPlatform/config-validator/pkg/gcv"
)

type Validator struct {
        gcv *gcv.Validator
}

func NewValidator() (*Validator, error) {
        cv, err := gcv.NewValidator([]string{"./policy-library/policies"}, "./policy-library/lib")
        if err != nil {
                return nil, err
        }
        return &Validator{
                gcv: cv,
        }, nil
}

If you have existing Policy Library files, copy the constraints and templates to ./policy-library/policies and the library files to ./policy-library/lib.

If you start new Policy Library, you can clone Policy Library from github:

git clone https://github.com/GoogleCloudPlatform/policy-library

Policy Library from github has no constraint added, we can create ./policy-library/constraints/gcp_storage_location.yamlto test the service later. This constraint will report violation with high severity if you have Cloud Storage bucket created not in allowlist, in this case asia-southeast1. This is optional step.

gcp_storage_location.yaml

apiVersion: constraints.gatekeeper.sh/v1alpha1
kind: GCPStorageLocationConstraintV1
metadata:
  name: allow_some_storage_location
  annotations:
    description: Checks Cloud Storage bucket locations against allowed or disallowed
      locations.
    bundles.validator.forsetisecurity.org/healthcare-baseline-v1: security
spec:
  severity: high
  match:
    ancestries:
    - "organizations/**"
  parameters:
    mode: "allowlist"
    locations:
    - asia-southeast1
    exemptions: []

The directory structure will look like this:

go.mod
go.sum
main.go
validator.go
policy-library
 +- policies
 |   +- constraints
 |   |   |- gcp_storage_location.yaml
 |   |   |- ...
 |   +- templates 
 |       |- ***.yaml
 |       |- ...
 +- lib
     |- ***.rego
     |- ...

Now we have the service initialization part ready. You should be able to test running the server:

go run gcp-resource-validator

If everything is going well you will notice this at the console output last line:

...
[GIN-debug] Listening and serving HTTP on :8080

Ctrl-C to terminate the server.

Note: as of April 2023 if you use sample Policy Library here, there will be some warning message pertaining deprecated v1alpha1 constraint templates. You can safely ignore this.

Next let’s add the route handler.

main.go

        …
        router.POST("/validator", validator.Handler)
        …

validator.go

...

import (
        "context"
  
        ...
        ...
    

        "cloud.google.com/go/asset/apiv1/assetpb"
        "cloud.google.com/go/pubsub"
        cvassetpb "github.com/GoogleCloudPlatform/config-validator/pkg/api/validator"
        libCvAsset "github.com/GoogleCloudPlatform/config-validator/pkg/asset"
        "github.com/gin-gonic/gin"
        "google.golang.org/protobuf/encoding/protojson"
)

type pubsubMsg struct {
        Message pubsub.Message
}

...
...


func (v *Validator) Handler(c *gin.Context) {
        var msg pubsubMsg
        if err := c.ShouldBindJSON(&msg); err != nil {
                log.Println(err)
                return
        }

        var tprAsset assetpb.TemporalAsset
        protoUm := protojson.UnmarshalOptions{
                AllowPartial:   true,
                DiscardUnknown: true,
        }

        if err := protoUm.Unmarshal([]byte(msg.Message.Data), &tprAsset); err != nil {
                log.Println(err)
                return
        }

        cvAsset := &cvassetpb.Asset{
                Name:         tprAsset.Asset.Name,
                AssetType:    tprAsset.Asset.AssetType,
                AncestryPath: libCvAsset.AncestryPath(tprAsset.Asset.Ancestors),
                Resource:     tprAsset.Asset.Resource,
                IamPolicy:    tprAsset.Asset.IamPolicy,
                OrgPolicy:    tprAsset.Asset.OrgPolicy,
        }

        log.Printf("Validating asset: %s\n", cvAsset.Name)
        violations, err := v.gcv.ReviewAsset(context.Background(), cvAsset)
        if err != nil {
                log.Println(err)
        }

        // process the output result here
        ...
        ... 

}

pubsubMsg wraps Google pubsub.Message, we use this struct to bind to HTTP Post request data, and extract the payload. Unmarshal the decoded payload to assetpb.TemporalAsset, and use its Asset (not PriorAsset) to map to ConfigValidator Asset. We are ready to execute ReviewAsset at this stage, and it will return list of violations as the output.

You can add final step to print the output to console, or insert to database. Complete code is available for your reference on the next section.

Deploying to Cloud Run

Prerequisite:

  • Docker installed on your local machine
  • A repository on Artifact Registry or Container Registry on your GCP project
  • You have necessary access to your GCP project

Clone the code

The code is intended for demonstration only, not for production use.

git clone https://github.com/edyw/gcp-resource-validator.git

Build and deploy

Build the image locally, tag and push to container repository. Replace ${…} with your setup.

docker build --tag gcp-resource-validator .

docker tag gcp-resource-validator ${REGION}-docker.pkg.dev/${PROJECT}/${REPO_NAME}/gcp-resource-validator:latest
docker push ${REGION}-docker.pkg.dev/${PROJECT}/${REPO_NAME}/gcp-resource-validator:latest

Deploy Cloud Run using the image from container repository.

gcloud run deploy gcp-resource-validator \
--image ${REGION}-docker.pkg.dev/${PROJECT}/${REPO_NAME}/gcp-resource-validator \
--region=${REGION} \
--allow-unauthenticated \
--service-account=${CLOUD_RUN_SERVICE_ACCOUNT}

Note: You should enable Cloud Run authentication and restrict ingress in production setup

At the end of the deployment, you will have a Service URL: https://gcp-resource-validator-xxxxxxxxxx-as.a.run.app

Pub/Sub Push Subscription

Go to the Pub/Sub Topic created earlier, and use it to create a Push Subscription with the Cloud Run Service URL + /validator as the Endpoint URL.

End to end test

Trigger a change on your GCP resource to test the end to end flow. If you use the sample constraint gcp_storage_location.yaml, create a Cloud Storage bucket in location other than asia-southeast1 to test the violation output. For simplicity, sample code streams the violations output to console, use Cloud Logging to check the result.

Testing on Local Machine

To test the service on your machine, follow these steps.

Clone the code and download dependencies:

git clone https://github.com/edyw/gcp-resource-validator.git
go mod download

Run the server:

go run gcp-resource-validator

You should see this from the server last line:

...
[GIN-debug] Listening and serving HTTP on :8080

The sample code use this constraint gcp_storage_location.yaml located in directory ./policy-library/policies/constraints/ to evaluate if the location of Cloud Storage bucket is in allowlist (asia-southeast1).

To test this, use TemporalAsset data wrapped as Pub/Sub Message payload. In ./test-asset directory, storage_location_us_msg.json and storage_location_sg_msg.json are json formatted Pub/Sub Message, the TemporalAsset payload is base64 encoded, you can refer to storage_location_us.txt and storage_location_sg.txt for decoded version.

Let’s test storage_location_us_msg.json, open another terminal:

curl -X POST localhost:8080/validator -d @./test-asset/storage_location_us_msg.json

You should this output from 1st terminal:

Validating asset: //storage.googleapis.com/bucket-test-1
Violation 1 (high-GCPStorageLocationConstraintV1.allow_some_storage_location): //storage.googleapis.com/bucket-test-1 is in a disallowed location.

Next test storage_location_sg_msg.json:

curl -X POST localhost:8080/validator -d @./test-asset/storage_location_sg_msg.json

No violation detected, the bucket is created in allowlist region.

Validating asset: //storage.googleapis.com/bucket-test-1
No violation detected
Google Cloud Platform
Config Validator
Asset Inventory
Policy Library
Cloud Compliance

--

--

Edi Wiraya
Google Cloud - Community

Written by Edi Wiraya

44 Followers
Writer for

Strategic Cloud Engineer, Google

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams