Responding to changes in cloud resources with Eventarc and Cloud Run

In this post we explore how to use Eventarc to asynchronously react to cloud events that signal changes in cloud infrastructure. We also walk through an implementation of a basic Cloud Run service that can react to such an event and perform downstream operations. In this example we use audit logging events for Apigee to trigger events in Eventarc. Because the trigger mechanism is based on the generic GCP audit log, this example can easily be applied to any other GCP service that supports audit logging.

What is Eventarc?

When thinking about cloud computing one of the first things that comes to mind is the dynamic nature of the infrastructure and that even complex architectures can be provisioned with ease at the click of a button or arguably better by merging a PR in a GitOps pipeline.

By democratizing the process of creating infrastructure and reducing the provisioning time from days to minutes, we created new requirements for processes and governance. Higher degrees of automation and resilience are mandatory such that you can react appropriately to changes in infrastructure. This is where Eventarc comes into the picture. Eventarc lets you asynchronously deliver events from a range of different event sources and respond to them in a number of serverless event handlers.

Eventarc is able to react to a range of types of events including:

Directly exposed events such as changes in Cloud Storage, Firebase Alerts or Firebase Remote Config

Audit Log entries for a large number of GCP services

Generic Pub/Sub Messages

Partner Sources

Please refer to the full list of supported events here.

These events can be handled by Cloud Run, a Kubernetes Service or a Workflow. As there are many different combinations of event source and event handlers, this demonstration will focus on Audit log events and how to handle them in Cloud Run. The available documentation and open source examples should allow for a straightforward transfer of these concepts to other combinations of the available options.

Defining an Eventarc handler

The basic architecture for this example looks as follows:

Refer to this link for an editable version of the diagram above.

We use Apigee issued audit logs as the source for the events that we want to handle with Eventarc. The events are then added to the Evenarc-managed Pub/Sub toic and handled by a Cloud Run Node.js service.

Cloud Run Event Handler

First we create a sample application of a simple Cloud Run service to react to the audit log events. For this we create a folder called eventarc-handler and a package.json and index.js file in it. We also create a gcloudignore file such that the node modules aren’t sent to cloud build when we later build this as a container for Cloud Run.

code_block[StructValue([(u’code’, u’mkdir eventarc-handler && cd eventarc-handlerrnrncat >package.json <<EOFrn{rn “name”: “eventarc-handler”,rn “description”: “Eventarc Demo”,rn “version”: “1.0.0”,rn “private”: true,rn “main”: “index.js”,rn “scripts”: {rn “start”: “node index.js”rn },rn “engines”: {rn “node”: “>=12.0.0″rn },rn “author”: “danistrebel”,rn “license”: “Apache-2.0”,rn “dependencies”: {rn “express”: “^4.17.1”,rn “@google/events”: “^3.1.0”,rn “cloudevents”: “^4.0.1″rn }rn}rnEOFrnrncat >index.js <<EOFrnconst { HTTP } = require(‘cloudevents’);rnconst express = require(‘express’);rnconst {toLogEntryData} = require(‘@google/events/cloud/audit/v1/LogEntryData’);rnrnconst app = express();rnapp.use(express.json());rnrnapp.post(‘/’, async (req, res) => {rn try {rn const cloudEvent = HTTP.toEvent({ headers: req.headers, body: req.body });rn const logEntryData = toLogEntryData(cloudEvent.data);rn console.info(‘Received EventarcEvent:’, logEntryData.protoPayload.methodName);rn console.info(logEntryData);rnrn res.status(200).send(‘OK’);rn } catch (error) {rn console.error(‘Error in handing Eventarc Event’);rn console.error(error);rn res.status(500).send(‘Internal Error’);rn }rn});rnrnconst port = parseInt(process.env.PORT) || 8080;rnapp.listen(port, () => {rn console.log(“eventarc-handler: listening on port”, port);rn});rnEOFrnrncat >.gcloudignore <<EOFrnnode_modulesrnEOF’), (u’language’, u”), (u’caption’, <wagtail.wagtailcore.rich_text.RichText object at 0x3e9d0b636610>)])]

To enable the required Google APIs and deploy the event handler above as a Cloud Run service we run run the following code within the eventarc-handler folder

code_block[StructValue([(u’code’, u’export PROJECT_ID=my-projectrnexport REGION=europe-west1rnrngcloud services enable run.googleapis.com \rnartifactregistry.googleapis.com \rncloudbuild.googleapis.com –project $PROJECT_IDrnrngcloud run deploy eventarc-handler –region $REGION –project $PROJECT_ID –source . –no-allow-unauthenticated’), (u’language’, u”), (u’caption’, <wagtail.wagtailcore.rich_text.RichText object at 0x3e9ce7e2d190>)])]

Once the service is deployed we can test it with a sample cloud event. For this we have to obtain the Cloud Run service URI and store it in a SERVICE_URI (format: https://eventarc-handler-some-hash.a.run.app) and send an example event payload. This example event has the request and response details removed for readability. An actual cloud event would contain additional information.

code_block[StructValue([(u’code’, u’SERVICE_URI=$(gcloud run services describe eventarc-handler –region $REGION –format=”value(status.url)”) –project $PROJECT_IDrnrnecho $SERVICE_URIrnrncurl -X POST “$SERVICE_URI” \rn-H “Authorization: Bearer $(gcloud auth print-identity-token)” \rn-H ‘Content-Type: application/json’ \rn-H “ce-id: 1” \rn-H “ce-source: //pubsub.googleapis.com/projects/my-apigee-org/topics/my-topic” \rn-H “ce-specversion: 1.0” \rn-H “ce-type: google.cloud.audit.log.v1.written” \rn-H “ce-dataschema: type.googleapis.com/google.logging.v2.LogEntry” \rn-H “ce-subject: apigee.googleapis.com/organizations/my-org/developers/[email protected]” \rn-H “ce-methodname: google.cloud.apigee.v1.SampleResource.SampleMethod” \rn-d ‘{rn “resource”: {rn “labels”: {rn “service”: “apigee.googleapis.com”,rn “project_id”: “my-apigee-org”,rn “method”: “google.cloud.apigee.v1.SampleResource.SampleMethod”rn },rn “type”: “audited_resource”rn },rn “insertId”: “1de11nde7ys3b”,rn “timestamp”: “2022-08-04T14:56:05.729242115Z”,rn “protoPayload”: {rn “authenticationInfo”: {rn “principalEmail”: “[email protected]”rn },rn “authorizationInfo”: [],rn “response”: {},rn “serviceName”: “apigee.googleapis.com”,rn “serviceData”: {},rn “request”: {},rn “requestMetadata”: {},rn “resourceName”: “organizations/my-apigee-org/developers/[email protected]”,rn “methodName”: “google.cloud.apigee.v1.SampleResource.SampleMethod”rn },rn “receiveTimestamp”: “2022-08-04T14:56:05.729242115Z”,rn “logName”: “projects/my-apigee-org/logs/cloudaudit.googleapis.com%2Factivity”rn}”), (u’language’, u”), (u’caption’, <wagtail.wagtailcore.rich_text.RichText object at 0x3e9d0aa4eb90>)])]

We can check the log statement from the cloud event in the Cloud Run logs from either the cloud logging UI for with the following gcloud command:

code_block[StructValue([(u’code’, u’gcloud logging read “resource.type=cloud_run_revision AND resource.labels.service_name=eventarc-handler” –project $PROJECT_ID –limit 25 –format “value(textPayload)”‘), (u’language’, u”), (u’caption’, <wagtail.wagtailcore.rich_text.RichText object at 0x3e9ce5e03b50>)])]

Which should contain the following log entry:

code_block[StructValue([(u’code’, u’Received EventarcEvent: google.cloud.apigee.v1.SampleResource.SampleMethod’), (u’language’, u”), (u’caption’, <wagtail.wagtailcore.rich_text.RichText object at 0x3e9d0ad850d0>)])]

Now that we have the event handler ready we can move on to the events that we would like to capture.

Audit Logs Event Source

Eventarc is able to react to events that are written to the Cloud Audit Log. In this example we want to react to a specific method of the Apigee Management API so we need to enable Audit Logging for the Apigee service. This can be done either in through the API as described here or in the GCP UI:

To validate that the audit logging is working as intended we can open the Apigee UI and open a proxy. We then run the following command to validate that there are audit logs created for this action:

code_block[StructValue([(u’code’, u’gcloud logging read “logName : projects/$PROJECT_ID/logs/cloudaudit.googleapis.com AND protoPayload.serviceName: apigee.googleapis.com” –project=$PROJECT_ID –limit 25 –format “value(protoPayload.methodName)”‘), (u’language’, u”), (u’caption’, <wagtail.wagtailcore.rich_text.RichText object at 0x3e9d0b42afd0>)])]

Eventarc to route the audit logging events to the eventhandler

Now that we have both the event source and the event handler configured, we can create the eventarc to combine the two components into an asynchronous event delivery pipeline.

For this we need to enable the eventarc API through either the GCP Console or the following gcloud command:

code_block[StructValue([(u’code’, u’gcloud services enable eventarc.googleapis.com –project $PROJECT_ID’), (u’language’, u”), (u’caption’, <wagtail.wagtailcore.rich_text.RichText object at 0x3e9d0b42a2d0>)])]

Once the service is enabled, we can create a new eventarc trigger. In the cloud console we create a new trigger with the following information (and follow the suggested role assignments for the service accounts for pub/sub and the compute):

Trigger name: apigee-developer-app
Event Provider: Apigee
Event: google.cloud.apigee.v1.DeveloperApps.CreateDeveloperApp
Resource: Any resource
Region: global (because the Apigee control plane is a global service)
Service Account: Compute Engine default service account
Event Destination: Cloud Run
Cloud Run Service: eventarc-handler
Service URL Path: /

Or if you prefer the gcloud version:

code_block[StructValue([(u’code’, u’PROJECT_NUMBER=$(gcloud projects describe $PROJECT_ID –format=”value(projectNumber)”)rnrngcloud projects add-iam-policy-binding $PROJECT_ID –member=serviceAccount:[email protected] –role=roles/iam.serviceAccountTokenCreatorrnrngcloud projects add-iam-policy-binding $PROJECT_ID –member=serviceAccount:[email protected] –role=roles/eventarc.eventReceiverrnrngcloud projects add-iam-policy-binding $PROJECT_ID –member=serviceAccount:[email protected] –role=roles/run.invokerrnrngcloud eventarc triggers create apigee-developer-app \rn–location=global \rn–service-account=$PROJECT_NUMBER-compute@developer.gserviceaccount.com \rn–destination-run-service=eventarc-handler \rn–destination-run-region=$REGION \rn–destination-run-path=”/” \rn–event-filters=”type=google.cloud.audit.log.v1.written” \rn–event-filters=”serviceName=apigee.googleapis.com” \rn–event-filters=”methodName=google.cloud.apigee.v1.DeveloperApps.CreateDeveloperApp” \rn–project $PROJECT_ID’), (u’language’, u”), (u’caption’, <wagtail.wagtailcore.rich_text.RichText object at 0x3e9ce650ddd0>)])]

We can the look at the the Pub/Sub subscription that was automatically created for the eventarc

code_block[StructValue([(u’code’, u’gcloud pubsub subscriptions list’), (u’language’, u”), (u’caption’, <wagtail.wagtailcore.rich_text.RichText object at 0x3e9ce650db10>)])]

And see that the push endpoint is listed as the Cloud Run application we created and specified before.

End to End Validation

From the Apigee UI for the same Apigee organization we can create the following resources:

Create an App Developer under Publish > Developers

Create an API Product under Publish > Products

Create an APP for the Product and Developer under Publish > Apps

We can check the log statement from the cloud event in the Cloud Run logs from either the cloud logging UI or with the following gcloud command:

Which should contain the following log entry:

code_block[StructValue([(u’code’, u’Received EventarcEvent: google.cloud.apigee.v1.DeveloperApps.CreateDeveloperApp’), (u’language’, u”), (u’caption’, <wagtail.wagtailcore.rich_text.RichText object at 0x3e9ce63f3390>)])]

We can also explore the eventarc executions in the eventarc UI or the corresponding explorer UI:

Apigee-specific use case ideas for Eventarc

Based on the example above the Cloud Run service can now be extended to trigger more useful downstream actions that can be used to asynchronously react to changes in cloud architectures or other resources that are governed by Google APIs.

Within the context of Apigee the following non-exhaustive list can give some ideas on where such an eventarc trigger could be applied.

Approval Flows for Developer Apps
Apigee has the concept of optional approvals for Applications. With the Eventarc architecture described before one could implement a workflow to trigger an approval process for newly registered developer applications that require manual approval by the API product owner. The necessary steps could be kicked off via cloud logging alerts, email, chatbot or by creating an issue in an issue tracking tool. A description of this approach can be found in this community post.

Dynamically register Applications and/or Developers
The Apigee mechanism for managing applications and developers can be easily integrated with third party identity providers (IdP). This allows for instance for the creation of access tokens that can be used for both Apigee and backend services. By using eventarc events on newly created DeveloperApps the Apigee credentials can be synchronized with the external IdP such that the Apigee analytics data semantics can be preserved even if the external access token is used.

Automatically publish and update API products in a Developer Portal or the API Hub
Publishing APIs is an important aspect of an API program. In Apigee the published API artifacts in a developer portal are based on API products but loosely coupled. To keep the published APIs in the developer portal and in the products API in sync we can leverage the eventarc and automatically register, update and delete APIs in the portal. The same mechanism can also be applied for publishing APIs on the Apigee API Hub.

Conclusion

Eventarc is a powerful service to trigger workflows based on changes in infrastructure. The supported trigger events and event handlers make it an ideal tool to implement a range of automation or governance processes and complements the dynamic nature of cloud infrastructure.

To get started, check out our quickstart for a step-by-step guide on creating an Eventarc for a number of different event types and targets.

Cloud BlogRead More

Responding to changes in cloud resources with Eventarc and Cloud Run