Introducing Eventarc triggers for Workflows

We’re happy to announce that you can now create Eventarc triggers to directly target Workflows destinations. Available as a preview feature, it simplifies event-driven orchestrations by enabling you to route Eventarc events to Workflows without having an intermediary service. 

Integrating Eventarc and Workflows

In a previous post, we talked about how to integrate Eventarc and Workflows. Since there was no direct integration between the two services, we had to deploy an intermediary Cloud Run service to receive events from Eventarc and then use the Workflows API to kick off a Workflows execution. Let’s take a look at a concrete example on how this worked. 

Image processing pipeline (before)

Before the Workflows destination feature was available, I designed an image processing pipeline (code):

In this pipeline, users save images into a Cloud Storage bucket. This triggers a Cloud Storage event to a Cloud Run service via an Eventarc trigger. The Filter service checks if the image is safe and initiates an image processing workflow with the Workflows API. The image processing workflow calls three Cloud Functions: Labeler to extract labels from an image, Resizer to resize the image to 400×400, and Watermarker to add a watermark of the top three labels on the resized image. In the end, the resized and labeled image is saved to the output bucket:

This worked as an event-driven orchestration but it wasn’t ideal:

The Filter service not only filters images but it also kicks off the execution of the workflow. We needed this Cloud Run service because there was no direct integration between Eventarc and Workflows. The code was unnecessarily complex and not robust as a result (for example, there is no error handling or retry if the workflow does not succeed). 

 The Filter service is a Cloud Run service while the other services are Cloud Functions. I love Cloud Run but I very much prefer the simplicity of Cloud Functions and I’d rather not worry about containers at all. In this iteration of the application, Eventarc could only route events to Cloud Run services, so we needed to have this extra service type, instead of relying on a simpler Cloud Function.

Image processing pipeline (after)

When the Workflows destination feature became available, I rewrote the application to take advantage of it (code). The resulting architecture is simpler and more uniform:

There’s no need for a separate Cloud Run service to trigger the workflow. Eventarc now routes Cloud Storage events directly to Workflows.

The Filter service is now a simple Cloud Function with the only one job: filtering incoming images.

The Filter service is now part of the overall orchestration with the same error and retry policies.

In this setup, you simply create an Eventarc trigger with the right Cloud Storage event and the workflow destination:

You can also create the trigger from Google Cloud Console. Notice the new Workflows event destination:

In the workflow, you receive a CloudEvent as a runtime parameter from Eventarc and extract the bucket and file information from the event:

The rest of the workflow tasks make calls to the Cloud Functions with the right inputs and capture outputs. The workflow.yaml file and detailed set up instructions are here.

For documentation and more Eventarc and Workflows integration examples, check out the following resources:

Create a trigger for Workflows

Eventarc (AuditLog-BigQuery) and Workflows

Eventarc (Cloud Storage) and Workflows

Eventarc (Pub/Sub) and Workflows

As always, feel free to reach out to me on Twitter @meteatamel with questions or feedback.