Sunday, April 2, 2023
No menu items!
HomeCloud ComputingGitOps your service orchestrations

GitOps your service orchestrations

GitOps takes DevOps best practices used for application development (such as version control and CI/CD) and applies them to infrastructure automation. In GitOps, the Git repository serves as the source of truth and the CD pipeline is responsible for building, testing, and deploying the application code and the underlying infrastructure.

Nowadays, an application is not just code running on infrastructure that you own and operate. It is usually a set of first-party and third-party microservices working together in an event-driven architecture or with a central service orchestrator such as Workflows.

Service orchestrations, which have their own definition files and deployment cycles, can benefit from a GitOps approach. This blog post describes how to set up a simple Git-driven development, testing, and deployment pipeline for Workflows using Cloud Build

Architecture

Let’s take a look at the overall approach. 

Architechture

In this approach, you have a staging branch where you make changes to a workflow. This triggers a Cloud Build configuration that deploys a test staging workflow and runs some staging tests against it. If all tests pass, Cloud Build deploys the staging workflow. After more manual testing of the staging workflow, you merge changes from the staging branch to the main branch. This triggers the same Cloud Build configuration to deploy a test production workflow, run more production tests, and if all tests pass, deploy the production workflow.

This approach enables you to have an automated and staged rollout of workflow changes with tests along the way to minimize risk. 

Configuration

Setting up such an automated workflow deployment pipeline is straightforward. 

First, you need a workflow definition file that can benefit from such automation. You can use one of your workflow definition files or workflow.yaml, which simply returns Hello World. 

Next, define a Cloud Build configuration file (see cloudbuild.yaml) with all the stages. In this configuration, Cloud Build deploys a test workflow with the branch name and commit hash, runs the workflow and captures the output, deletes the test workflow, and tests the workflow with the supplied test script. If all the tests pass, it deploys the final workflow in the branch.

Tests for the branch are defined in appropriate test-{branchname}.sh files. For example, test-staging.sh runs against the workflows deployed in the staging branch and only checks the workflow execution state. On the other hand, test-main.sh runs against the main branch, checks the workflow execution state, and also checks the output of the execution. You can add more tests as you see fit.

Connect your repository to Cloud Build

Now that you have the basic configuration in place, you connect your (or workflows-demos) repository to Cloud Build before creating triggers. Follow the instructions here.

Create a Cloud Build trigger

You now create a Cloud Build trigger to watch for commits to the main and staging branches. General instructions are here.

Go to the Create Trigger section of Cloud Build in the console and create a trigger with the following properties:

Name: workflows-trigger

Event: Push to a branch

Repository: GoogleCloudPlatform/workflows-demos

Branch: ^main$|^staging$

Included files filter: gitops/workflow.yaml

Configuration type: Cloud build configuration file

Cloud build configuration file location: gitops/cloudbuild.yaml

Add a Substitution variable with key/value: _WORKFLOW_NAME and workflows-gitops

Test the staging workflow

You’re now ready to test the build pipeline with the staging branch. 

Switch to the staging branch:

git checkout staging

Change Hello World in workflow.yaml to Bye World:

code_block[StructValue([(u’code’, u’- init:rn assign:rn- – message: “Hello World”rn+ – message: “Bye World”‘), (u’language’, u”), (u’caption’, <wagtail.wagtailcore.rich_text.RichText object at 0x3e099b360890>)])]

Commit and push the change to the staging branch:

code_block[StructValue([(u’code’, u’git add workflow.yamlrngit commit -m “Update workflow.yaml in staging”rngit push’), (u’language’, u”), (u’caption’, <wagtail.wagtailcore.rich_text.RichText object at 0x3e099ac423d0>)])]

You should see the trigger running:

Build trigger running

After a few seconds, the build (including all its stages) is successful:

Staging build details

And a staging workflow has been deployed:

Workflows staging

Test the production workflow

Once you’re ready to deploy the staging workflow to production, simply merge the staging branch to the main branch.

code_block[StructValue([(u’code’, u’git checkout mainrngit merge stagingrngit push’), (u’language’, u”), (u’caption’, <wagtail.wagtailcore.rich_text.RichText object at 0x3e0998a25b50>)])]

In this case, however, the build fails:

Production build details

This is because the test script for the production workflow in test-main.sh is expecting to see Hello World as output of the workflow.

You need to go back to the staging branch, and change Bye World in workflow.yaml back to Hello World:

code_block[StructValue([(u’code’, u’- init:rn assign:rn- – message: “Bye World”rn+ – message: “Hello World”‘), (u’language’, u”), (u’caption’, <wagtail.wagtailcore.rich_text.RichText object at 0x3e0998a25e50>)])]

Check in your changes to staging, see the build succeed, and merge to main. Finally, you should also see the build succeed and see that a production workflow has been deployed alongside staging:

Workflows production

Next steps

This post covered how to set up a Git-driven development, testing, and deployment pipeline for Workflows using Cloud Build. All the details and sample configuration files are in our workflows-demos/gitops repository. 

Of course, Cloud Build is not the only way to set up such a pipeline. GitHub Actions is another useful tool that can help to set up similar service orchestration pipelines. Feel free to contribute to our repository with GitHub Actions based pipelines and reach out to me on Twitter @meteatamel for any questions or feedback.

Cloud BlogRead More

RELATED ARTICLES

LEAVE A REPLY

Please enter your comment!
Please enter your name here

Most Popular

Recent Comments