As the Public Sector Solutions team at Google Cloud, we have many opportunities to develop solutions that make the most of the cutting-edge Cloud technologies. In many projects, we collaborate with multiple development teams to develop, test, and deploy microservice-based solutions, often to GKE clusters.
In this blog, we will address these challenges of managing such complex development environments. We will also discuss using Skaffold and Kubernetes for repeatable development and the DevOps process for large development teams.
Complexity of Developing with Kubernetes
Microservices have become a common system architecture allowing for higher scalability and flexibility, and Kubernetes has become the go-to option for deploying a complex microservice-based system. One of the benefits of microservices is that each individual service is standalone, self-encapsulated, and less dependent on other components in the system. It’s also easy to spin up a new microservice to meet a new requirement as a Kubernetes service.
Although the benefits of developing on Kubernetes are clear, it’s often difficult to design and implement an efficient and simple development process. This complexity increases greatly when a project involves a number of microservices managed by multiple teams.
Dedicated DevOps efforts and costs
It is critical for a successful project to start with a robust and reliable development process. This setup usually requires full-time DevOps or Platform engineers to design and develop a process to bootstrap the infrastructure, including environment configurations, building, testing, deployment, and CI/CD. These DevOps tasks not only take up a significant portion of the project in its early stage, but also are repetitive across new projects.
Similarly, development teams may not be able to avoid these DevOps tasks. A software engineer usually spends a significant amount of time handling the environment setup rather than feature development, e.g. tuning configurations to troubleshoot a microservice in a remote cluster with a close-to-production scenario.
Repetitive steps for a single deployment
Oftentime, engineers run various tools and commands just to deploy to either a local or a remote copy of the application. These steps in the development lifecycle are repetitive and often an extra tax on developer’s time. For example, to build and run a backend and a frontend microservices at the same time, an engineer would need to run commands like below:
When there’s any code changes, the engineer would need to rerun these commands to redeploy code to the Kubernetes pods to reflect the changes. If there are 10+ microservices to test with, things get more complicated. These steps may also be subject to change for different environments. For example, the steps for deploying microservices to a local minikube setup may be different from deploying to a GKE-based cluster.
This complexity slows down the development progress, and inevitably increases the overall cost.
Solution: Skaffold for all environments
In our solution development projects, we have developed a repeatable infrastructure using Skaffold to develop and run microservice-based solutions in GKE. Skaffold is a command line tool that facilitates continuous development for Kubernetes-native applications. Skaffold handles the workflow for building, pushing, and deploying your application, and provides building blocks for creating CI/CD pipelines. This enables you to focus on iterating on your application locally while Skaffold continuously deploys to your local or remote Kubernetes cluster.
We use an infrastructure template that encapsulates our best practices and most common designs can be easily copied and applied to any new project with the same structure using Skaffold, Kustomize and Kubernetes. In a nutshell, we use Skaffold to unify and simplify our development process with one command for all development and deployment environments.
Largely simplified local development process:In the local development environment, our engineers run Skaffold CLI command to build one or multiple microservices and deploy to either a local minikube setup or a remote cluster in GKE. They don’t need to worry about maintaining and modifying deployment scripts for different environments. All they need to do is to use Skaffold to build and test with desired services. The overall development process has become much more efficient than before, and engineers can focus on feature development rather than DevOps.Streamlined deployment to all environments in GKE: All deployment to various environments including local, staging and production is unified with Skaffold using Profiles to differentiate the deployment configuration. Using Skaffold profiles not only simplifies the CI/CD configurations but also makes troubleshooting a lot easier.
Microservice-based structure with Skaffold and Kustomize
We have designed a generic project structure to streamline development with Skaffold. This structure is organized with microservices and components using Skaffold profiles and Kustomize. The overall file structure of a project looks like below:
The project structure contains a “main Skaffold YAML” that defines the entire project, breaking it down with common components and microservices. In this root level, we define Skaffold profiles for various environments like Dev, Staging, Production or any custom environment. These predefined profiles cover all scenarios from local development to CI/CD deployment to remote cluster.
On the other hand, in each microservice folder we define the same set of profiles and use Kustomize to configure how to build and deploy in each environment. Each microservice can also define its preferred configuration including pods auto-scaling, CPU and memory settings, etc. See the example below.
In addition, Skaffold has a built-in Watch Mode with “File Sync” feature to automatically live-reload changes in the local files and swap them to a remote cluster. This enables engineers to write code and see the changes immediately on a remote cluster.
In general, engineers rely not only on Watch Mode but also Skaffold’s ability to stream logs while testing new code in a remote cluster in GKE. These features enable engineers to effectively live-troubleshoot issues in a remote cluster, as if it were a local cluster like minikube.
All of these development workflow mentioned above can be done with a simple Skaffold command of “skaffold dev”.
Conclusion
Skaffold provides our teams with not only the benefits mentioned above, but also additional perks including debugging with breakpoints. With the solution infrastructure using Skaffold, we are able to effectively manage the development with multiple teams in collaboration, and we can easily expand the teams and projects to any scale using the same structure.
To summarize the achievements with Skaffold:
Development and deployment processes are unified and largely simplified.
Engineers spend very minimal time on DevOps, and more on feature development.
Skaffold enables the teams to quickly build and test with livereload and live-troubleshooting in a remote cluster.
The infrastructure is easy to replicate and expand to other projects.
If you haven’t started using Skaffold, we encourage you to check out the following Skaffold resources to get started:
Getting Started with Skaffold (Youtube Video)
Cloud BlogRead More
