What You Should Know Before Setting Up Your First CI/CD Pipeline

Characteristics of a minimum viable deployment pipeline for growth-focused startups

You want to set up your first automatic deployment pipeline that builds, tests, and deploys code changes to your target cloud environment. You’ve spent days reading docs and blogs to figure out what your automatic deployment pipeline should comprise. But it all seems really overwhelming. They mention various tools like AWS, Azure, GitHub Actions, Ansible, Jenkins, CircleCI, Terraform, and Kubernetes — the list is endless. And you’re not sure which one is necessary for your initial automatic deployment pipeline.

What does your first continuous delivery pipeline need to do and not do?

What are the necessary components of such a minimum viable pipeline?

Which tools are the right choice for your initial pipeline?

This article focuses on laying down the thought process behind how you should go about answering the above questions so that you can quickly start building your first delivery pipeline.

Assumptions

The post recommends the thought process for building your first CI/CD pipeline assuming the following:

• You want to host your application on the cloud.
• A startup like yours wants to optimize for speed while retaining the flexibility to customize your pipeline as you scale.

A Quick Recap of CI/CD Definitions

Continuous integration is the process of merging changes to the main branch as often as possible. These changes are then validated by creating a build and running automated tests against the build.

Continuous delivery is an extension of continuous integration since it automatically deploys all code changes to testing and/or production environments after the build stage. This means that while your build and testing are automated, the deployment trigger (say a button-click) is manual. But once the deployment is started, there is no need for manual intervention.

Continuous deployment is like continuous delivery, except that the trigger to deployment is also automated.

So,

• Continuous delivery is part of continuous deployment.
• Continuous integration is part of both continuous delivery and continuous deployment.

The rest of the post uses CD to refer to both continuous delivery and continuous deployment. Your thought process and the choice of tools to set them up will be the same.

What Should an Initial CD Pipeline Automate?

The guiding principle while deciding what your initial pipeline (also called a minimum viable pipeline) should be is to solve current problems and leave the theoretical problems for the future. Take small steps, and don’t try to build a fully mature pipeline at the start.

While what your small steps should be is a function of your application use-cases, listed below are the most necessary steps — divided into two stages.

The prerequisite for building any CI/CD pipeline is that developers regularly commit their code to a central repository. For feature branches that remain unmerged to the main branch for a long time, developers should keep it up to date by merging upstream as often as possible.

Stage 1: Solves Continuous Integration

1.     Pull the latest code of the branch from your VCS.

2.     Run unit tests on the branch code to check that the application is not broken because of new commits pushed into the branch.

3.     Trigger a build of the branch code whenever an event, that you configure, occurs.

4.     Use a build tool to run a build of your code on a server.

5.     Create one or more build artifacts of the code as part of the build process.

6.     Store the build artifact(s) in a secure and accessible cloud location

Stage 2: Solves Continuous Delivery

1.     Enable triggering the deployment of the build artifact(s)/application to the target cloud environment (testing/staging/production and so on).

2.     On manually triggering the deployment, it automatically deploys the application to the target environment without any downtime.

3.     Provide an easy way to determine if the deployment succeeded or failed, and give into the details.

Note: Your initial pipeline need not implement continuous deployment.

What Should an Initial CD Pipeline NOT Do?

Since implementing every step of the pipeline costs time, it is worthwhile to do a cost-benefit analysis of every step that you want to automate beyond the ones mentioned in the previous section. In our experience, in the beginning, most applications don’t need a fully automated CD workflow for the following:

• Provisioning and managing resources using infrastructure as code
• Rolling back deployment
• Multi-region or multi-cloud deployment
• Auto-scaling to dynamically add or remove instances
• Managing many testing phases like performance testing, UI testing, and so on

Note: While determining the responsibilities of your minimum viable CD pipeline, recognize that you may need to add one or more steps depending on your application’s use case. For example, a payment processing application might be more sensitive to a bad deployment than an employee-leaves management software. In such a case, the minimum viable CD pipeline for the former should have a step to quickly roll back a bad deployment, while the latter can skip it at first.

What Are the Necessary Components of a Minimum Viable CD Pipeline?

Given what an initial pipeline needs to do, let’s take a look at the necessary components to build such a pipeline.

1.     A version control system (VCS) such as GitHub and GitLab

2.     A public cloud provider to host your application infrastructure, such as AWS, Azure, and GCP

3.     A CI tool: To build and run tests on the application code

4.     A CD tool: To deploy the application code to a target environment

Key Criteria to Choose the Right Tools for Your Initial Pipeline

For every component of your initial CD pipeline, you have a plethora of tools to choose from. We recommend that you choose tools that have the following characteristics:

• Fully managed: A fully managed service manages the resources required to do its job, along with any infrastructure, so you don’t have to invest the time and people to do it. Only if your application requires strict compliance that demands tight control and regulated access to your code or data, should you consider self-hosting while building your initial delivery pipeline.
• Easily extendable: Can be easily modified and extended by using code or integrating with third-party libraries.
• Rich plugin ecosystem: An extensive pool of plugins provides support to accelerate the automation of your pipeline.
• Solves for one or more stages of your pipeline: The more steps that a tool solves, the fewer tools you need to integrate with to build your pipeline. For example, GitHub provides both a version control system and direct integration with its own CI/CD.
• Cost: The pricing structure of the tools should be such that it fits in your budget even after you scale.

Recommended Minimum Viable Pipelines

While building your first delivery pipeline, you have two categories to choose from.

Pipeline as code means that you configure the steps in your deployment pipeline — build, test, and deploy, with code that you store in a repository, such as Git. It enables you to track and manage changes to these configurations in the same way as you manage your application code — using version control and pull requests.

We recommend that you choose one of the three options specified below.

• GitHub for VCS & GitHub Actions for CI/CD
• GitLab for VCS & GitLab Pipeline for CI/CD
• BitBucket for VCS & BitBucket Pipelines for CI/CD

Pros

• They enable you to build, test, and deploy directly from their platform without the need for integrating with any other tool.
• They all offer both:
• Fully-managed services: they provision, manage, and scale your build servers to scale continuously and process multiple builds concurrently so that your builds are not left waiting in a queue.
• Self-hosted runners: you can point your builds to run on a machine that you specify. This can be a server that you host yourself behind the firewall or on a private cloud that you manage.
• Cloud and platform-independent.
• Built-in templates for common tasks.

Cons

• There’s a steep learning curve involved when writing a deployment pipeline. It takes time to learn how to properly write one with the correct syntax and modules.
• These pipelines can get to more than 1000 lines of code, and updating and maintaining them could become a headache.
• Choose BitBucket only if your team also uses other Atlassian tools like Jira and Confluence, as it integrates smoothly with them.

Note: Solutions like AWS CodePipeline and Azure Pipeline are also examples of pipelines as code, and are easy to set up. However, they aren’t customizable because integration with non-AWS or non-Azure tools/libraries is difficult. They also make you completely dependent on the specific cloud provider. When you are setting up your first pipeline it might be wiser to keep your options open about how you’d like to evolve your pipeline and then decide whether you’d like to be completely dependent on a specific cloud provider or not.

Type 2: Release Automation Platforms

Release automation platforms remove the need to write code for creating pipelines. They provide a layer of abstraction over pipeline as code. This abstraction further simplifies the creation and management of your pipeline.

Argonaut is one such tool. You can get your first CI/CD pipeline running in less than five minutes in your own cloud.

How You Should Go About Evolving Your Pipeline

Once you’ve built an initial pipeline, keep track of what you still do manually, and how often. You can evolve your initial pipeline by automating the following:

• Code coverage analysis: Add a code coverage tool in your build step to determine the percentage of code being covered by the tests, and fail your build if it’s below a certain threshold. Assuming that the developers have added useful tests, this ensures code quality.
• Multiple environments: It’s hard to test how your app interacts with other services, queues, and databases, in your local development environment. Use staging environments to test this before deploying to production.
• Security integrations: Use tools like Snyk to monitor your application dependencies for vulnerabilities.
• Quick deployment rollback: Use a deployment strategy that ensures that if your application is unusable right after the deploy, you can simply choose the last version you want to restore it to, and immediately roll back to that version. Since that version was already previously tested, it shouldn’t have to go through the pipeline stages once again. You can choose from various deployment methods like canary rollouts, blue/green deployments, and so on.
• Quick hotfix release: Some production bugs may require you to commit and push the fix by bypassing your pipeline stages like testing. While risky, you need to configure your pipeline such that it supports the deployment of a hotfix in a few seconds.
• GitOps practices: As your software scales, you should adopt GitOpspractices — Git to version control and manage your infrastructure and application configurations.
• Use infrastructure as code tools such as Terraform to manage your infrastructure.
• Use ArgoCD or Flux for Kubernetes, and Serverless Stack for your serverless Lambda applications.

To Conclude

At the early stages of product development, when you’re racing against time to release product updates, you might be tempted to postpone automating your CI/CD pipeline. While shipping new features to your customer is the topmost priority, taking small steps toward building a continuous CI/CD pipeline will help you release features much faster and more reliably.

• Automate one step at a time. Don’t try to build a mature pipeline on day 1.
• Automate the most repetitive tasks before others, like the build step before deployment rollback.
• Choose tools that solve your current need, are quick to get started with, and don’t lock you in so that you can easily modify your pipeline as you scale.

We Provide consulting, implementation, and management services on DevOps, DevSecOps, Cloud, Automated Ops, Microservices, Infrastructure, and Security

Services offered by us: https://www.zippyops.com/services

Our Products: https://www.zippyops.com/products

Our Solutions: https://www.zippyops.com/solutions

For Demo, videos check out YouTube Playlist: https://www.youtube.com/watch?v=4FYvPooN_Tg&list=PLCJ3JpanNyCfXlHahZhYgJH9-rV6ouPro

If this seems interesting, please email us at [email protected] for a call.

Relevant blogs:

The Database CI/CD Best Practice With GitHub 

7 Great Terminal/CLI Tools Not Everyone Knows 

Chef InSpec: Where Compliance and Security Blend 

Is NoOps the End of DevOps?