How to Secure Your CI/CD Pipeline

After the CircleCI breach, it's a good moment for any team relying on CI/CD infrastructure to review their pipeline security, as there are some steps they can take to be proactive.

Recently, we learned of a breach of CircleCI. They strongly suggest that all their customers:

Immediately rotate any and all secrets stored in CircleCI.
Review internal logs for their systems for unauthorized access from December 21, 2022, through January 4, 2023, or the date you rotated the secrets in CircleCI.

The CircleCI team also invalidated all Project API tokens and informed users that they would need to be replaced.

This moment can serve as an inflection point for any team relying on CI/CD infrastructure as a chance to review their pipeline security. While the steps the CircleCI team laid out are solid, they are reactions to an event. There are some steps DevOps teams can take to become more proactive.

Identity Management and Access Control in CI/CD

If you are new to this topic, a continuous integration/continuous delivery, CI/CD pipeline is a way to deliver applications through a consistent automated process. Developers push their code to repository hosts like GitHub, GitLab, Azure DevOps, or BitBucket, which starts the automated processes of building the application, testing it, and finally deploying that code to production. While CI/CD pipelines allow teams to work faster and with fewer chances for manual errors, they also present a new attack surface for malicious actors to exploit.

One of the best ways to ensure attackers cannot exploit your CD/CD pipelines is by locking down who can access which tools and resources. If someone is not explicitly granted access privileges, they should not be able to gain access. This can include passwords, access keys, and other access control mechanisms and should be implemented as granularly as possible.

While it might be convenient to grant access to all the members of the DevOps teams, developers and engineers accessing everything; this is a very bad idea, as it significantly increases the chances that a single breached account will allow attackers full access to all related systems.

We encourage everyone to enforce single sign-on (SSO) or use role-based access controls (RBAC) wherever possible. These can help you tightly scope access to grant access to only the relevant systems to the right people.

Secure Non-Human Access

We have discussed human access, but it is also critical to properly scope and manage access of third-party services and tools. When deploying any element of your CI/CD pipeline, you should know exactly what systems and services will send requests and where those responses will go. If malicious code does find its way into your pipeline, make sure it cannot phone home or communicate with unauthorized users.

When relying on container-based systems, ensure you employ authenticators to verify machine identity. An authenticator verifies a container's attributes and only grants the scoped access level if approved.

One final note on non-human access: You should eliminate any unused assets as soon as possible. This means destroying containers and VMs once they are no longer needed and blocking access to third-party tools if they are not being actively used.

Don't Expose Secrets in Your CI/CD Pipeline

One of the first things attackers look for within any system is a way to gain further access, namely hardcoded credentials, also known as secrets. While it might seem obvious that you should not store your secrets in plaintext, this happens more often than you might expect. Given the complexity of CI/CD pipelines and the rate at which they evolve, it is often the easiest and quickest solution developers turn to when rushing to meet deadlines. It is also easy to overlook how secrets are presented in your pipeline logs, where they can show up if not properly redacted.

Fortunately, most CI/CD providers offer built-in solutions that allow you to store credentials and then programmatically call them when needed. Alternatively, most solutions offer paths to integrate key managers like Hashicorp Vault into your pipelines.

A critical step towards keeping plaintext credentials away from your pipelines is implementing secrets detection as early as possible. The best case is when developers catch hardcoded secrets before they can commit them, which they can do on their local machine with pre-commit hooks or a pre-push hook, in a pre-receive hook, or in a CI environment.

Automate Secrets Rotation

Another important element of secrets management is controlling the lifetime of each credential. The longer a secret remains valid, the more likely it can be found and used by an attacker. One of the biggest reasons many credentials live for as long as they do is the pain caused by manually rotating keys and the fear of disrupting production systems. That is where automatic key rotation saves the day.

Most platform providers like AWS, Google Cloud, and Azure DevOps either offer or provide straightforward ways to script automated key rotation. This can be executed manually or on a fixed schedule. Automating key rotation can help everyone rest easy in an emergency, as it allows the invalidation and replacement process to be tested outside of an incident.

When automating secrets creation, remember to scope those credentials rightly to their task. Always follow the principle of least privilege, limiting access to only the necessary data and systems to complete the job at hand.

Actively Monitor for Suspicious Behavior

Looking for suspicious activity after an event can help you identify what went wrong or who accessed what systems during an attack. But you don't need to wait for something bad to happen to leverage your logs.

When most people think about actively motoring CI/CD, it is with the intention of watching for resource consumption or availability. However, you can also configure alerts for suspicious activity, such as the unexpected addition of a privileged role or attempts to connect to an unauthorized service.

Honeytokens for Intrusion Detection

Another way to protect your CI/CD pipelines is to set traps that intruders can trigger throughout your environment. These are commonly referred to as honeytokens. Honeytokens are credentials that look like real secrets but are not valid against any service, but when used, they will trigger an alert and report on the user's address.

One way to consistently deploy honeytokens is to make them a part of your automated environment builds, leveraging tools like Terraform and our open-source project ggcanary. Canary tokens can be created and deployed in your CI/CD pipelines, as well as code repositories or project management systems.

Your CI/CD Security Is a Shared Responsibility

One of the main things users expect from their service providers is that they will keep their platform secure. The reality is that malicious attackers will continue to penetrate and exploit any and all targets, including CI/CD providers. Ultimately securing your pipelines is a joint effort of all the players involved, including your team.

While it's unfortunate that CircleCI has been breached and so many customers have been affected, we also want to make this a call to action to anyone relying on a CI/CD pipeline to review their security practices.

This article lays out just a few of the ways you can secure your CI/CD pipeline. For more information on CI/CD security, we recommend reviewing the OWASP Top 10 CI/CD Security Risks.

We Provide consulting, implementation, and management services on DevOps, DevSecOps, Cloud, Automated Ops, Microservices, Infrastructure, and Security

Services offered by us: https://www.zippyops.com/services

Our Products: https://www.zippyops.com/products

Our Solutions: https://www.zippyops.com/solutions

For Demo, videos check out YouTube Playlist: https://www.youtube.com/watch?v=4FYvPooN_Tg&list=PLCJ3JpanNyCfXlHahZhYgJH9-rV6ouPro

If this seems interesting, please email us at [email protected] for a call.