DevSecOps Easter Eggs

Being aware of the blind spots in an organization (and people) can go a long way toward defending against the unknowns. But where to start? Find out here.

Security can often feel like a Sisyphean endeavor. Day after day, we roll what feels like a growing pile of rocks up the hill, building resilience from threats, internal weaknesses, resourcing challenges, hostile politics, hubris, laziness, inertia. Sifting through potentially overwhelming issues — as well as too many tools — is a steady discipline cybersecurity professionals must master. Not to mention the accumulated fatigue that can cause you to gloss over the details and daily doings. 

But some of the deepest, most important wins can happen when you cut through the noise, get down to what matters, and consider daily routines with more attention and purpose. It's not a new line item on the budget or a new unicorn hire. Instead, it requires focus and a relatively small investment of time to reconsider and refactor regular routines, and this can pay big dividends. Often, the most critical security holes are opened during regular coding, building, and deploying. 

Perfection is impossible, but putting care and attention into the work is possible. That's where we should aim. Being aware of the blind spots in an organization (and people) can go a long way toward defending against the unknowns. But where to start?

What Is DevSecOps and Why Does It Matter? 

DevSecOps is an evolution from DevOps — the idea of integrating core principles of development and operations teams into each other's practices and toolsets. In that process of incorporating a developer mindset into operations, as well as an operations mindset into development, security must not be forgotten. The joke: "DevOps is when you give the root keys to the developer" is dark humor for security professionals who have seen many interpretations of DevOps agility where security and resilience are forgotten or an afterthought. Truly excellent DevOps means that security is integrated into DevOps from the beginning, not deprioritized in order to conduct more insecure, unstable projects faster. The art is accomplishing this without sacrificing agility and flexibility.

Because there are numerous ways companies can implement DevOps, it often feels like taking a Rorschach test — where the type of implementation is an answer that illustrates how the company views the importance of operations fundamentals relative to development. For example, of the many definitions for DevOps, DevSecOps can be realized as an extension of the core definitions of DevOps. 

What DevSecOps Isn't

Just as importantly, what isn't DevSecOps? DevSecOps is not just deployment (or a secure deployment). A lot goes into producing secure code before and after it gets to production, meaning the code becomes accessible to the customer. Ask yourself: 

1. Is the build itself — code and configurations — secure?

2. Does that change the security of the application?

3. What attack vectors are most common?

4. What does the code depend on to run reliably and securely?

Deployment is a critical gate — one of the most important ones — but it's merely one step in the lifecycle of a project. The whole journey must be looked at, including what is picked up along the way in terms of data and access, and what is potentially lost. 

DevSecOps and Its Easter Eggs

DevSecOps is a value stream of integrating security needs into the tools that engineers use to build and deploy. Within that, there is a wealth of opportunities (or landmines) to do things securely and efficiently. Or its reverse. As it is a fundamental virtue of technology for the tech to be independent of the policy, any tool can be used for good or for ill. It all depends on the practitioner and the care with which it is handled. 

In the following section, we will hunt the five main DevSecOps Easter eggs. Easter eggs in this context refer to a hidden joke or treasure often found in video games which rewards the user as an insider. The reward for the Easter egg hunter is foiling potential security attackers. 

1. Secrets, Passwords, and Credentials: Hidden in Plain Sight

One doesn't need to look far for major public breaches to see why this is important, e.g., hardcoded passwords and tokens in Travis CI or an Uber developer's private GitHub repo containing credentials to critical company apps and data that led to 57 million user and driver account compromises — and major regulatory and legal impacts. So what Easter eggs should we look for when it comes to passwords and credentials? You don't want credentials hiding in the following places, so take this opportunity to fix major vulnerabilities. 

1. Hardcoded passwords and tokens – these must go in a secured password manager or use a standard, hardened authentication/authorization system

2. Files on systems – this consists of ownerships and application access

3. Shared credentials and keys

4. Credentials and keys stored in source control

5. Credentials loaded in app or memory – this includes credentials that may be defined in environmental variables, as well as those in applications that are left running indefinitely and broadly accessible

6. Unchanged default credentials – this can be vendor or institutional

7. Weak credentials – this is vendor as well as core system credentials 

2. How Wide Is Your Circle of Trust?

For critical information or operations like deployments or permissions changes, do you know everybody who can touch or access production to deploy or change key permissions/creds? Do you know every app? (See previous section on passwords and credentials stored in running instances or environments.) Tools developed to create no-friction deployment have made it hard to maintain cross-department visibility over time. 

We have to get rid of that old canard "security by obscurity," especially with regard to this last point. Obscurity is not security — vulnerabilities have lifetime exposure: the attacker who can lie in wait has the luxury of time for reconnaissance. The attacker who knows your weaknesses and habits very well has their pick of what to exploit and when. And it is foolish hubris to assume that outsiders would have a harder time understanding or exploiting your systems than insiders. Outsiders often have the advantage of fresh eyes and sharpened toolkits to exploit common mistakes, even those made by smart people. Don't make it easy for an enemy to gain insight on you. 

3. Your Gold Repo Source(s) Of Truth

Your source control repository should be considered your crown jewels: Not only is your intellectual property there, but your coding patterns, architectural insights, and occasionally (unfortunately) critical data resides there as well. Some key questions to ask regarding how well your repositories are being guarded and maintained include: 

1. How are you securing your software?

2. Do you have known vetted repos?

3. Do you trust these?

4. How are they maintained?

5. What software is allowed to be on there?

6. Do you have a Bill of Materials?

7. Can you guarantee its secure provenance — its history and pathway?

It is a behemoth to be considered, but you get a lot of bang for the buck in securing code repositories and the pathways in and out of there. This gives not only a good DevOps architectural view but also provides key insights on your main leverage point for change, security, deployments, and reliability.

4. Memories: What Does Your App (And System) Remember?

Credentials and other sensitive information — credit card information, private personally identifiable information (PII), customer data, sensitive IP intellectual property (and sometimes literal IP addresses) — are often stored in app or instance memory. How well is this being handled and secured? Ask yourself:

1. When are references cleaned out, access controlled, garbage collected, or restarted?

2. What controls information from being dumped from memory or improperly accessed?

This is a more subtle yet dangerous point precisely because of the more tenuous state of logging or easily accessible forensics for data and operations in memory. Attackers that gain access to the memory of applications and systems can also gain visibility into an arbitrary volume and range of data that can be hard to pinpoint. Limiting or managing disclosures for this upper bound of information could get ugly. Help nip it in the bud by making sure sensitive data is encrypted wherever possible; decryption is handled precisely with layered defenses and strongly defended keys, and access controls and forensic logs are well-instrumented on sensitive systems. 

5. Stormy Weather: Are Your Cloud Hatches Battened Down?

Everything's going to "the cloud." Great. But have you heard that quip saying the cloud is just a name for someone else's computer? There's truth to that. It's basically a leased or rented stack that (hopefully) has been equipped with more flexible scaling tools than the one had originally. What does that mean for companies? Many teams and organizations gain an illusory ease in spinning up and scaling compute power. But do they understand what they're spinning up, how they're constructed, and where they are vulnerable? 

There are some key questions to consider and answer as you secure your cloud stack (and orgs): 

1. Who or what gets to deploy stuff — code, builds, changes, or new instances — into your cloud network?

2. Have the application-to-application access patterns (risks) been vetted?

3. Have instances been secured and hardened? (e.g., unnecessary services shut down, programs stripped off, network rules vetted, logging tuned up, default creds changed, access systems integrated, etc.)

4. Are all administrators of the systems truly administrators?

The evolution of DevSecOps lets engineers stretch their wings to expand into new competency areas across all three pillars: development, security, and operations. Developers build out full-stack capability by constructing applications from start to finish and understanding the full lifecycle; every operations or system administrator must gain programming proficiency with at least one language. And any of those who neglect security will often learn the hard way how to master it. 

Conclusion: What Does This All Mean?

The connecting theme is trust, but not blind nor even zero trust — instead, earned trust. Defense-in-depth is a team sport and a full-stack discipline, all the way from hardware to humans. Take all of that into account when setting your org's priorities. For example, ask: 

• How do you know to trust where your toolchain is secure?
• How do you know to trust what has been hardened against a vulnerability?
• How do you know to trust who is making changes and commits to your systems?

Asking good questions can be as important —or more — as coming up with the answers. It gives the opposing party the room to develop their own pathways and muscle memory to a solution, and it can open up new lines of inquiry that previously had been hidebound in its own assumptions. Asking good questions allows room for new solutions to evolve that benefit — and even improve — the original solutions.

We ZippyOPS, Provide consulting, implementation, and management services on DevOps, DevSecOps, Cloud, Automated Ops, Microservices, Infrastructure, and Security

Services offered by us: https://www.zippyops.com/services

Our Products: https://www.zippyops.com/products

Our Solutions: https://www.zippyops.com/solutions

For Demo, videos check out YouTube Playlist: https://www.youtube.com/watch?v=4FYvPooN_Tg&list=PLCJ3JpanNyCfXlHahZhYgJH9-rV6ouPro

Relevant Blogs:

Practical Guide to SRE: Incident Severity Levels 

A Brief Introduction to SBOM and How to Use It With CI 

What Is Continuous Merge? 

Improve Microservices Security by Applying Zero-Trust Principles