The Art of Protecting Secrets: Eight Essential Concepts for SecOps Practitioners

Secrets management is an art, and mastering it requires understanding security protocols, plus a meticulous and proactive approach to staying ahead of threats.

Secrets, secrets, … and more secrets! You probably know that in an ever-expanding world of digital services, secrets are sprawling faster than ever. As security practitioners, we are expected to manage this ever-growing list of sensitive tokens, keys, and certificates with the same fluidity and with the same security guarantees, no matter the scale of operations.

Who Takes What?

The problem is that secrets management is still a major challenge for organizations, as highlighted by our recent State of Secrets in AppSec study, where 75% of respondents reported having experienced a past leak and less than half (48.1%) were completely confident in their capacity to prevent future leaks.

At the same time, stolen credentials are leveraged in nearly half (49%) of the breaches for initial access to an organization’s systems:

Verizon 2023 DBIR: Credential Leaks Continue To Be A Major Issue

So it’s not difficult to see there is a significant risk of seeing pressure accumulating on the shoulders of security people.

Today, I'd like to share with you some valuable insights on credential management that we gathered from discussions with SecOps practitioners. These insights can help you strike a balance between security and efficiency when approaching secrets.

These “thumb rules” are not operational guidelines, nor will they tell you what tool best fits your particular use case. They serve as reminders of important concepts related to protecting your organization’s secrets.

End of the disclaimer, let’s dive into this!

The Eight Concepts

1. Rotation ≠ Revocation

First and foremost, let us all remember that in case of a leak, rotation does nothing; it's revocation that matters. When a compromise is detected, it is essential first to revoke the credential to invalidate its permissions (and then replace it). The consequence of that is that having a rotation policy in place is never a replacement for a secret leakage incident response.

So always make sure you have a well-tested mechanism for revoking or invalidating credentials that have been compromised.

2. Closed Loops

Like a well-oiled machine, your credential management system should run smoothly and without hiccups. Secrets management should have closed loops (a mechanism that auto-regulates a system to maintain a desired state without needing interaction, like a thermostat). This is essential for you to be able to see a new credential everywhere it needs to be before using it and to see a deactivated credential gone from use before deactivating it (deactivated credentials are a common source of outages!).

And, of course, keep an eye out for any attackers trying to override these controls.

3. Detective Controls and Logging

You need to be able to see when and where a credential is being used or leaked. This is important for operational safety and security. Otherwise, how would you even detect a stolen credential? It's like trying to catch a thief without security cameras. You need to have that extra layer of protection to keep your secrets safe. The good news is that GitGuardian can help you.

4. Expiry Times

Expiry times can be both beneficial and risky when it comes to secrets management. Most secret managers offer the option to set time-based conditions for secrets, but it's important to approach this feature with caution. On the one hand, setting a long expiry time provides attackers with a larger window of opportunity to exploit stolen credentials, defeating the purpose of the expiry time.

On the other hand, setting a short expiry time requires diligent renewal to avoid service account lockouts and potential difficulties in resolving issues. Super-short-lived credentials take a lot of resources and diligence that most organizations simply don't have!

Striking the right balance between security and practicality is a delicate task, but it is crucial to achieve. By using roles and expiry dates wisely, you can prevent service account lockouts and make the resolution process smoother and more efficient.

5. Credentials Storage

This one is a simple one, yet often overlooked: store credentials only where they are needed. It’s all too common to see “single point of failure” centralized secrets stores. It is more secure to distribute them wisely and make sure only the necessary parties have access.

6. Enclaves

The security of your secrets ultimately relies on the security measures implemented by the hosting system. Secure enclaves are considered to provide the highest level of security due to their ability to isolate application code and data from privileged users, as well as encrypt the memory on every server. This is achieved through CPU hardware-level isolation and memory encryption, ensuring that unauthorized access to the secrets is prevented.

This is where the “secret zero,” such as root certificates, should be kept. Just like how you'd keep your valuables in a safe, keep your credentials inaccessible by storing them in places like TPMs, TEEs, or HSMs.

7. Authentication

To make it clear, never let anyone implement their own authentication! There's nothing wrong with admitting when you're out of your depth. Sometimes, it's better to leave things to the experts. Fortunately, there are numerous mature open-source solutions available nowadays that can provide reliable authentication mechanisms.

8. Weak Secrets

Final tip: entropy systems failures and coding errors are common enough that it’s become good advice to check for “all-zeros” credentials and repeated values from time to time. You can do this by comparing the hashes.

Conclusion

It is crucial for organizations to prioritize the protection of their secrets and continuously improve their secrets management practices to mitigate the risks associated with unauthorized access and breaches. We hope that these concepts or pieces of advice will help you enhance the resilience of your systems against secrets leaks. To wrap up, here is a checklist to keep in mind:

Prioritize revocation over rotation.
Implement closed loops
Utilize detective controls and logging.
Carefully consider expiry times and credentials storage.
Leverage secure enclaves
Don’t implement your own authentication.
Regularly check for weak secrets.

Until next time, stay tuned!

We Provide consulting, implementation, and management services on DevOps, DevSecOps, DataOps, Cloud, Automated Ops, Microservices, Infrastructure, and Security

Services offered by us: https://www.zippyops.com/services

Our Products: https://www.zippyops.com/products

Our Solutions: https://www.zippyops.com/solutions

For Demo, videos check out YouTube Playlist: https://www.youtube.com/watch?v=4FYvPooN_Tg&list=PLCJ3JpanNyCfXlHahZhYgJH9-rV6ouPro

If this seems interesting, please email us at [email protected] for a call.