Skip to content
Knowledge base

Secrets management: API keys and passwords out of code and repositories

A developer removes an API key that slipped into the repository, but that does not end the matter. The key is still in the commit history and in the clones and forks made after the push. Secrets management moves keys and passwords into a system that hands them to applications for a short time and records every use.

A developer adds a configuration file with an API key for the payment gateway to the repository. An hour later the developer spots the mistake, removes the key in the next commit and goes back to work. Meanwhile the key is still in the repository history, in clones on the whole team’s laptops, in a fork made by a subcontractor and in the log of a job that had already run in the CI/CD pipeline.

Such a key is a secret, and the discipline that keeps it out of code is secrets management, one of the pillars of cybersecurity in software development. Under it, the application receives the key at startup, the key lives briefly and every use leaves a trace. A leak then means replacing one entry in one system rather than reviewing the history of every repository.

What is secrets management?

Secrets management is the central storage, issuing, auditing and rotation of the credentials used by applications and automation. The OWASP Secrets Management Cheat Sheet counts API keys, database credentials, IAM permissions, SSH keys and certificates as secrets. The same document notes that many organizations keep them hardcoded in source code in plain text, scattered across configuration files and configuration management tools.

OWASP justifies centralization with a simple argument: when several services share the same secret, it is hard to tell where it leaked from. A secrets management system reverses that. Each application gets its own secret, the system knows who fetched it and when, and replacement happens in one place instead of in many configuration files.

In practice, the place where the secret lives changes. Instead of a variable in a .env file, the application authenticates to the secrets system at startup and fetches what it needs. The code contains only the name of the secret, never its value.

How is secrets management different from a password vault?

A password vault protects credentials used by people, while a secrets management system serves credentials used by machines. A password vault controls who may check out an administrator password and when, rotates it and records every checkout. It belongs to the PAM family and is built around a person who requests access to a credential.

An application does not request access through a form. It needs the database password at startup, often many times a day, in a container that will cease to exist within the hour. That is why secrets management systems are built around APIs, integrations with CI/CD pipelines and Kubernetes, and secrets issued on demand. In Delinea’s portfolio the two roles are separate products: Secret Server is a PAM vault for IT and security teams, while DevOps Secrets Vault handles secrets management for DevOps teams.

If a person asks for the secret, you need a password vault. If a process asks for it, you need secrets management.

Why does a secret stay in the repository even after you delete it?

The secret stays because Git keeps the entire history of changes, and deleting a line in a new commit does not alter earlier commits. The risk description CICD-SEC-6 from the OWASP Top 10 CI/CD Security Risks is unambiguous: from the moment of the push, the credential is exposed to anyone with read access to the repository, and after deletion from the branch it still appears in the commit history.

The weakness has its own number in the MITRE catalogue. CWE-798 “Use of Hard-coded Credentials” describes a product that contains hard-coded credentials, such as a password or a cryptographic key. The catalogue distinguishes the inbound variant, where the product checks logins against a built-in password, from the outbound variant, where it connects to another system with a hard-coded password. An API key in an application repository is the outbound variant.

Rewriting history does not close the case. GitHub’s documentation on removing sensitive data points out that commits containing the secret may remain accessible in clones and forks of the repository, in cached views under their SHA hashes and in pull requests that reference them. Nothing can be removed remotely from other people’s clones. A secret that has once reached a repository must therefore be treated as exposed.

Where do secrets leak outside the code?

Secrets leak through every artifact the CI/CD pipeline produces along the way. CICD-SEC-6 names three such places outside the repository. The first is container image layers: a credential needed only to build the image remains in one of the layers, and anyone who downloads the image can read it. The second is job logs, which secrets enter deliberately or by accident and from which they flow on into log management systems.

The third place is time. CICD-SEC-6 points out that credentials spread across the whole engineering environment are known to a large number of employees and contractors, and without rotation the circle of people holding a valid key grows every month. Deploy keys, according to OWASP, often stay untouched for years under the “if it isn’t broken, don’t fix it” rule.

The OWASP Secrets Management Cheat Sheet adds two channels: maintainers of a GitLab or GitHub project can often see or extract secrets from pipeline settings, and a fork of the repository or a copy of a job definition should not copy the secret. The latter has to be checked, not assumed.

Key takeaways:

  • A secret pushed to a repository stays in the commit history, clones and forks even after it is removed from the code.
  • Outside the repository, secrets leak through container image layers, job logs and pipeline settings.
  • A key that nobody rotates becomes known to more people every month.

What does the secret lifecycle look like?

Every secret goes through four stages: creation, rotation, revocation and expiration. That is how OWASP describes it, and each stage has its own rule. A new secret must be cryptographically strong and carry the least privileges its role requires. Rotation is meant to ensure that a stolen credential works only for a short time. A secret that is no longer needed or is suspected of leaking is revoked at once, and expiration is best set at creation.

The rationale for rotation comes from NIST SP 800-57 Part 1 Rev. 5 of May 2020. The publication defines the cryptoperiod as the time span during which a key is authorized for use, and lists among its benefits limiting the exposure when a single key is compromised. The same document states that a compromised key loses its validity regardless of the planned period.

One exception is worth knowing before anyone sets everything to rotate. OWASP excludes user credentials from regular rotation and cites NIST recommendations: a person’s password is changed on suspicion or evidence of compromise. Regular rotation applies to machine secrets, because those can be replaced automatically and without harm to users.

What are dynamic secrets and when should you choose them over rotation?

A dynamic secret is created on demand for a specific consumer and stops working when that consumer finishes its work. OWASP gives the example of an application that requests database credentials at startup and receives new ones, generated for that session. If someone steals them, they will already be invalid after the application restarts.

Rotation and dynamic secrets attack the same problem from opposite ends. Rotation shortens the life of a static secret that is still shared by all instances of the application. A dynamic secret is unique to its consumer, so OWASP highlights an extra benefit: when someone uses it from an IP address other than the consumer’s, the misuse is easy to detect.

The CICD-SEC-6 recommendations set the order: temporary credentials over static ones first, and where static ones are unavoidable, a periodic rotation procedure and detection of stale credentials. A static secret remains where the target system cannot issue credentials on demand, for example with a third-party provider’s API key, and then it has to be rotated automatically.

Where should you keep secrets in a CI/CD pipeline?

OWASP points to three places, and the best is the one where the pipeline does not touch the secret at all. The first is the built-in secrets of the CI/CD tool, for example in GitHub or GitLab, which according to OWASP is not the same as committing the secret to code. The second is a secrets management system that the pipeline connects to with its own credential. The third is a setup in which the pipeline only instructs the orchestrator to start a service with a given account, and the service retrieves the secret itself.

Each of these places has its rules. In the CI/CD tool, OWASP recommends secrets that are short-lived, have a narrow blast radius and low value, with no single password for all administrators. The credential the pipeline uses to log in to the secrets system should be rotated frequently, expire when the job completes and cover only the secrets that job needs. Every call must be attributable to the person or service that triggered the job.

Human access follows the same principle of least privilege: an engineer should not see every secret, and the system should allow permissions on individual entries. OWASP also recommends logging every action in the CI/CD tool, because an attacker can exfiltrate a secret through an administrative interface or a newly created job.

How do you catch a secret before it reaches the repository?

A secret is cheapest to stop on the developer’s machine, before the commit exists. OWASP recommends secrets detection at the developer level: in the IDE, in tests or in a pre-commit hook. The tool rejects a commit containing a string that resembles a key before anything leaves the machine.

The second line is the repository server. Push protection in GitHub blocks pushes that contain secrets before they reach the repository, rather than alerting after the fact. According to GitHub’s documentation, repository-level protection is disabled by default and is enabled by an administrator, while user-level protection on github.com is enabled by default and covers pushes to public repositories.

The third line is scanning what is already in the repository. Secret scanning in GitHub goes through the entire Git history on all branches and reports the credentials it finds as alerts. The documentation advises replacing the secret immediately after an alert and adds that removing it from history is time-intensive and often unnecessary once the secret has been revoked. CICD-SEC-6 recommends the same regardless of platform: automatic scanning on every push and periodic scanning of the repository together with its past commits.

What should you do when a secret has already leaked?

Revoke the secret first, then replace it, and clean up only at the end. OWASP describes containing a leak in four steps: revoking the exposed key, quickly creating and deploying a new one, deleting the secret from the system where it leaked, including code and logs, and gathering information about its history. The response team needs to know who had access to the secret, when it was used and when it was last rotated.

The order matters because rewriting repository history is costly and risky. GitHub’s documentation on removing sensitive data is clear: if the sensitive data is a password or token, the first step is to revoke or rotate it, and that may be sufficient. Rewriting history changes the hashes of all later commits, requires turning off branch protection against force pushes, and a developer with an old clone can bring the secret back to the repository with git pull followed by git push.

After replacing the secret, check whether anyone managed to use it. Only the logs of the system the key gave access to let you judge whether the leak was a repository mistake or an incident involving access to data.

Key takeaways:

  • After a leak the order is fixed: revocation, replacement, deletion, usage analysis.
  • Rewriting repository history does not remove the secret from other people’s clones and forks.
  • Pre-commit scanning and push blocking stop a secret before you ever have to revoke it.

How do you implement secrets management step by step?

Implementation starts with an inventory and ends with secrets that exist only for the duration of a job. The order follows a simple rule: you cannot protect secrets you do not know about, and automating the rotation of secrets that still sit in code makes no sense.

StageDecision to makeProof the stage is closedTypical mistake
1. InventoryWhich repositories, pipelines and images we scan across their full historyList of found secrets with an owner for eachScanning only the current branch instead of the full history
2. Revoking findingsWhich secrets we replace at once and which in a planned windowEvery found secret revoked or given a replacement dateRemoving the secret from code without revoking it
3. Choosing the secrets systemOne system or several, and where secret zero livesDocumented split of responsibilities and location of the primary credentialPrimary secret of the system kept in the same system
4. Migrating applicationsOrder of applications and how each fetches its secret at startupCode without secret values, only references to their namesMoving the secret from code to a variable in the pipeline repository
5. Secrets in CI/CDWhich jobs need which secretsPipeline credentials scoped to one job and with an expiry dateOne service account with access to every secret
6. Gate at the entrancePre-commit hook, push blocking, scanner in the pipelineA test commit with a fake key is rejectedA warn-only tool whose output nobody reads
7. Dynamic secrets and rotationWhich systems issue credentials on demand and which we rotateRotation schedule and dynamic secrets for databasesManual rotation that stops happening after the first holiday
8. Leak responseWho revokes the secret and how fastProcedure rehearsed on a test secretA procedure whose first step is rewriting history

Repeat stage six after every change to the repository configuration: a commit with a fake key in a real provider’s format shows at once whether the gate still works.

How can nFlo help with secrets management?

nFlo connects secrets management with the security of the whole software delivery pipeline. As part of the DevSecOps Implementation service, we build secrets detection with GitLeaks into the CI/CD pipeline, blocking commits that contain secrets, alongside code and dependency analysis. The wider background of these practices is collected in the knowledge base section on DevSecOps.

Where a secrets management system is needed, we implement vendor solutions. Delinea DevOps Secrets Vault generates dynamic secrets on demand, gives them a lifetime with automatic revocation and integrates with Jenkins, GitLab, GitHub Actions, Azure DevOps and Kubernetes. IBM HashiCorp Vault issues dynamic credentials for databases and cloud platforms, rotates secrets automatically, records access in audit logs and provides encryption as a service.

If you want to find out how many secrets sit in the history of your repositories today and where to start cleaning them up, contact us.

What else should you know about secrets management?

Is it enough to delete the secret from the code and make a new commit?

No. According to the CICD-SEC-6 risk description, a credential deleted from a branch still appears in the commit history. Revoke and replace the secret first, and only then decide whether to rewrite history.

Does putting the .env file in .gitignore solve the secrets problem?

Only partly: the file will not reach the repository, but the secret stays static and sits in plain text on developers’ and servers’ disks. A secrets management system hands it to the application at startup and lets you revoke it in one place.

How often should you rotate API keys?

As often as automation allows, and immediately when you suspect a leak. NIST SP 800-57 links the cryptoperiod of a key to limiting the damage of its compromise. User passwords are the exception: according to OWASP, they are changed when compromise is suspected.

Are secrets stored as CI/CD pipeline variables safe?

They are safer than secrets in code, but not out of reach. OWASP notes that project maintainers can often see or extract them, and CICD-SEC-6 describes secrets printed to job logs. Hence the advice: short lifetime and narrow scope.

What is secret zero in a secrets management system?

It is the credential an application or pipeline uses to log in to the secrets system itself. OWASP advises keeping the primary secret of such a system somewhere else, for example in the cloud provider’s facilities, and that provider’s administrative credentials elsewhere again.

Share:

Talk to an expert

Have questions about this topic? Get in touch with our specialist.

Sales Representative
Grzegorz Gnych

Grzegorz Gnych

Sales Representative

Response within 24 hours
Expert consultation
Individual approach

Providing your phone number will speed up contact.

Want to Reduce IT Risk and Costs?

Book a consultation - we respond within 24h

Response in 24h Custom quote No obligations