Managed Secrets in CI/CD and Deployment
#secrets#cicd#security#devops#vault#kubernetes
Secrets in CI/CD and deployment—API keys, database credentials, certificates, tokens—must be handled so they never live in source code or logs and are available only where needed. Managed secrets means using a dedicated store (a vault or your CI/CD / platform secrets) and injecting them at build time or runtime instead of hardcoding or copying them manually.
Why managed secrets matter
- Security: Secrets in code or config get committed, forked, and logged. One leak can compromise multiple systems.
- Rotation: When you rotate a credential, you want one place to update; pipelines and apps should pull the new value automatically.
- Audit: Who accessed which secret and when should be logged. Managed stores support access logging and integration with IAM.
- Least privilege: Pipelines and apps get only the secrets they need, for the time they need them.
Where secrets are needed
| Phase | Examples | Where they live |
|---|---|---|
| CI (build/test) | Registry credentials, test DB URL, API keys for lint/scan, deployment tokens | CI/CD secret store or vault; injected as env vars or masked variables in the pipeline. |
| Deployment | Cloud credentials (e.g. Terraform service principal), K8s kubeconfig, Helm chart values | Pipeline secrets or vault; sometimes short-lived tokens from OIDC or workload identity. |
| Runtime (app) | DB passwords, third-party API keys, TLS certs | Secret manager or vault; app or platform (e.g. Kubernetes) fetches at startup or via sidecar. |
You often use the same backend (e.g. Azure Key Vault, HashiCorp Vault) for both CI and runtime, with different access policies for pipeline identities vs app identities.
Secret stores and where to use them
Cloud-native secret managers
| Store | Typical use | Notes |
|---|---|---|
| AWS Secrets Manager | Apps and CI in AWS; Lambda, ECS, EC2 | Rotation, IAM policies, cross-account possible. |
| Azure Key Vault | Apps and CI in Azure; AKS, App Service, Azure DevOps | Certificates, keys, secrets; managed identity for apps and pipelines. |
| GCP Secret Manager | Apps and CI on GCP; GKE, Cloud Run | IAM-bound; workload identity for pods. |
Use these when your CI and/or apps run on the same cloud; they integrate with IAM and managed identity so you avoid long-lived static keys where possible.
HashiCorp Vault
- Central store for multiple clouds and runtimes. Supports dynamic secrets (short-lived DB credentials), PKI, and encryption as a service.
- CI: Pipeline authenticates (e.g. JWT for GitHub Actions, OIDC); reads secrets from Vault; injects into env or writes to a temp file (never logged).
- Runtime: App or an init container/sidecar authenticates (e.g. K8s auth, JWT) and reads secrets at startup or on a schedule.
- Fits hybrid and multi-cloud; more operational overhead than a single cloud’s managed service.
CI/CD built-in secrets
| System | Feature | Usage |
|---|---|---|
| GitHub Actions | Repository / org secrets, environments | Reference as ${{ secrets.MY_SECRET }}; never echoed in logs. |
| GitLab CI | CI/CD variables (masked, protected) | Use in .gitlab-ci.yml; restrict by branch/environment. |
| Azure DevOps | Variable groups (linked to Key Vault or manual), secret variables | Reference in YAML or classic pipeline; use for service connections and app secrets. |
| Jenkins | Credentials plugin, Vault plugin | Bind credentials to jobs; use Vault for dynamic secrets. |
These are the right place for pipeline-only secrets (e.g. registry login, deployment token). For app runtime secrets, prefer a vault or cloud secret manager and have the app (or platform) pull from there so the same secret isn’t duplicated in CI and production config.
Kubernetes
- Secrets (and ConfigMaps for non-sensitive config): Stored in etcd; often base64; consider encryption at rest and RBAC. Good for cluster-internal, non-critical secrets.
- External Secrets Operator (ESO): Syncs from AWS Secrets Manager, Azure Key Vault, Vault, GCP Secret Manager into Kubernetes Secrets. Keeps the source of truth outside the cluster; ESO refreshes on a schedule or on change.
- Sealed Secrets: Encrypt secrets so the ciphertext can be committed; only the controller in the cluster can decrypt. Reduces risk of plaintext in Git but doesn’t replace a proper vault for rotation and audit.
- CSI Secret Store (e.g. AWS, Azure): Mount secrets as volumes so the pod reads from the provider API; no Kubernetes Secret object. Good for strict “never store in cluster” policies.
Patterns for CI/CD and deployment
Pattern 1: CI pulls from vault, passes to pipeline
Pipeline authenticates to the vault (OIDC, JWT, or a short-lived token), reads the secrets it needs, and exposes them as environment variables or temporary files. Build and deploy steps use those; they are never written to the repo or logs.
+-------------+ authenticate +------------------+
| CI pipeline| ------------------> | Vault / Key |
| (GitHub, | read secrets | Vault / Secrets |
| Azure | <------------------- | Manager |
| DevOps) | (env / file) +------------------+
+-------------+
|
| env: REGISTRY_PASSWORD, DB_URL, ...
v
+-------------+ deploy with +------------------+
| Build & | ------------------> | Runtime |
| Deploy | secrets in env | (K8s, VM, etc.) |
+-------------+ or config +------------------+
- Example (Azure DevOps + Key Vault): Variable group linked to Key Vault; pipeline gets secrets as variables; use for ARM/Terraform and for writing app config (e.g. env file) during release. App can also use managed identity to read from Key Vault at runtime so the same secret isn’t stored in the release artifact.
Pattern 2: Runtime pulls from secret manager (no secrets in deployment artifact)
Build and deployment produce only image + non-sensitive config. At runtime, the app (or an init container / sidecar) fetches secrets from the vault or cloud secret manager using workload identity or a minimal credential.
- Kubernetes + ESO: ESO syncs Key Vault / Secrets Manager / Vault into K8s Secrets; pods mount them or env from them. Rotation in the vault can trigger ESO to refresh.
- Kubernetes + CSI: Pod mounts a volume backed by the cloud API; no Secret in etcd.
- VM/container: App uses instance metadata + IAM (e.g. AWS IMDS, Azure IMDS) to get a token and call the secret manager API.
This keeps long-lived secrets out of CI and out of deployment manifests.
Pattern 3: Short-lived credentials and OIDC
Where supported, avoid storing static cloud credentials in CI:
- GitHub Actions + AWS: OIDC federation; the pipeline gets a short-lived token from GitHub and exchanges it for AWS credentials. No long-lived access key in GitHub Secrets.
- Azure DevOps + Azure: Use a service connection with a service principal; consider OIDC-based federation where available so the pipeline doesn’t hold a client secret.
- GCP: Workload identity federation for GitHub/GitLab so the pipeline uses a short-lived token.
Secrets that must exist (e.g. third-party API key, DB password) still go in a vault or CI secret store and are injected as above.
Best practices
| Practice | Description |
|---|---|
| Never in repo | No API keys, passwords, or certs in source code, config files in Git, or Dockerfile. Use env, vault, or CI variables. |
| Least privilege | CI identity and app identity get only the secrets they need; scope by path or label in the vault. |
| Rotate | Use rotation in the vault or secret manager; apps and pipelines pull the current value. Prefer short-lived tokens (OIDC, dynamic secrets). |
| Audit | Enable access logging in the secret store; alert on unusual access or failed auth. |
| Mask in CI | Mark values as secret so the CI system never prints them in logs. Avoid echo or logging of env vars that might contain secrets. |
| One source of truth | Prefer one store (e.g. Key Vault) for a given secret; CI and runtime read from it instead of duplicating in multiple systems. |
| Encryption | Secrets at rest and in transit (TLS, encrypted storage). In K8s, enable encryption at rest for Secrets and consider ESO/CSI so the cluster holds less sensitive data. |
Example: Azure DevOps + Azure Key Vault (deployment)
- Create a Key Vault and store the secret (e.g. DB connection string).
- In Azure DevOps, create a variable group and link it to the Key Vault (service connection with managed identity or service principal).
- In the pipeline, reference the variable group and use the variable (e.g.
$(DB-CONNECTION-STRING)) in a task or in a config file generated during release. Do not log it. - For runtime, either (a) the release writes the value to the target (e.g. App Service app settings, or a K8s Secret created by the pipeline), or (b) the app uses managed identity and reads from Key Vault at startup so the pipeline only needs permission to deploy, not to read the secret long-term.
Example: Kubernetes + External Secrets Operator
- Install ESO in the cluster; configure a SecretStore or ClusterSecretStore that points to your vault (e.g. Azure Key Vault, AWS Secrets Manager) and uses IRSA, workload identity, or a service account token.
- Create an ExternalSecret that maps a vault secret (or key) to a Kubernetes Secret name and keys.
- ESO creates/updates the Kubernetes Secret; pods reference it as usual (env or volume). When the secret rotates in the vault, ESO can refresh the K8s Secret on its sync interval.
This keeps the authoritative copy in the vault and avoids storing production secrets in Git or in the pipeline.
Summary
- Managed secrets = store in a vault or CI/CD secret store; inject at build or runtime; never in code or logs.
- CI: Use CI secret variables or integrate with a vault (OIDC/short-lived where possible); pass secrets only as env or temp files and mark them masked.
- Deployment: Prefer runtime pull from a secret manager (ESO, CSI, or app calling the vault) so deployment artifacts stay free of long-lived secrets.
- Best practices: One source of truth, least privilege, rotation, audit, and no secrets in the repository. Combining a central store (Key Vault, Secrets Manager, Vault) with CI and runtime patterns keeps secrets manageable and secure across CI/CD and deployment.
Comments