HashiCorp Vault is a secrets management platform that provides secure storage, dynamic credentials, encryption, and audit logging. Integrating Vault with Kubernetes lets workloads authenticate using their Kubernetes identity (ServiceAccount) and consume secrets without hardcoding them in manifests or images. This tutorial covers the architecture, data flow, and concrete examples for Vault–K8s integration.

Why Vault with Kubernetes?

• Centralized secrets: One source of truth for DB credentials, API keys, TLS certs; no duplication in ConfigMaps/Secrets or CI variables.
• Dynamic secrets: Vault can generate short-lived credentials (e.g. database users) that are created on demand and revoked when the lease expires.
• Kubernetes-native auth: Pods authenticate to Vault using their ServiceAccount JWT; no long-lived static tokens in the cluster.
• Audit and compliance: All secret access is logged; supports compliance requirements and rotation policies.

---

Architecture Overview

The integration involves: (1) Vault (outside or inside the cluster), (2) Kubernetes Auth Method in Vault so that K8s ServiceAccounts can log in, and (3) a delivery mechanism that gets secrets from Vault into your pods. The main options are: Vault Agent (injector), CSI Secrets Store driver with Vault provider, and External Secrets Operator.

High-level architecture

+---------------------- EXTERNAL / IN-CLUSTER ------------------------+
|                                                                     |
|   +-------------------+         +---------------------------------+ |  
|   |  HashiCorp Vault  |         |     Kubernetes Cluster          | |
|   |  - Secret engines |         |  +--------------------------+   | | 
|   |  - K8s auth method| <------>|  | API Server / etcd        |   | |
|   |  - Policies       |  HTTPS  |  | (ServiceAccount JWTs)    |   | |
|   +--------+----------+         |  +------------+-------------+   | |
|            |                    |               |                 | |
|            | 1. Login (JWT)     |               v                 | |
|            | 2. Read secrets    |  +--------------------------+   | |
|            |                    |  | Vault Agent Injector     |   | |
|            +--------------------+->| or CSI driver / ESO      |   | |
|                                 |  | (fetch & inject secrets) |   | |
|                                 |  +--------------------------+   | |
|                                 +------------+--------------------+ |
|                                               |                     |
|                                               v                     |
|                                  +------------------------+         |
|                                  | Pod (your app)         |         |
|                                  | - env / files from     |         |
|                                  |   Vault                |         |
|                                  +------------------------+         |
|                                                                     |
+---------------------------------------------------------------------+

Components

Component	Role
Vault server	Stores secrets, runs auth methods (e.g. Kubernetes), applies policies. Can run outside the cluster or as a Helm deployment inside.
Kubernetes Auth Method	Configured in Vault. Validates JWT from a pod’s ServiceAccount against the K8s API; issues a Vault token with limited TTL and policies.
Vault Agent (Injector)	Mutating webhook that injects an init container and sidecar into annotated pods. The sidecar fetches secrets from Vault and writes them to a shared volume; the app reads from that volume (or env).
Secrets Store CSI driver + Vault provider	Kubernetes CSI driver that mounts a volume; the volume content is fetched from Vault at mount time. No sidecar; kubelet mounts the volume.
External Secrets Operator (ESO)	Syncs secrets from Vault (and other stores) into native Kubernetes Secret objects. Workloads use standard K8s Secrets; ESO refreshes them on an interval or when the external secret changes.

---

Data Flow

1. Authentication flow (Kubernetes auth method)

When a pod (or the injector/CSI/ESO) needs a Vault token, it uses the Kubernetes auth method:

+--------+    1. SA JWT      +-------------+    2. Verify JWT   +------------+
|  Pod   | ----------------->|    Vault    | ------------------>|  K8s API   |
| (SA)   |                   | (k8s auth)  | <------------------|  Server    |
+--------+                   +-------------+    3. Token +      +------------+
       ^                            |            policies
       |                            | 4. Vault token (TTL)
       |                            v
       |                     +-------------+
       +---------------------|  Read       |
        5. Use token to      |  secrets    |
           read secrets      +-------------+

Steps:

1. Pod has a ServiceAccount; the projected token (or mounted SA token) is a JWT.
2. Pod (or injector/CSI/ESO) sends a login request to Vault: role=<vault-role>&jwt=<service-account-jwt>.
3. Vault (Kubernetes auth method) calls the Kubernetes API to validate the JWT (TokenReview).
4. Vault checks that the request matches the auth role (allowed service_account_names, namespaces, etc.); if valid, it issues a Vault token with the policies attached to that role.
5. That token is used (by the injector, CSI driver, or ESO) to read secrets from Vault; the token has a short TTL and can be renewed.

So: no static Vault tokens in the cluster—only short-lived tokens derived from Kubernetes identity.

2. Secret delivery flow (Vault Agent Injector example)

With the Vault Agent Injector:

1. User applies Deployment with annotations (vault.hashicorp.com/agent-inject, etc.)
2. Mutating webhook sees the pod; injects init container + sidecar.
3. Init container: logs in to Vault (K8s auth), writes secrets to shared volume (e.g. /vault/secrets).
4. Main app container starts; reads env or files from /vault/secrets.
5. Sidecar (optional): keeps token alive and can re-render templates on lease renewal.

Data flow: Vault → (login with SA JWT) → Vault token → read secret → write to volume → app reads.

---

Prerequisites

• A Kubernetes cluster (kind, minikube, or cloud).
• Vault installed and unsealed (e.g. vault server -dev for local dev, or Helm in cluster).
• kubectl and vault CLI configured to talk to the cluster and Vault.

For the examples below we assume:

• Vault is reachable at http://vault.vault-system.svc.cluster.local:8200 (in-cluster) or at a URL you set in the examples.
• You have enabled the Kubernetes auth method and created a role that allows your app’s ServiceAccount/namespace.

---

Example 1: Enable Kubernetes auth and create a Vault role

This is the foundation: Vault trusts your cluster and issues tokens to a specific role.

1.1 Enable Kubernetes auth and configure it

Vault must know how to talk to the Kubernetes API to validate JWTs. You need:

• Kubernetes API URL (e.g. https://kubernetes.default.svc.cluster.local).
• ServiceAccount token that Vault uses to call the TokenReview API. In many setups this is the token of the vault ServiceAccount in the namespace where the Vault Helm chart (or injector) runs.

# Enable the auth method
vault auth enable kubernetes

# Configure it (replace with your K8s API and token/cert if needed)
vault write auth/kubernetes/config \
  kubernetes_host="https://kubernetes.default.svc.cluster.local" \
  token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
  kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt

Explanation:

• kubernetes_host: So Vault can call the K8s API (from inside or outside the cluster).
• token_reviewer_jwt: JWT of the SA that Vault uses for TokenReview; must have system:auth-delegator or equivalent.
• kubernetes_ca_cert: CA that signed the K8s API server cert, so Vault can verify TLS.

1.2 Create a policy and role for an app

Create a policy that allows reading one path (e.g. KV v2 secret for a DB password):

# Policy: allow read for a specific path
vault policy write app-policy - <<EOF
path "secret/data/myapp/*" {
  capabilities = ["read", "list"]
}
EOF

Then create a Kubernetes auth role that maps a K8s ServiceAccount + namespace to this policy:

vault write auth/kubernetes/role/myapp \
  bound_service_account_names=myapp-sa \
  bound_service_account_namespaces=default \
  policies=app-policy \
  ttl=1h

Explanation:

• bound_service_account_names: Only SAs named myapp-sa can use this role.
• bound_service_account_namespaces: Only in the default namespace (restrict in production).
• policies: The Vault token issued will have app-policy (read secret/data/myapp/*).
• ttl: Token lifetime; injector/CSI/ESO will renew before expiry.

1.3 Store a secret

Using KV v2 engine at secret/:

vault kv put secret/myapp/config \
  username="appuser" \
  password="s3cr3t-p@ss"

Now any pod that authenticates with the role myapp can read secret/myapp/config.

---

Example 2: Vault Agent Injector (annotation-based injection)

The Vault Agent Injector is a mutating webhook. You add annotations to your Deployment; the webhook injects an init container (and optionally a sidecar) that logs in and fetches secrets.

2.1 Install Vault Helm chart with injector

helm repo add hashicorp https://helm.releases.hashicorp.com
helm install vault hashicorp/vault \
  --set "injector.enabled=true" \
  --set "server.dev.enabled=true"   # dev mode for demo only

For production you would use external Vault or HA setup and disable dev server.

2.2 Deployment with annotations

The pod must use the ServiceAccount that matches the Vault role (myapp-sa in default). Annotations control which secrets are injected and where.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: myapp-sa
  namespace: default
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
      annotations:
        # Enable injection
        vault.hashicorp.com/agent-inject: "true"
        # Vault role from Kubernetes auth
        vault.hashicorp.com/role: "myapp"
        # Path in Vault (KV v2)
        vault.hashicorp.com/agent-inject-secret-config: "secret/data/myapp/config"
        # File name under the volume (default mount is /vault/secrets)
        vault.hashicorp.com/agent-inject-template-config: |
          {{- with secret "secret/data/myapp/config" -}}
          export DB_USER="{{ .Data.data.username }}"
          export DB_PASSWORD="{{ .Data.data.password }}"
          {{- end -}}
    spec:
      serviceAccountName: myapp-sa
      containers:
        - name: app
          image: my-registry.io/myapp:latest
          command: ["/bin/sh", "-c", "source /vault/secrets/config && exec ./myapp"]
          volumeMounts:
            - name: vault-secrets
              mountPath: /vault/secrets
              readOnly: true
      volumes:
        - name: vault-secrets
          emptyDir:
            medium: Memory

Explanation:

• vault.hashicorp.com/agent-inject: Turns on the injector for this pod.
• vault.hashicorp.com/role: Kubernetes auth role name (myapp).
• vault.hashicorp.com/agent-inject-secret-<key>: Which Vault path to read; the key (config) is used for the filename.
• vault.hashicorp.com/agent-inject-template-<key>: Template for the file content. The init container fetches the secret and renders this template; output is written to /vault/secrets/config. The app then sources it to get DB_USER and DB_PASSWORD.

Data flow: Pod created → webhook injects init container → init container logs in (K8s auth) → reads secret/data/myapp/config → renders template → writes to /vault/secrets/config → main container starts and sources the file.

2.3 Multiple secrets and file format

You can inject multiple secrets; each needs a pair of annotations (secret + template):

vault.hashicorp.com/agent-inject-secret-db: "secret/data/myapp/config"
vault.hashicorp.com/agent-inject-template-db: |
  {{- with secret "secret/data/myapp/config" -}}
  DB_USER={{ .Data.data.username }}
  DB_PASSWORD={{ .Data.data.password }}
  {{- end -}}
vault.hashicorp.com/agent-inject-secret-api-key: "secret/data/myapp/api"
vault.hashicorp.com/agent-inject-template-api-key: |
  {{- with secret "secret/data/myapp/api" -}}
  {{ .Data.data.key }}
  {{- end -}}

The app can read /vault/secrets/db and /vault/secrets/api-key (or source them). Templates can output JSON, env format, or raw values.

---

Example 3: CSI Secrets Store driver with Vault provider

With the Secrets Store CSI driver, you request a volume whose content is provided by a driver. The Vault provider fills that volume with data from Vault. No sidecar; the volume is mounted by kubelet.

3.1 Install CSI driver and Vault provider

# Install Secrets Store CSI driver
helm install csi-secrets-store secrets-store-csi-driver/secrets-store-csi-driver \
  --namespace kube-system

# Install Vault provider (fills the volume from Vault)
helm install vault-provider hashicorp/vault-secrets-store \
  --set vault.address="http://vault.vault-system.svc.cluster.local:8200" \
  --set vault.authPath="kubernetes"

3.2 SecretProviderClass (Vault path and auth)

A SecretProviderClass tells the CSI driver what to fetch from Vault and how to authenticate (which Vault role).

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: myapp-vault-spc
  namespace: default
spec:
  provider: vault
  parameters:
    vaultAddress: "http://vault.vault-system.svc.cluster.local:8200"
    roleName: "myapp"
    objects: |
      - objectName: "config"
        secretPath: "secret/data/myapp/config"
        secretKey: "username"
      - objectName: "config"
        secretPath: "secret/data/myapp/config"
        secretKey: "password"

Explanation:

• roleName: Same Kubernetes auth role (myapp).
• objects: List of key-value pairs. Each objectName becomes a file name in the mount; for KV v2 you specify secretPath and secretKey. Here we get username and password from secret/data/myapp/config; the driver can create two files or you structure keys (e.g. one file per path).

(Note: Exact format for multiple keys can vary by provider version; some use one file per path with JSON or one key per file. Adjust objectName/file names as needed.)

3.3 Pod that mounts the CSI volume

apiVersion: v1
kind: Pod
metadata:
  name: myapp-csi
  namespace: default
spec:
  serviceAccountName: myapp-sa
  containers:
    - name: app
      image: my-registry.io/myapp:latest
      volumeMounts:
        - name: vault-secrets
          mountPath: "/mnt/secrets-store"
          readOnly: true
  volumes:
    - name: vault-secrets
      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
          secretProviderClass: "myapp-vault-spc"

When the pod starts, kubelet calls the CSI driver; the driver (with Vault provider) uses the pod’s ServiceAccount to log in to Vault, fetches the secrets, and fills the volume at /mnt/secrets-store. Your app reads files from that path. No init container or sidecar.

Data flow: Pod scheduled → kubelet mounts volume → CSI driver + Vault provider use SA JWT → Vault login → read secrets → populate volume → app reads files.

---

Example 4: External Secrets Operator (sync to Kubernetes Secret)

External Secrets Operator syncs secrets from Vault into native Kubernetes Secrets. Your Deployment uses a normal secretKeyRef or volume; ESO keeps the Secret updated.

4.1 Install ESO and create a SecretStore (Vault backend)

helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets external-secrets/external-secrets -n external-secrets --create-namespace

Define a SecretStore that tells ESO how to reach Vault and authenticate (e.g. Kubernetes auth):

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: vault-backend
  namespace: default
spec:
  provider:
    vault:
      server: "http://vault.vault-system.svc.cluster.local:8200"
      path: "secret"
      version: "v2"
      auth:
        kubernetes:
          mountPath: "kubernetes"
          role: "myapp"
          serviceAccountRef:
            name: "myapp-sa"

Explanation:

• server / path / version: Vault address and KV v2 engine.
• auth.kubernetes: Use Kubernetes auth; role is the Vault role; serviceAccountRef is the SA that ESO uses when talking to Vault (often a dedicated SA with a Vault role that can read the paths you sync).

4.2 ExternalSecret (what to sync and target K8s Secret)

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: myapp-external-secret
  namespace: default
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault-backend
    kind: SecretStore
  target:
    name: myapp-secret
    creationPolicy: Owner
  data:
    - secretKey: username
      remoteRef:
        key: myapp/config
        property: username
    - secretKey: password
      remoteRef:
        key: myapp/config
        property: password

Explanation:

• secretStoreRef: Use the vault-backend SecretStore.
• target: Create/update a Kubernetes Secret named myapp-secret with keys username and password.
• data: Each entry maps a secretKey (in the K8s Secret) to a Vault path and property. For KV v2, key is the path under the engine (e.g. myapp/config), property is the field name.

ESO periodically (e.g. every refreshInterval) or on webhook re-fetches from Vault and updates the Secret.

4.3 Deployment using the synced Secret

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
    spec:
      containers:
        - name: app
          image: my-registry.io/myapp:latest
          env:
            - name: DB_USER
              valueFrom:
                secretKeyRef:
                  name: myapp-secret
                  key: username
            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: myapp-secret
                  key: password

No Vault annotations or CSI; the app just uses standard Kubernetes Secrets. ESO keeps myapp-secret in sync with Vault.

Data flow: ESO controller watches ExternalSecret → uses SecretStore (K8s auth) to get Vault token → reads Vault path → creates/updates K8s Secret → Deployment mounts/uses Secret.

---

Comparison and when to use which

Approach	Pros	Cons	Best for
Vault Agent Injector	Flexible templates, no extra CRDs, per-pod injection	Sidecar/init container, annotation-heavy	Apps that need secrets as files or env with custom format
CSI + Vault provider	No sidecar, standard volume mount, good for many pods	Less flexible than templates; driver install per cluster	Simple file-based secrets, high pod count
External Secrets Operator	Native K8s Secrets, works with any app, multi-backend (Vault, AWS, Azure, etc.)	Secret lives in etcd (encrypt at rest); refresh delay	Teams that prefer standard Secrets and multi-cloud stores

---

Summary

• Architecture: Vault holds secrets; Kubernetes auth method issues short-lived tokens to pods (by ServiceAccount); an injector, CSI driver, or ESO delivers secrets to workloads.
• Data flow: Pod (or operator) sends SA JWT to Vault → Vault validates with K8s API → Vault returns token → secrets are read and either injected into the pod (Agent/CSI) or synced into a K8s Secret (ESO).
• Examples: (1) Enable K8s auth and create a role and policy. (2) Use Vault Agent Injector with annotations and a custom template. (3) Use CSI Secrets Store with a SecretProviderClass and volume mount. (4) Use External Secrets Operator with SecretStore and ExternalSecret to sync into a Kubernetes Secret.

Choose the delivery mechanism based on whether you need templating, want to avoid sidecars, or prefer standard Kubernetes Secrets with multi-backend support.