External Secrets Operator
External Secrets Operator (ESO) synchronizes secrets from external APIs into Kubernetes Secrets. It decouples secret lifecycle management from Kubernetes — secrets live in your existing secret store (AWS SM, Vault, GCP SM, Azure KV) and ESO keeps them in sync automatically.
Installation
Install via Helm
# Add Helm repository
helm repo add external-secrets https://charts.external-secrets.io
helm repo update
# Install ESO
helm install external-secrets external-secrets/external-secrets \
--namespace external-secrets \
--create-namespace \
--set installCRDs=true
# Verify
kubectl get pods -n external-secrets
kubectl get crds | grep external-secrets.io
Install with Cert-Manager Integration
helm install external-secrets external-secrets/external-secrets \
--namespace external-secrets \
--create-namespace \
--set installCRDs=true \
--set webhook.certManager.enabled=true \
--set webhook.certManager.cert.issuerRef.name=letsencrypt-prod \
--set webhook.certManager.cert.issuerRef.kind=ClusterIssuer
Install via Manifest
kubectl apply -f \
https://raw.githubusercontent.com/external-secrets/external-secrets/main/deploy/crds/bundle.yaml
Configuration
AWS Secrets Manager SecretStore
# First, create a Kubernetes Secret with AWS credentials
kubectl create secret generic aws-secret \
--namespace production \
--from-literal=access-key=<ACCESS_KEY> \
--from-literal=secret-access-key=<SECRET_KEY>
---
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
name: aws-secretsmanager
namespace: production
spec:
provider:
aws:
service: SecretsManager
region: us-east-1
auth:
secretRef:
accessKeyIDSecretRef:
name: aws-secret
key: access-key
secretAccessKeySecretRef:
name: aws-secret
key: secret-access-key
AWS SecretStore with IRSA (IAM Roles for Service Accounts)
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
name: aws-secretsmanager-irsa
namespace: production
spec:
provider:
aws:
service: SecretsManager
region: us-east-1
auth:
jwt:
serviceAccountRef:
name: external-secrets-sa # SA with IRSA annotation
HashiCorp Vault SecretStore
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
name: vault-backend
namespace: production
spec:
provider:
vault:
server: https://vault.example.com
path: secret # KV mount path
version: v2 # KV v1 or v2
caBundle: <base64-ca>
auth:
kubernetes:
mountPath: kubernetes
role: external-secrets-role
serviceAccountRef:
name: external-secrets-sa
GCP Secret Manager ClusterSecretStore
# ClusterSecretStore — available to all namespaces
apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
name: gcp-secretmanager
spec:
provider:
gcpsm:
projectID: my-gcp-project
auth:
workloadIdentity:
clusterLocation: us-central1
clusterName: my-cluster
serviceAccountRef:
name: external-secrets-sa
namespace: external-secrets
Azure Key Vault SecretStore
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
name: azure-keyvault
namespace: production
spec:
provider:
azurekv:
tenantId: <TENANT_ID>
vaultUrl: https://my-keyvault.vault.azure.net
authType: WorkloadIdentity
serviceAccountRef:
name: external-secrets-sa
Core Commands
| Command | Description |
|---|---|
kubectl get secretstores -A | List all SecretStores |
kubectl get clustersecretstores | List all ClusterSecretStores |
kubectl get externalsecrets -A | List all ExternalSecrets |
kubectl get clusteexternalsecrets | List all ClusterExternalSecrets |
kubectl describe secretstore <name> | Show SecretStore status and conditions |
kubectl describe externalsecret <name> | Show ExternalSecret sync status |
kubectl get secret <name> -o json | Inspect synced Kubernetes Secret |
kubectl annotate es <name> force-sync=$(date +%s) | Force immediate sync |
kubectl get pushsecrets -A | List PushSecret resources |
kubectl logs -n external-secrets deploy/external-secrets | View controller logs |
kubectl get events -n <ns> --field-selector reason=Updated | Watch sync events |
Advanced Usage
ExternalSecret — Map Multiple Keys
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: database-credentials
namespace: production
spec:
refreshInterval: 1h # How often to sync
secretStoreRef:
name: aws-secretsmanager
kind: SecretStore
target:
name: database-secret # Kubernetes Secret name to create
creationPolicy: Owner # Owner | Merge | None
deletionPolicy: Retain # Delete | Merge | Retain
template:
type: Opaque
data:
connection-string: |
postgresql://{{ .username }}:{{ .password }}@{{ .host }}:5432/{{ .database }}
data:
- secretKey: username # Key in Kubernetes Secret
remoteRef:
key: prod/db/credentials # Path in AWS SM
property: username # JSON property within secret
- secretKey: password
remoteRef:
key: prod/db/credentials
property: password
- secretKey: host
remoteRef:
key: prod/db/credentials
property: host
ExternalSecret — Bulk Import All Keys
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: all-app-secrets
namespace: production
spec:
refreshInterval: 30m
secretStoreRef:
name: vault-backend
kind: SecretStore
target:
name: app-secrets
creationPolicy: Owner
dataFrom:
- extract:
key: prod/app/config # Pull all key-value pairs from this path
rewrite:
- regexp:
source: "(.+)"
target: "APP_$1" # Prefix all keys with APP_
ClusterExternalSecret (Multi-Namespace)
# Push the same secret into multiple namespaces
apiVersion: external-secrets.io/v1beta1
kind: ClusterExternalSecret
metadata:
name: shared-tls-cert
spec:
namespaceSelector:
matchLabels:
inject-tls: "true"
refreshTime: 1h
externalSecretSpec:
refreshInterval: 1h
secretStoreRef:
name: aws-secretsmanager
kind: ClusterSecretStore
target:
name: shared-tls
creationPolicy: Owner
data:
- secretKey: tls.crt
remoteRef:
key: shared/tls/certificate
property: cert
- secretKey: tls.key
remoteRef:
key: shared/tls/certificate
property: key
PushSecret (Write Kubernetes Secrets to External Store)
# Sync a Kubernetes Secret into AWS Secrets Manager
apiVersion: external-secrets.io/v1alpha1
kind: PushSecret
metadata:
name: push-db-secret
namespace: production
spec:
refreshInterval: 10m
secretStoreRefs:
- name: aws-secretsmanager
kind: SecretStore
selector:
secret:
name: database-secret
data:
- match:
secretKey: password
remoteRef:
remoteKey: prod/db/credentials
property: password
Secret Template with Go Templating
spec:
target:
name: app-config
template:
engineVersion: v2
data:
config.yaml: |
database:
host: {{ .dbhost }}
port: 5432
password: {{ .dbpassword | b64dec }}
redis:
url: redis://{{ .redispassword }}@redis:6379
metadata:
annotations:
managed-by: external-secrets
labels:
app: myapp
Common Workflows
Debug a Failed Sync
# Check ExternalSecret status
kubectl describe externalsecret database-credentials -n production
# Look for: Status.Conditions — Ready/False with reason
# Check SecretStore connectivity
kubectl describe secretstore aws-secretsmanager -n production
# Look for: Status.Conditions — Valid/False
# View ESO controller logs
kubectl logs -n external-secrets \
-l app.kubernetes.io/name=external-secrets \
--since=10m | grep -i error
# Check Kubernetes events
kubectl get events -n production \
--field-selector involvedObject.name=database-credentials \
--sort-by='.lastTimestamp'
Force an Immediate Sync
# Annotate the ExternalSecret to trigger a sync
kubectl annotate externalsecret database-credentials \
-n production \
force-sync=$(date +%s) \
--overwrite
# Watch the sync
kubectl get externalsecret database-credentials -n production -w
Rotate a Secret End-to-End
# 1. Update the secret in AWS Secrets Manager
aws secretsmanager update-secret \
--secret-id prod/db/credentials \
--secret-string '{"password":"new-password-123"}'
# 2. Force ESO to sync immediately
kubectl annotate externalsecret database-credentials \
-n production force-sync=$(date +%s) --overwrite
# 3. Verify the Kubernetes Secret was updated
kubectl get secret database-secret -n production -o jsonpath='{.data.password}' | base64 -d
# 4. Restart pods to pick up new secret (if not using env injection)
kubectl rollout restart deployment/my-app -n production
Validate SecretStore in CI
# Test SecretStore connectivity without deploying ESO
kubectl apply -f secretstore.yaml --dry-run=server
# Check SecretStore is ready after apply
kubectl wait secretstore/aws-secretsmanager \
-n production \
--for=condition=Ready \
--timeout=60s
Tips and Best Practices
- Use
ClusterSecretStorefor shared infrastructure secrets — avoids duplicating SecretStore configs (and credentials) in every namespace. - Use IRSA/Workload Identity instead of static credentials — pod-level IAM roles eliminate the need to store cloud credentials as Kubernetes Secrets.
- Set
refreshIntervalbased on rotation policy — for frequently rotated secrets (API keys), use5m; for long-lived certs,1his sufficient. - Use
creationPolicy: Owner— ESO owns the Secret and garbage-collects it when the ExternalSecret is deleted, preventing orphaned secrets. - Use
deletionPolicy: Retainfor critical secrets — prevents accidental deletion if the ExternalSecret is removed; safeguard for databases. - Never commit SecretStore credentials to Git — use Workload Identity/IRSA, or bootstrap with Sealed Secrets/Vault agent for the initial credential.
- Use secret templates to build structured configs — Go templating lets you compose secrets into connection strings, config files, or URLs directly.
- Monitor
externalsecret_sync_calls_errorin Prometheus — ESO exposes metrics; alert on sync errors before they cause application failures. - Use
ClusterExternalSecretwith namespace label selectors — automatically inject shared secrets (TLS, API keys) into new namespaces that match a label. - Audit with
kubectl get externalsecrets -A -o wide— shows the last sync time and status for every ExternalSecret at a glance.