How to manage Secret rotation?

· Category: Kubernetes

Short answer

Secret rotation involves updating credentials in Kubernetes Secrets and triggering application reloads. Use versioning, rolling restarts, or hot-reload mechanisms to rotate secrets without downtime.

Steps

  1. Create a new Secret version or update the existing Secret.
  2. Update the application to read the new Secret.
  3. Trigger a rolling restart or signal the app to reload.
  4. Revoke old credentials in the backend.

Example

kubectl create secret generic db-secret   --from-literal=password=newpass   --dry-run=client -o yaml | kubectl apply -f -

Trigger a rolling restart:

kubectl rollout restart deployment/myapp

Tips

  • Use external secret operators like External Secrets Operator for automatic rotation.
  • Implement application support for hot-reloading configuration.
  • Audit Secret access with RBAC logs.

Common issues

  • Updating a Secret does not automatically update environment variables in running Pods.
  • Rolling restarts may briefly increase resource usage.
  • Old Secrets cached in application memory can cause connection failures after rotation.