Automating Secret Rotation with HashiCorp Vault

Introduction

Secret management remains one of the most critical challenges in modern cloud infrastructure. Static credentials, hardcoded API keys, and long-lived tokens create security vulnerabilities that attackers actively exploit. According to recent studies, compromised credentials are involved in over 80% of data breaches.

HashiCorp Vault provides a robust solution for dynamic secret generation and automated rotation, eliminating the risks associated with static credentials. In this article, I'll walk you through implementing automated secret rotation at scale across multi-cloud environments.

Why Secret Rotation Matters

Traditional secret management has several fundamental problems:

Static Credentials Live Forever: Once created, database passwords or API keys often remain unchanged for months or years, providing a persistent attack surface.

Manual Rotation is Error-Prone: Rotating secrets manually across dozens of services inevitably leads to outages when something is missed.

No Audit Trail: When everyone shares the same password, tracking who accessed what becomes impossible.

Blast Radius: If a static credential leaks, every system using it is compromised until rotation occurs.

Dynamic secrets solve these problems by generating credentials on-demand with short time-to-live (TTL) values that automatically expire.

HashiCorp Vault Architecture for Secret Rotation

Vault's secret rotation capabilities are built on several key components:

Secret Engines

Vault uses pluggable secret engines to generate and manage different types of secrets:

• Database Engine: Dynamic database credentials
• AWS Engine: IAM credentials with temporary tokens
• Azure Engine: Service principals and role assignments
• PKI Engine: X.509 certificates
• SSH Engine: One-time SSH credentials
• KV Engine: Key-value pairs with versioning

Lease Management

Every dynamic secret comes with a lease:

Secret Generated → Lease Created → TTL Expires → Secret Revoked

Clients can renew leases before expiration, and Vault automatically revokes expired secrets.

Authentication Methods

Vault supports multiple authentication methods for different environments:

• Kubernetes Auth: Pod service accounts
• AWS Auth: IAM roles and EC2 instance metadata
• Azure Auth: Managed identities
• AppRole: Machine-to-machine authentication
• JWT/OIDC: Identity provider integration

Implementing Database Secret Rotation

Let's start with a practical example: rotating database credentials for PostgreSQL.

Step 1: Configure the Database Engine

# Enable the database engine
vault secrets enable database

# Configure PostgreSQL connection
vault write database/config/postgres \
  plugin_name=postgresql-database-plugin \
  connection_url="postgresql://{{username}}:{{password}}@postgres:5432/mydb" \
  allowed_roles="readonly,readwrite" \
  username="vault_admin" \
  password="secure_password" \
  password_authentication="scram-sha-256"

Step 2: Create Dynamic Roles

Define roles that specify what permissions generated credentials should have:

# Read-only role
vault write database/roles/readonly \
  db_name=postgres \
  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}' IN ROLE readonly;" \
  revocation_statements="REVOKE readonly FROM \"{{name}}\"; DROP ROLE IF EXISTS \"{{name}}\";" \
  default_ttl="1h" \
  max_ttl="24h"

# Read-write role  
vault write database/roles/readwrite \
  db_name=postgres \
  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}' IN ROLE readwrite;" \
  default_ttl="30m" \
  max_ttl="4h"

Step 3: Generate Dynamic Credentials

Applications request credentials on-demand:

# Request credentials
vault read database/creds/readonly

# Output:
Key                Value
---                -----
lease_id          database/creds/readonly/2f6a614c
lease_duration    1h
lease_renewable   true
password          A1a-BbCcDdEeFfGg
username          v-root-readonly-48hr9t

Step 4: Application Integration

Here's how an application uses dynamic credentials:

import hvac
import psycopg2

# Authenticate to Vault
client = hvac.Client(url='https://vault.example.com')
client.auth.kubernetes.login(
    role='myapp',
    jwt=open('/var/run/secrets/kubernetes.io/serviceaccount/token').read()
)

# Get dynamic database credentials
db_creds = client.secrets.database.generate_credentials(
    name='readonly'
)

# Connect to database
conn = psycopg2.connect(
    host="postgres.example.com",
    database="mydb",
    user=db_creds['data']['username'],
    password=db_creds['data']['password']
)

# Use connection...
# Credentials automatically expire after TTL

Multi-Cloud Secret Rotation

Managing secrets across AWS, Azure, and GCP requires consistent patterns:

AWS Dynamic Credentials

# Enable AWS secrets engine
vault secrets enable aws

# Configure AWS
vault write aws/config/root \
  access_key=$AWS_ACCESS_KEY_ID \
  secret_key=$AWS_SECRET_ACCESS_KEY \
  region=us-east-1

# Create role for EC2 read-only
vault write aws/roles/ec2-readonly \
  credential_type=iam_user \
  policy_document=-<<EOF
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": "ec2:Describe*",
    "Resource": "*"
  }]
}
EOF

# Generate credentials
vault read aws/creds/ec2-readonly

Azure Service Principal Rotation

# Enable Azure secrets engine  
vault secrets enable azure

# Configure Azure
vault write azure/config \
  subscription_id=$AZURE_SUBSCRIPTION_ID \
  tenant_id=$AZURE_TENANT_ID \
  client_id=$AZURE_CLIENT_ID \
  client_secret=$AZURE_CLIENT_SECRET

# Create role
vault write azure/roles/contributor \
  azure_roles=-<<EOF
[{
  "role_name": "Contributor",
  "scope": "/subscriptions/$AZURE_SUBSCRIPTION_ID/resourceGroups/production"
}]
EOF
  ttl=1h \
  max_ttl=24h

# Generate service principal
vault read azure/creds/contributor

Implementing Root Credential Rotation

Even Vault's root credentials (the credentials Vault uses to access databases, cloud providers, etc.) should rotate regularly.

Automatic Root Rotation

Vault can automatically rotate root credentials for supported engines:

# Enable automatic rotation for database
vault write -f database/rotate-root/postgres

# Configure rotation schedule (requires Vault 1.16+)
vault write database/config/postgres \
  rotation_period="24h"

Manual Root Rotation Script

For engines that don't support automatic rotation:

#!/bin/bash
# rotate-db-root.sh

set -e

# Generate new password
NEW_PASSWORD=$(openssl rand -base64 32)

# Update database directly
PGPASSWORD=$OLD_PASSWORD psql -h postgres -U vault_admin -d postgres -c \
  "ALTER ROLE vault_admin WITH PASSWORD '$NEW_PASSWORD';"

# Update Vault configuration
vault write database/config/postgres \
  password="$NEW_PASSWORD"

# Store new password securely
vault kv put secret/vault/db-rotation \
  password="$NEW_PASSWORD" \
  rotated_at="$(date -Iseconds)"

echo "Root credential rotation completed"

Lease Management at Scale

Managing thousands of leases requires automation:

Automatic Lease Renewal

Applications should renew leases before expiration:

package main

import (
    "context"
    "log"
    "time"
    
    "github.com/hashicorp/vault/api"
)

func maintainLease(client *api.Client, leaseID string) {
    for {
        // Renew at 80% of TTL
        secret, err := client.Sys().Lookup(leaseID)
        if err != nil {
            log.Fatal(err)
        }
        
        ttl := secret.Data["ttl"].(float64)
        sleepDuration := time.Duration(ttl*0.8) * time.Second
        
        time.Sleep(sleepDuration)
        
        // Renew the lease
        _, err = client.Sys().Renew(leaseID, 0)
        if err != nil {
            log.Printf("Failed to renew lease: %v", err)
            // Handle renewal failure - get new credentials
            break
        }
    }
}

Lease Revocation Strategy

Implement graceful secret retirement:

# Revoke specific lease
vault lease revoke database/creds/readonly/abc123

# Revoke all leases under a path
vault lease revoke -prefix database/creds/readonly/

# Force revocation (careful!)
vault lease revoke -force -prefix database/creds/

Monitoring and Alerting

Track secret rotation health with metrics:

Key Metrics to Monitor

Metric	Alert Threshold	Why It Matters
Failed rotations	> 1%	Indicates credential issues
Lease renewal failures	> 5%	Apps may lose access
Expired credentials in use	> 0	Security risk
Root credential age	> 30 days	Compliance requirement
Rotation duration	> 5 minutes	Performance degradation

Vault Telemetry Configuration

# vault.hcl
telemetry {
  prometheus_retention_time = "30s"
  disable_hostname = false
}

# Key metrics to track:
# - vault.expire.num_leases
# - vault.secret.lease.creation
# - vault.database.{engine}.query
# - vault.rollback.attempt.{mount}

Prometheus Alerting Rules

groups:
  - name: vault_secrets
    rules:
      - alert: HighSecretRotationFailureRate
        expr: |
          rate(vault_database_query_error_total[5m]) / 
          rate(vault_database_query_total[5m]) > 0.01
        for: 5m
        annotations:
          summary: "High secret rotation failure rate"
          
      - alert: LeaseExpirationBacklog
        expr: vault_expire_num_leases > 10000
        for: 10m
        annotations:
          summary: "Large number of leases pending expiration"

Best Practices and Common Pitfalls

DO: Start with Short TTLs

Begin with aggressive rotation (15-30 minutes) and increase only if necessary:

# Short TTLs force you to handle rotation correctly
vault write database/roles/app \
  default_ttl="30m" \
  max_ttl="4h"

DON'T: Store Static Secrets in Vault

Vault's KV engine should only be for secrets that must be static. Prefer dynamic secrets whenever possible.

DO: Implement Circuit Breakers

Prevent cascading failures when Vault is unavailable:

from circuitbreaker import circuit

@circuit(failure_threshold=3, recovery_timeout=60)
def get_vault_credentials():
    return vault_client.read('database/creds/app')

DON'T: Forget About Rotation Downtime

Some systems can't handle credential rotation without downtime. Plan accordingly:

• Use connection pooling with credential refresh
• Implement blue-green credential deployment
• Test rotation in staging first

DO: Audit Everything

Enable comprehensive audit logging:

vault audit enable file file_path=/vault/logs/audit.log

# Monitor for:
# - Failed authentication attempts
# - Unusual credential access patterns  
# - Privilege escalation
# - Revocation failures

Conclusion

Automated secret rotation with HashiCorp Vault transforms security from a manual, error-prone process into a reliable, scalable system. By generating dynamic credentials with short lifespans, you eliminate the risks of static secrets while providing detailed audit trails and fine-grained access control.

Implementation Roadmap

1. Phase 1 (Week 1-2): Deploy Vault in HA configuration
2. Phase 2 (Week 3-4): Configure database dynamic secrets for non-production
3. Phase 3 (Week 5-6): Integrate applications with Vault SDK
4. Phase 4 (Week 7-8): Roll out to production with monitoring
5. Phase 5 (Week 9-10): Add cloud provider secret engines
6. Phase 6 (Ongoing): Reduce TTLs and improve observability

Next Steps

• Set up a Vault development environment
• Identify your highest-risk static credentials
• Implement dynamic secrets for one system first
• Measure rotation success rates and iterate

Secret rotation isn't just a security best practice—it's a requirement for modern cloud-native infrastructure. Start small, automate everything, and build confidence through testing.

---

Questions about implementing Vault in your environment? Feel free to reach out—I'd love to discuss your secret management strategy.