HashiCorpVault on AWS

HashiCorp Vault on AWS

Vault is an enterprise secret management and encryption-as-a-service platform. On AWS it sits alongside IAM, KMS, and Secrets Manager — owning the domains those services do not cover as well: dynamic database credentials with sub-hour TTLs, transit-engine encryption for many small items, multi-cloud policy consistency, and centralised PKI for both AWS and on-prem certificates.

Licensing note (2026): IBM closed its acquisition of HashiCorp in early 2025. Vault remains under the Business Source License 1.1 adopted in August 2023 — free for all non-competing production use, with enterprise features behind a commercial licence. OpenBao (community fork) exists as a Linux Foundation project but lacks DR replication, namespaces, and FIPS transit for regulated workloads. Always verify current terms at hashicorp.com.

Why Vault on AWS

Centralised secret storage

• Passwords, API keys, TLS certificates, and tokens in one audited store.
• Encryption at rest (AES-256-GCM) with automatic key rotation; audit device captures every read, write, and token operation.
• KMS auto-unseal removes the operator burden of handling unseal keys in person.

Dynamic credentials

• Generate temporary RDS/Aurora/Redshift/MongoDB/Snowflake passwords on demand, valid for 1 hour by default, auto-revoked at expiry.
• Shrinks blast radius from “replay leaked creds forever” to “replay for under an hour”.
• Per-app credentials and per-session audit means you can answer “which microservice or pipeline run caused this DB lock?” without guesswork.

Transit engine (encryption-as-a-service)

• Send plaintext, receive ciphertext without Vault ever storing the plaintext.
• Convergent encryption, derived keys, and key rotation without re-encrypting every record.
• Throughput and cost profile that outperforms raw KMS calls for many small items.

Multi-cloud & on-prem

• Same policy model across AWS, Azure, GCP, and on-prem workloads — important for M&A, hybrid, and regulated environments that cannot put all secrets in one cloud.

Vault vs AWS Secrets Manager — decision matrix

Default recommendation: start with Secrets Manager for AWS-only workloads; add Vault when a specific driver above applies. Many regulated customers run both — Secrets Manager for AWS-service consumers, Vault for dynamic DB creds and transit, with Vault Secrets Sync keeping a one-way mirror to Secrets Manager for ECS/Lambda ergonomics.

Vault architecture on AWS

Self-hosted (control-plane-sensitive workloads)

• 3–5 node cluster on EC2 in an Auto Scaling Group across AZs.
• Integrated Storage (Raft) is now the HashiCorp-recommended backend — DynamoDB/S3 backends are still supported but Raft is simpler, faster, and enables performance replication to DR regions.
• Network Load Balancer in front for TLS termination via ACM.
• KMS auto-unseal — Vault uses an AWS KMS key to unseal itself after restart; rotate the KMS key annually.
• VPC endpoints for KMS, STS, and CloudWatch to keep traffic off the internet.

HCP Vault Dedicated (managed cluster)

• HashiCorp runs the cluster; you consume via AWS PrivateLink.
• Dev tier starts ~$200/month; production tiers scale by node count and replication.
• Best when you want a full Vault feature set without running the cluster yourself.

HCP Vault Secrets (lightweight SaaS) — GA 2024

• REST API for static secrets; free tier up to 25 secrets; paid from ~$0.03/secret/month.
• Best starting point for teams that need a managed key-value store with better audit than Parameter Store but do not yet need dynamic or transit.

Authentication methods we deploy

• AWS auth method — EC2 instances and Lambda functions authenticate to Vault using their instance identity document or IAM role; Vault verifies via AWS STS.
• Kubernetes auth — pods authenticate with their projected ServiceAccount token; Vault verifies against the cluster’s TokenReview API. On EKS, pair with Pod Identity for outbound calls.
• OIDC / JWT — authenticate GitHub Actions, GitLab CI, and human SSO via an OIDC trust relationship; pairs with OIDC subject-claim filtering similar to the pattern we use for AWS IAM + GitHub Actions.
• AppRole — service-to-service authentication for on-prem or legacy workloads that cannot use IAM/OIDC.

Secret engines we deploy on AWS

• Database — dynamic RDS/Aurora (Postgres/MySQL), Redshift, MongoDB Atlas, and Snowflake credentials with configurable TTL and max-TTL.
• AWS — generate temporary IAM access keys or assume-role credentials; useful for short-lived CLI sessions or third-party tools that cannot use IAM directly.
• Transit — encryption-as-a-service with convergent encryption, key rotation, and datakey generation.
• PKI — issue X.509 TLS certs for services running on AWS and on-prem; ACME server (Vault 1.14+) means cert-manager and traditional ACME clients can pull from Vault directly.
• SSH CA — sign short-lived SSH certs for engineer access to EC2 bastion hosts or on-prem Linux fleets.

Vault Secrets Operator (VSO) for EKS

The 2026 default pattern for Kubernetes workloads on EKS:

1. Install VSO via Helm; configure a VaultConnection and VaultAuth pointing at your Vault cluster with Kubernetes or JWT auth.
2. App teams declare VaultStaticSecret or VaultDynamicSecret CRDs in their namespace; VSO reconciles them into native Kubernetes Secrets that the app consumes as normal.
3. VSO handles renewal and rotation automatically; dynamic secrets flow into a rolling Deployment restart when TTL approaches expiry.
4. Pair with EKS Pod Identity for the outbound AWS calls VSO makes during auth verification.

This replaces the legacy Vault Agent sidecar + init-container pattern for most workloads. Fall back to Agent-sidecar + tmpfs when secrets must never sit in etcd.

Vault 1.17 / 1.18 features worth enabling

• Adaptive overload protection — targeted request-type throttling so a misbehaving client cannot take the whole cluster down.
• Multi-issuer PKI with ACME — Vault can be the ACME server for cert-manager on EKS and for external workloads.
• KV v2 transformations — patch subkeys atomically; useful for complex configuration documents.
• Secrets Sync GA — one-way sync Vault → AWS Secrets Manager / GitHub / GCP / Vercel / HCP Terraform; keeps AWS-native consumers working while Vault stays the source of truth.
• Workload identity federation — authenticate Vault to other systems without stored IAM credentials.

Implementation: VaultDynamicSecret with VSO

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultDynamicSecret
metadata:
  name: orders-db-creds
  namespace: orders
spec:
  vaultAuthRef: orders-eks-auth
  mount: database
  path: creds/orders-app
  destination:
    create: true
    name: orders-db
  refreshAfter: 30m # rotate before TTL expires
  rolloutRestartTargets:
    - kind: Deployment
      name: orders-api

The rolloutRestartTargets restarts the Deployment on rotation so applications pick up new credentials cleanly. For applications that hot-reload secrets without restart, omit and have the app re-read the projected secret on 403 PERMISSION_DENIED.

Failure modes & resilience

1. Leader election timeout / quorum loss. Raft requires a majority — losing 2 of 3 nodes makes the cluster unavailable. Recovery: identify the surviving node, run vault operator raft remove-peer for the dead nodes, then vault operator raft join for replacements. Critical: never force-leave a healthy node by mistake. Practice this in a non-prod cluster every quarter.

2. KMS auto-unseal key revocation. If the auto-unseal KMS key is deleted, scheduled for deletion, or has its key policy revoked, restarted Vault nodes cannot unseal. Mitigation: KMS key with DeletionWindowInDays = 30; alert on key policy changes via CloudTrail; replicate the key to a DR region with aws_kms_replica_key. Test annual key rotation in staging — the auto-unseal seal handles rotation transparently if both old and new versions are accessible.

3. Transit-engine throughput ceiling. A single Vault node maxes around 2,000–5,000 transit ops/s depending on instance type and key type (AES-GCM faster than RSA). Symptom: 429 Too Many Requests on transit endpoints. Mitigation: scale horizontally via performance replication standby clusters serving read-heavy transit traffic; use derived = true keys to amortize key derivation; consider client-side envelope encryption with KMS-wrapped DEKs for very high throughput.

4. Dynamic-DB credential renewal collisions. A burst of new pods all requesting credentials simultaneously can hit per-database connection limits while Vault creates short-lived users. Mitigation: set DB role max_connections_per_role; use default_ttl ≥ 1h so steady-state pods don’t churn; alarm on database_connections_used for the upstream RDS/Aurora.

5. Audit device backpressure. Vault blocks all requests if audit logging fails. Symptom: cluster appears healthy but every request 500s. Mitigation: configure two audit devices (file + syslog or file + socket); the request succeeds if at least one writes. Monitor audit-device disk usage and rotate logs aggressively.

6. Token-policy drift. Long-lived service tokens accumulate over years; orphaned tokens from departed engineers persist. Mitigation: enforce token max-TTL via auth/token/tune; run vault list auth/token/accessors quarterly and revoke orphans; prefer auth methods (AWS, K8s, OIDC) over raw tokens.

7. Performance replication lag. Read-heavy secondaries can lag the primary during heavy write traffic. Stale reads on a secondary may serve outdated dynamic credentials. Mitigation: route writes and reads-after-writes to the primary; alert on vault.replication.wal_lag exceeding the SLA.

Observability runbook

Metrics to scrape (Vault Prometheus endpoint or Datadog Vault integration):

Raft snapshot + restore runbook:

# Daily snapshot — schedule via systemd timer or EventBridge-triggered runner
vault operator raft snapshot save /tmp/vault-$(date +%F).snap
aws s3 cp /tmp/vault-$(date +%F).snap \
  s3://acme-vault-snapshots/$(date +%F)/ \
  --sse aws:kms --sse-kms-key-id alias/vault-snapshot

# Restore (DR — only in a fresh cluster)
vault operator raft snapshot restore -force /tmp/vault-2026-04-29.snap
# Re-init unseal: each operator unseals with their key share or KMS auto-unseal

Test restore quarterly to a staging cluster. Tabletop drill: “primary AZ is gone — recover in under 30 minutes”. Document the actual time and improve.

Unseal-key rotation cadence: Rotate the KMS auto-unseal key annually by adding a new alias and updating the seal stanza:

seal "awskms" {
  region     = "eu-west-1"
  kms_key_id = "alias/vault-unseal-2026"
}

The seal supports key migration via vault operator seal-migration. Old material remains accessible for decrypt during the transition; do not delete the previous key until the migration is verified across all nodes.

Debug path: “client getting 403”:

1. vault token lookup (with the client token) — check expiry, policies, accessor.
2. vault read sys/policies/acl/<policy> — confirm path and capability match the failed request.
3. Audit log on the Vault node: search for the request ID; look at error field.
4. If using AWS or K8s auth: confirm the bound entity (instance role, ServiceAccount) still matches the auth role; trust source rotated?
5. Adaptive overload: check vault.adaptive_overload.throttled_count — the client may be throttled, not denied.

When Vault is NOT the right call

• Single-cloud AWS workload with simple static/rotated secrets — use AWS Secrets Manager + KMS CMKs; audit via CloudTrail; done.
• No need for dynamic credentials and no multi-cloud ambition — Vault’s operational cost outweighs the benefit.
• Tiny team with no dedicated platform engineer — HCP Vault Secrets is the lightweight option, but even that is extra surface vs Secrets Manager for most use cases.
• Hard requirement for OSI-approved open source — OpenBao exists, but at the cost of enterprise features you probably need if you were considering Vault in the first place.

Best practices

Security

• Enable audit device to a forward-only CloudWatch Logs stream and S3 Object Lock bucket; treat audit log integrity as part of the SOC 2 / PCI evidence pipeline.
• Human access to Vault admin functions requires MFA and a separate admin namespace — never the root token.
• Rotate KMS auto-unseal keys annually; tie the rotation into the control-plane pipeline.
• IP allow-list the Vault API via VPC endpoint policies or AWS Network Firewall rules for public-facing ALBs.

Operations

• Back up Raft snapshots to an S3 bucket with Object Lock; test restore quarterly.
• Performance replication or DR replication to a secondary region for RPO-sensitive workloads.
• Monitor with Datadog, CloudWatch, or the native Prometheus endpoint; alert on leader changes, sealed state, and adaptive-overload throttling.

Application integration

• Prefer Vault Secrets Operator over sidecars for EKS workloads.
• Cache secrets in-process with a TTL slightly shorter than Vault’s TTL; handle 403 gracefully by re-fetching.
• Never log secret values — enable Vault’s secret-ID filtering on audit devices so secret material never hits logs.

Related reading

• AWS Secrets Manager vs Parameter Store: when to use which
• PCI DSS compliance on AWS: architecture guide for fintech
• How to achieve SOC 2 compliance on AWS in 2026

Related services

• AWS Cloud Security
• Cloud Compliance Services
• DevOps Pipeline Setup