System Design: Secrets Management System

Requirements

Functional Requirements:

• Store secrets (API keys, DB passwords, TLS certs, tokens) encrypted at rest
• Provide fine-grained access control: who can read/write which secret paths
• Support dynamic secrets: generate short-lived credentials on demand (DB passwords, cloud IAM tokens)
• Automatic secret rotation with zero-downtime for consuming services
• Full audit log of every secret access, including identity and timestamp
• Support multiple auth methods: AppRole, Kubernetes service accounts, AWS IAM, OIDC

Non-Functional Requirements:

• 99.999% availability — secret reads are on the critical path for service startup
• Sub-10ms p99 read latency for cached secrets
• Encryption keys must never exist unencrypted outside of a hardware security module (HSM) or trusted execution environment
• Horizontal scalability to serve 100,000 services across thousands of namespaces

Scale Estimation

A large enterprise: 10,000 services, each reading secrets at startup and every 5 minutes (lease renewal). That's ~33,000 reads/sec at steady state, with spikes to 200,000/sec during mass deployments. Secret values are small (< 10 KB each), so bandwidth is negligible. The audit log grows at ~1 KB per access × 33,000/sec = 33 MB/sec — requires a dedicated high-throughput append-only store. Total secrets stored: ~1 million entries, totaling < 10 GB of encrypted data.

High-Level Architecture

The secrets manager follows a server-agent model. A cluster of server nodes holds the encrypted secret store and handles all reads/writes. Client agents (sidecar containers or SDK calls) authenticate to the server, obtain short-lived tokens, and retrieve secrets — storing them in tmpfs mounts or environment variable injection.

The server cluster uses a consensus protocol (Raft) for strong consistency across replicas. One leader handles all writes; followers serve reads with bounded staleness. The storage backend (encrypted) can be an embedded BoltDB for small deployments or an external HA store (Consul, etcd, PostgreSQL) for production. All data is encrypted before being written to the backend using envelope encryption: each secret is encrypted with a data encryption key (DEK), which is itself encrypted with a key encryption key (KEK) stored in an HSM or cloud KMS.

Unseal keys protect the master key. At startup, server nodes are sealed — they cannot decrypt any data. Unsealing requires a quorum of key shares (Shamir's Secret Sharing: e.g., 3-of-5 key holders must provide their share). Auto-unseal integrations with AWS KMS or Azure Key Vault allow automated cluster restarts without human intervention.

Core Components

Auth Engine

Handles identity verification across multiple methods. Kubernetes auth validates service account JWTs against the cluster's OIDC endpoint. AppRole auth uses a role ID (public) + secret ID (private) pair — secret IDs are single-use and short-lived. AWS auth validates IAM signatures. Upon successful authentication, the auth engine issues a client token with attached policies and a TTL. Token renewal extends the TTL without re-authenticating, reducing auth overhead for long-running services.

Policy Engine

Policies are HCL or JSON documents defining path-based ACLs (read, write, list, delete, sudo). Policies are attached to tokens via auth roles. A policy like path "secret/data/prod/*" { capabilities = ["read"] } grants read access to all secrets under that prefix. Policies are evaluated on every API call — cached in memory for performance. Sentinel integration allows complex policy logic (e.g., time-of-day restrictions, MFA requirements for sensitive paths).*

Secret Engines

Pluggable backends that generate or store secrets. The KV engine stores arbitrary key-value pairs with versioning (KV v2 tracks up to N versions). The database engine connects to upstream databases and issues ephemeral credentials (username/password) on demand with a configurable TTL (e.g., 1 hour); when the lease expires, the engine revokes the credentials directly in the database. The PKI engine acts as a certificate authority, issuing X.509 certs with short TTLs (hours, not years) to eliminate manual rotation.

Database Design

The underlying storage is a simple key-value store where keys are paths (e.g., secret/data/myapp/db) and values are encrypted blobs. No SQL schema — the storage layer only sees opaque encrypted bytes. Metadata (key existence, version numbers, lease expiry) is stored in a separate logical namespace. This design allows swapping storage backends without changing encryption logic.

Audit logs are written to a separate, append-only sink (file, syslog, or direct to a SIEM like Splunk). Each log entry is HMAC-signed for tamper detection. Audit log writes are synchronous — if the audit sink is unavailable, the request is rejected to maintain the integrity guarantee that every access is logged.

API Design

Scaling & Bottlenecks

Read scaling: followers can serve reads if applications tolerate slight staleness (configurable). For strict consistency, all reads go through the leader, making leader throughput the bottleneck. Horizontal read scaling requires replication lag tolerance. Performance standby nodes (Vault Enterprise) serve stale reads while remaining up-to-date via replication, allowing near-linear read scaling.

Dynamic secret generation bottlenecks on upstream systems (databases, cloud IAM). Mitigation: pre-generate a pool of credentials asynchronously; issue from the pool on demand; replenish the pool in the background. Lease expiry processing requires a background goroutine scanning expiring leases — this can lag under high load, so lease expiry scans must be prioritized to avoid credential accumulation.

Key Trade-offs

• Auto-unseal vs. manual unseal: Auto-unseal (cloud KMS) improves operational resilience but creates a dependency on the cloud provider's KMS availability
• Dynamic secrets vs. static secrets: Dynamic secrets reduce blast radius (short TTL limits exposure window) but increase complexity and upstream system load
• Lease TTL length: Short TTLs improve security but increase renewal traffic and the risk of outages if the secrets manager is briefly unavailable
• Audit log synchronicity: Synchronous audit logging guarantees every access is recorded but creates a dependency — audit sink downtime blocks all secret access