Theory
Secrets Management & Infrastructure Security
📋 Overview
Secrets Management is the specialized practice of securely storing, distributing, and controlling access to sensitive digital credentials such as API keys, database passwords, SSL certificates, and SSH keys. In a world of automated infrastructure and microservices, "Secret Sprawl"—the unintentional leaking of credentials in source code, logs, or environment variables—is a major security risk. A robust secrets management system ensures that sensitive data is encrypted, audited, and automatically rotated.
🏗️ Core Principles & Characteristics
- Encryption at Rest and Transit: Secrets must never be stored in plain text. Use AES-256 or similar standards.
- Dynamic Secrets: The ability to generate temporary, short-lived credentials on-demand (e.g., a DB user that exists only for 1 hour).
- Programmatic Access: Applications should fetch secrets via APIs or sidecar injectors at runtime, rather than reading from static config files.
- Secret Rotation: Automatically changing passwords on a schedule to minimize the "blast radius" of a potential leak.
- Leasing & Revocation: Every secret access has a Time-To-Live (TTL). If a service is compromised, its specific "lease" can be revoked immediately.
⚖️ Trade-offs: Pros & Cons
Pros
- Centralized Control: One place to manage all credentials across multiple clouds and on-premise servers.
- Compliance: Provides the audit trails required by SOC2, HIPAA, and PCI-DSS (who accessed what and when?).
- Reduced Human Error: Eliminates the need for developers to share passwords over Slack or email.
Cons
- The "Secret Zero" Problem: How do you securely give the application the initial token it needs to talk to the secret manager? (Solved via IAM roles or K8s Service Accounts).
- Operational Complexity: Maintaining a high-availability cluster like HashiCorp Vault requires significant engineering effort.
- Single Point of Failure: If the secret manager goes down, the entire infrastructure may fail to start or scale.
🌍 Real-World Implementation
- HashiCorp Vault: The gold standard for platform-agnostic secrets management. Supports "Encryption as a Service" and dynamic backends for AWS, SQL, and more.
- AWS Secrets Manager / Azure Key Vault: Managed services that integrate natively with their respective cloud ecosystems (e.g., RDS auto-rotation).
- Kubernetes Secrets: Native but primitive. Warning: By default, K8s secrets are only Base64 encoded, not encrypted. Always enable "Encryption at Rest" for the Etcd store.
- External Secrets Operator (ESO): A Kubernetes operator that syncs secrets from external APIs (like Vault or AWS) into native K8s secrets for ease of use.
💡 Interview "Gotchas" & Tips
- Gotcha: Base64 is NOT Encryption. If an interviewer asks about K8s secrets, immediately clarify that Base64 is just an encoding format. Anyone with access to the YAML can decode it instantly.
- Gotcha: Hardcoded Secrets in Git. If found, "deleting" the line is not enough. The secret must be Revoked and Rotated because it remains in the Git history.
- Tip: Mention Sidecar Injection. Tools like the Vault Agent can inject secrets directly into a shared memory volume (
/dev/shm) so the application thinks it's reading a local file, but the secret never touches the disk. - Tip: Discuss the Blast Radius. Using one "Master Secret" for all services is a disaster waiting to happen. Use fine-grained IAM policies.
📐 Suggested Architecture Primitives
- Vault Cluster: High Availability (Raft or Consul backend).
- AppRole / IAM Auth: For machine-to-machine authentication.
- Sentinel / ACL Policies: To enforce "Least Privilege."
- Audit Sink: Streaming logs to Splunk or ELK for security monitoring.
- KMS (Key Management Service): For "Unsealing" the vault automatically.
Canvas