Theory
Amazon S3 & Object (Blob) Storage
📋 Overview
Amazon S3 (Simple Storage Service) is the industry-leading Object Storage service. Unlike traditional file systems (which use a hierarchy) or block storage (used by databases), S3 treats every file as an "Object" stored in a flat structure called a "Bucket." It is designed for 99.999999999% (11 nines) durability and is the foundational storage layer for modern cloud architectures, data lakes, and static asset hosting.
🏗️ Core Principles & Characteristics
- Buckets & Objects: Data is stored in containers called "Buckets." Each "Object" consists of data, a unique Key (the path), and Metadata.
- Scalability: Virtually infinite capacity. You can store petabytes of data without managing physical hardware.
- Durability (11 Nines): S3 automatically replicates data across multiple devices in at least three physically separated Availability Zones (AZs) within a region.
- Consistency Model: Since 2020, S3 provides Strong Read-After-Write Consistency for all applications, eliminating the old issues with eventual consistency during object creation or updates.
- Storage Classes:
- Standard: Frequent access.
- Intelligent-Tiering: Automatic cost savings for data with unknown access patterns.
- Glacier / Glacier Deep Archive: Extremely cheap for long-term backups (retrieval takes minutes to hours).
⚖️ Trade-offs: Pros & Cons
Pros
- Cost-Efficient: Pay-as-you-go pricing with tiered storage for archival data.
- Security: Deep integration with AWS IAM, Bucket Policies, and Encryption (SSE-S3, SSE-KMS).
- Accessibility: Every object is accessible via a unique RESTful URL.
Cons
- High Latency for Small Writes: Not suitable for high-frequency transactional data (use EBS or a DB instead).
- No Partial Updates: To change one byte in a 1GB file, you must re-upload the entire 1GB object.
- Metadata Search: You cannot "query" S3 metadata efficiently (usually requires an external index like DynamoDB or using S3 Select/Athena).
🌍 Real-World Implementation
- Static Asset Hosting: Serving images, CSS, and JS for web applications (often behind CloudFront).
- Data Lakes: Storing raw, unstructured data for Big Data processing (AWS Glue, EMR, Athena).
- Backup & Restore: Storing database snapshots and system logs for disaster recovery.
- Media Transcoding: Original video files uploaded to S3 trigger AWS Lambda for processing/compression.
💡 Interview "Gotchas" & Tips
- Gotcha: Public Access. By default, S3 buckets are private. "Leaking" data via misconfigured public buckets is a common real-world security failure. Always mention S3 Block Public Access.
- Gotcha: Performance Bottlenecks. If you have >3,500 PUT/POST/DELETE or >5,500 GET requests per second per prefix, you might hit limits. Use randomized prefixes (hashes) if you have extreme scale requirements.
- Tip: Mention Pre-signed URLs when asked how to allow a user to upload a file directly to S3 securely without exposing your AWS credentials.
- Tip: Mention Lifecycle Policies for automatic cost optimization (e.g., "Move files older than 30 days to Glacier").
📐 Suggested Architecture Primitives
- IAM Policies: Least privilege access control.
- Versioning: Protects against accidental deletes or overwrites.
- S3 Event Notifications: Triggering Lambda functions on file upload.
- Transfer Acceleration: Uses AWS Edge Locations for faster long-distance uploads.
- Multipart Upload: Essential for files larger than 100MB to ensure reliability and speed.
Canvas