Theory
Time-Series Databases (TSDB)
📋 Overview
A Time-Series Database (TSDB) is a specialized storage engine optimized for handling data that is indexed by time. Unlike general-purpose databases, TSDBs are designed to handle massive write throughput and complex aggregations (averages, sums, trends) over specific time intervals, which is critical for monitoring, finance, and IoT.
🏗️ Core Principles & Characteristics
- Time-Centric Schema: Data points consist of a timestamp, a set of tags/labels (metadata), and one or more numerical values (metrics).
- Immutable Data: Typically, data points are appended rather than updated.
- Downsampling: The process of aggregating high-resolution data into lower-resolution buckets (e.g., converting 1-second data into 1-minute averages) to save space.
- Retention Policies: Automatic deletion or archival of old data based on age (e.g., keep 1-minute data for 30 days, 1-hour data for 1 year).
⚖️ Trade-offs: Pros & Cons
Pros
- Write Efficiency: Optimized for high-velocity "append-only" workloads, often using LSM-Trees (Log-Structured Merge-Trees).
- Storage Compression: Can achieve 90%+ compression rates because time-series data is often repetitive or predictable.
- Specialized Queries: Built-in functions for time-windowing, rate of change, and moving averages.
Cons
- Limited Relational Power: Not suitable for complex JOINS between non-time-series datasets.
- Tag Cardinality: "High Cardinality" (too many unique tag combinations, like
user_id) can crash or significantly slow down many TSDBs. - Overwrite Costs: Updating a historical data point is often very expensive or unsupported.
🌍 Real-World Implementation
- DevOps & Infrastructure: Using Prometheus to monitor CPU, RAM, and error rates of Kubernetes pods.
- Financial Services: Using Kdb+ or TimescaleDB to store and analyze "Tick Data" (stock price changes every millisecond).
- IoT & Industrial: Using InfluxDB to collect sensor data from thousands of wind turbines or smart meters.
- Application Performance (APM): Monitoring request latencies and throughput in real-time.
💡 Interview "Gotchas" & Tips
- The Cardinality Trap: Always mention the danger of high cardinality labels. Explain that you shouldn't use
session_idoruser_idas tags in a system like Prometheus. - Pull vs. Push: Know the difference. Prometheus pulls (scrapes) data; InfluxDB and Graphite usually receive pushed data.
- LSM-Trees: Mention that TSDBs often use LSM-Trees to handle the high write volume by batching updates in memory before flushing to disk.
📐 Suggested Architecture Primitives
- Prometheus: The standard for pull-based cloud-native monitoring.
- InfluxDB: A purpose-built, high-performance TSDB.
- TimescaleDB: For when you need SQL power and relational joins along with time-series data.
- Grafana: The primary visualization tool for TSDB data.
Canvas