SYS ARCHITECTLearning Platform

Phase 1

  • Interview Framework
  • Back-of-the-Envelope Estimation
  • ACID
  • CAP
  • Consistency vs Integrity
  • Idempotency
  • NFR
  • SPOF
  • Stateless vs Stateful
  • SOLID
  • SRP
  • OCP
  • LSP
  • DIP vs LSP
  • Clear Architecture Principles
  • 12-Factor App

    • Phase 2

  • SQL vs NoSQL
  • Normalization vs Denormalization
  • Partition vs Sharding
  • Sharding
  • Partition Tolerance
  • Serializable Transactions
  • MVCC
  • Optimistic vs Pessimistic Locking
  • Row Level Locking
  • Distributed Locks
  • Redis Lock
  • Replication
  • Primary Replica Setup
  • Eventual Consistency
  • Data Consistency
  • Quorum
  • Write Ahead Logs
  • Redo Logs
  • DB Durability
  • Data Recovery
  • Two Phase Commit
  • Saga Pattern
  • Cassandra
  • S3
  • HDFS
  • Timeseries DB

    • Phase 3

  • Caching
  • Cache Eviction Policies
  • Hot Keys
  • Hot Reads and Writes
  • Latency
  • Latency vs Throughput
  • Tail Latency
  • Low Latency Patterns
  • Backpressure
  • Sudden Traffic Spike
  • Buffer Burst

    • Phase 4

  • Replication
  • Consistent Hashing
  • Leader Election
  • Distributed Locking
  • Fault Tolerance
  • Resiliency
  • Raft Consensus
  • Paxos
  • Event Driven Architecture
  • Event Sourcing
  • Outbox Pattern
  • CQRS

    • Phase 5

  • How HTTP Works
  • HTTP vs HTTPS
  • TCP vs UDP
  • TLS
  • Long Polling vs WebSockets
  • Server Sent Events
  • Load Balancer
  • L4 vs L7
  • Load Balancer vs Reverse Proxy vs API Gateway
  • CDN
  • Geo DNS

    • Phase 6

  • JWT
  • OAuth2
  • Opaque vs JWT Token
  • TLS
  • OWASP
  • Secrets Management

    • Phase 7

  • Creating Docker Image
  • Kubernetes ConfigMap
  • Kubernetes Ingress
  • Persistent Volume
  • Prometheus and Grafana

    • Phase 8

  • Monolith vs Microservices
  • API Gateway
  • Unique ID Generator
  • Retry Mechanism
  • Deployment Strategy
  • Rate Limiting

    • Phase 9

  • URL Shortener
  • Chat System
  • News Feed
  • Web Crawler
  • Notifications Design
  • System Design Problems
    • Settings
    Theory

    Outbox Pattern & Operations Dashboard

    📋 Overview

    In distributed microservices, ensuring transactional integrity between a database update and an event publication (e.g., to Kafka) is a classic "Dual-Write" problem. The Transactional Outbox Pattern provides a reliable way to ensure that events are eventually published if and only if the database transaction succeeds. Complementary to this, an Operations (Ops) Dashboard provides the visibility needed to monitor these flows and manually intervene when automated processes hit edge cases.

    🏗️ Core Principles & Characteristics

    1. Transactional Outbox Pattern

    • Atomic Commitment: Business data and the event payload are saved to the same database within a single ACID transaction.
    • Relay Mechanism: A separate "Message Relay" (Poller or CDC) reads the outbox table and publishes messages to the broker.
    • At-Least-Once Delivery: Guarantees that no message is lost, though consumers must handle potential duplicates (idempotency).

    2. Operations Dashboard

    • Visibility: Real-time monitoring of PENDING vs. SENT messages.
    • Manual Intervention: Ability to trigger retries for failed events or Dead Letter Queue (DLQ) messages.
    • Audit Trail: Logging of who performed what action on a specific record.

    ⚖️ Trade-offs: Pros & Cons

    Outbox Pattern

    • Pros: Eliminates the "Dual-Write" risk; high reliability; decoupling of database and message broker availability.
    • Cons: Increases database load (extra writes); requires additional infrastructure for the relay (poller or CDC).

    Ops Dashboard

    • Pros: Reduces Mean Time to Recovery (MTTR); empowers non-developers to fix data inconsistencies.
    • Cons: Security risk if not properly gated (RBAC); adds complexity to the admin tech stack.

    🌍 Real-World Implementation

    • CDC (Change Data Capture): Using Debezium to tail the database transaction log (Binlog for MySQL, WAL for Postgres) and automatically push outbox entries to Kafka.
    • Polling Publisher: A scheduled Spring Boot task using @Scheduled to query the outbox table every 500ms.
    • Tech Stack: ELK (Elasticsearch, Logstash, Kibana) for log aggregation; Prometheus/Grafana for metric dashboards; Retool or custom React/Angular for the manual action UI.

    💡 Interview "Gotchas" & Tips

    • The Poller vs. CDC Debate: Polling is easier to implement but adds latency and DB overhead. CDC is high-performance and "near real-time" but requires complex infra like Kafka Connect.
    • Scaling the Poller: If you have multiple instances of your service, ensure only one instance polls the outbox at a time (using a distributed lock or a specific "leader" instance).
    • Eventual Consistency: Acknowledge that while the Outbox pattern ensures the message gets to the broker, the system as a whole is "Eventually Consistent."

    📐 Suggested Architecture Primitives

    • Storage: PostgreSQL for ACID-compliant outbox storage.
    • Relay: Debezium / Kafka Connect for robust CDC.
    • Observability: Correlation IDs spanning from the DB transaction to the Kafka message to the dashboard.
    • Security: OAuth2/OIDC for securing the Ops Dashboard.
    Canvas