SYS ARCHITECTLearning Platform

Phase 1

  • Interview Framework
  • Back-of-the-Envelope Estimation
  • ACID
  • CAP
  • Consistency vs Integrity
  • Idempotency
  • NFR
  • SPOF
  • Stateless vs Stateful
  • SOLID
  • SRP
  • OCP
  • LSP
  • DIP vs LSP
  • Clear Architecture Principles
  • 12-Factor App

    • Phase 2

  • SQL vs NoSQL
  • Normalization vs Denormalization
  • Partition vs Sharding
  • Sharding
  • Partition Tolerance
  • Serializable Transactions
  • MVCC
  • Optimistic vs Pessimistic Locking
  • Row Level Locking
  • Distributed Locks
  • Redis Lock
  • Replication
  • Primary Replica Setup
  • Eventual Consistency
  • Data Consistency
  • Quorum
  • Write Ahead Logs
  • Redo Logs
  • DB Durability
  • Data Recovery
  • Two Phase Commit
  • Saga Pattern
  • Cassandra
  • S3
  • HDFS
  • Timeseries DB

    • Phase 3

  • Caching
  • Cache Eviction Policies
  • Hot Keys
  • Hot Reads and Writes
  • Latency
  • Latency vs Throughput
  • Tail Latency
  • Low Latency Patterns
  • Backpressure
  • Sudden Traffic Spike
  • Buffer Burst

    • Phase 4

  • Replication
  • Consistent Hashing
  • Leader Election
  • Distributed Locking
  • Fault Tolerance
  • Resiliency
  • Raft Consensus
  • Paxos
  • Event Driven Architecture
  • Event Sourcing
  • Outbox Pattern
  • CQRS

    • Phase 5

  • How HTTP Works
  • HTTP vs HTTPS
  • TCP vs UDP
  • TLS
  • Long Polling vs WebSockets
  • Server Sent Events
  • Load Balancer
  • L4 vs L7
  • Load Balancer vs Reverse Proxy vs API Gateway
  • CDN
  • Geo DNS

    • Phase 6

  • JWT
  • OAuth2
  • Opaque vs JWT Token
  • TLS
  • OWASP
  • Secrets Management

    • Phase 7

  • Creating Docker Image
  • Kubernetes ConfigMap
  • Kubernetes Ingress
  • Persistent Volume
  • Prometheus and Grafana

    • Phase 8

  • Monolith vs Microservices
  • API Gateway
  • Unique ID Generator
  • Retry Mechanism
  • Deployment Strategy
  • Rate Limiting

    • Phase 9

  • URL Shortener
  • Chat System
  • News Feed
  • Web Crawler
  • Notifications Design
  • System Design Problems
    • Settings
    Theory

    Web Crawler Design

    📋 Overview

    A web crawler (also known as a spider or bot) is a system that methodically browses the World Wide Web to index content, typically for search engines. Designing a crawler at scale involves managing billions of URLs while respecting website policies and maintaining high throughput.

    🏗️ Core Principles & Characteristics

    • The Frontier: A prioritized queue of URLs that need to be visited.
    • Politeness Policy: Respecting robots.txt and ensuring the crawler doesn't DDoS a site with too many simultaneous requests.
    • Deduplication: Preventing the crawler from getting stuck in "Spider Traps" or re-visiting the same content using Bloom Filters or Hash Sets.
    • DNS Resolution: Pre-fetching and caching IP addresses to avoid the DNS bottleneck.

    ⚖️ Trade-offs: Pros & Cons

    Pros

    • Data Enrichment: Enables search engines, price comparison tools, and data analytics.
    • Automation: Can monitor the entire web for changes without human intervention.
    • Global Reach: Provides a comprehensive view of the internet's structure (PageRank).

    Cons

    • Legal/Ethical Risks: Must balance data collection with copyright and privacy laws.
    • Computational Expense: Requires massive bandwidth, storage, and compute to process and store HTML content.
    • Dynamic Content: Traditional crawlers struggle with JavaScript-heavy sites (SPA), often requiring expensive "Headless Browsers" (Puppeteer/Playwright) to render the page.

    🌍 Real-World Implementation

    • Search Engines: Googlebot and Bingbot indexing the web for search.
    • E-commerce: Scrapers for price monitoring and competitor analysis.
    • Copyright Protection: Bots that scan for pirated content or trademark infringements.
    • Archiving: The Wayback Machine (Internet Archive) creating snapshots of the web for history.

    💡 Interview "Gotchas" & Tips

    • DFS vs. BFS: In interviews, always suggest BFS (Breadth-First Search). DFS can get stuck in a single site's deep hierarchy forever.
    • Spider Traps: Mention how to detect infinite URL loops (e.g., calendar.com/2023/jan/01, calendar.com/2023/jan/02...).
    • Bloom Filters: Use these for the "Visited URLs" set to save massive amounts of RAM.
    • Distribution: Explain how to shard the Frontier based on the hash of the domain name to ensure "Politeness" (only one thread hits a specific domain at a time).

    📐 Suggested Architecture Primitives

    • Message Queues (Kafka/RabbitMQ): To manage the massive URL Frontier.
    • Bloom Filters: For memory-efficient deduplication.
    • NoSQL (HBase/Cassandra): To store the raw HTML or extracted "Web Graph."
    • Headless Browsers: For crawling React/Vue-based dynamic websites.
    Canvas