Everything you need to know about building scalable, reliable, and maintainable
systems — from fundamentals to real-world architecture, explained simply with analogies, examples, and interview tips.
↓ let's begin
📐 Quick Reference Tables
Before diving in, here are two tables you'll use constantly during capacity estimation and back-of-the-envelope calculations.
System design is the process of defining the architecture, components, modules,
interfaces, and data flow of a system to meet specific requirements.
It involves designing how individual parts (databases, APIs, front-end, back-end, caching layers, etc.) will interact and scale.
🏙️
Think of building a city
You don't just build roads randomly. You plan roads, water supply, power, zoning, and emergency systems
— just like designing components of a scalable system.
Why it matters: System design ensures applications are scalable, reliable, maintainable, and efficient
— critical for high-traffic systems like Google, Netflix, or Uber.
Core Concepts
🏗️
Architecture
Client-server, microservices, monoliths, event-driven, etc.
Netflix: Designed for high throughput and fault tolerance, using microservices, distributed databases, and CDNs.
Uber: Combines real-time location data, message queues, and distributed storage to handle millions of concurrent users.
Pros & Cons / Trade-offs
✅ Advantages
Scalability and high availability. Easier maintenance and feature addition.
❌ Challenges
Complex architecture increases development time. Harder debugging in distributed systems.
Monolith vs. Distributed
Monoliths
Simpler, but harder to scale.
Distributed Systems
Complex, but scale efficiently.
Design Thinking — Step-by-Step
Understand Requirements
Clarify functional requirements (what the system should do) and non-functional requirements (scale, latency, reliability, cost).
Example: "Users can upload, view, and comment on videos." / "Handle 10M users with <200ms response time."
Define High-Level Architecture
Identify key components: frontend, backend APIs, database, cache, load balancer, etc. Draw a rough diagram showing how data flows between them.
Choose Technologies & Data Model
SQL vs NoSQL, message queues, caching (Redis), CDN, etc. Explain your reasoning — e.g., "Using Redis to cache frequent data to reduce DB load."
Address Scalability & Reliability
Discuss horizontal scaling, load balancing, replication, sharding, caching, and CDNs. Mention how you'd handle failures and retries.
Discuss Trade-offs
Every choice has pros and cons — highlight one. E.g., "Using NoSQL improves scalability but sacrifices strong consistency."
"I'd start with requirements, then move to components, scaling, and trade-offs."
⚡ Example: "Design a URL shortener like bit.ly"
Functional: shorten long URLs, redirect to original.
Non-functional: handle millions of requests, low latency.
Components: API server, database, cache, load balancer.
Scaling: use hash-based IDs, caching for popular URLs, replication for read-heavy load.
Trade-offs: collisions vs storage efficiency.
Cache is required to reduce database load and latency, especially since URL redirection is a highly read-heavy operation. Popular links are served directly from memory instead of hitting the database every time.
Key Takeaways
System design is about building scalable and reliable software architectures.
Focus on trade-offs (latency, cost, consistency, etc.).
Always address both functional and non-functional requirements.
2. Horizontal vs. Vertical Scaling
Choosing the right scaling method impacts cost, complexity, and scalability.
Vertical scaling upgrades one machine; horizontal scaling adds more machines behind a load balancer.
Vertical Scaling: Increasing resources (CPU, RAM) of a single machine. "Scaling up" → Bigger machine.
Horizontal Scaling: Adding more machines to distribute the load. "Scaling out" → More machines behind a load balancer.
Elastic Scaling: Dynamic adjustment based on demand.
Real-World Examples
Vertical: Upgrading a database server with more RAM.
Horizontal: Netflix scaling microservices across thousands of servers.
🚛
Truck Analogy
Vertical: Buying a bigger truck.
Horizontal: Adding more delivery trucks.
Pros & Cons Comparison
Type
Pros
Cons
Vertical
Simple, no code changes
Hardware limits, downtime
Horizontal
Infinite scale, high availability
Complex architecture, requires load balancing
Vertical = Simpler but Finite
There's a physical limit to how much one machine can handle. If that server fails, your system goes down. Simple to implement — just upgrade your server. No code changes usually required.
Horizontal = Complex but Limitless
Better fault tolerance — if one server fails, others handle traffic. Can handle millions of users by adding more nodes. Ideal for distributed, cloud-based systems. But more complex to design — needs load balancing, data partitioning, synchronization.
Step-by-Step Example
App slows due to traffic.
Your application starts experiencing performance issues as user count grows.
Try vertical scaling (upgrade CPU).
Add more power to the existing server — increase CPU, RAM, or storage.
If limits hit, move to horizontal scaling with load balancing.
Distribute data across multiple servers and use a load balancer — that's horizontal scaling.
🎯 Interview Tip
Say vertical first, then horizontal for long-term scale.
"I'd start with vertical scaling for simplicity when traffic is small, but switch to horizontal scaling for long-term growth and reliability."
✅ Example
"Initially, you can just upgrade your database server. But once users grow, you'd distribute data across multiple servers and use a load balancer — that's horizontal scaling."
Key Takeaways
Horizontal scaling = best for large-scale distributed systems.
Vertical scaling = quick and easy but has limits.
3. What is Capacity Estimation?
Capacity estimation is the process of predicting the resources (CPU, RAM, storage, bandwidth) needed for a system based on expected traffic and usage patterns.
Why it matters: It ensures your system can handle load without over-provisioning or crashing.
Core Concepts
📊
Traffic Estimation
Requests per second (RPS).
💾
Storage Needs
Based on data growth and retention.
🌐
Bandwidth
Based on content size × number of requests.
🖥️
Compute
Based on CPU usage per request.
Real-World Examples
Instagram estimates photo uploads per second and allocates storage accordingly.
YouTube estimates video storage growth to plan server capacity.
Pros & Cons
✅ Benefits
Prevents system overload. Helps cost optimization.
❌ Challenges
Hard to predict early-stage usage. Wrong estimates lead to under- or over-provisioning.
🍽️
Restaurant Analogy
Like estimating the number of seats in a restaurant based on expected daily customers.
Step-by-Step Walkthrough
Estimate users and growth rate.
How many users do you expect? What's the growth trajectory?
Calculate read/write requests per second.
Suppose 1 million users, 10% active at a time → 100,000 active users. If each sends 1 request per second → 100K requests/second.
Compute storage needs.
Each user uploads 5 photos/day (~2MB each). 1M users × 10MB/day = 10TB/day storage needed. Include metadata (likes, comments), and apply compression if used.
Estimate network and compute usage.
Network: If average image is 2MB and 100K users upload per second → 200GB/sec bandwidth (use CDN and caching to reduce). Compute: If 1 server handles 1K requests/sec, and you expect 100K requests/sec → need 100 servers, plus replicas for redundancy.
Where does "1 server handles ~1K RPS" come from?
It's a rough industry rule of thumb for a typical web server handling API requests. The actual number depends on your hardware, language, and workload — a lightweight Go server might handle 10K+ RPS, while a heavy Django app might handle 200-500. In interviews, 1K is a safe starting assumption unless you have a reason to think otherwise.
Add buffer (~20–30%) for safety.
Always add a safety buffer for unexpected traffic spikes (like viral posts or new feature launches).
🎯 Interview Tip
Break it into four areas: traffic, storage, network, and compute — then add a safety buffer.
"I'd estimate traffic, storage, network, and compute needs based on expected users and activity, then add a safety margin to handle peak loads and growth."
Latency Numbers Every Engineer Should Know (Jeff Dean):
Operation
Latency
Comparison
L1 cache reference
~0.5 ns
Blink of a thought
L2 cache reference
~7 ns
Main memory (RAM) reference
~100 ns
SSD random read
~16 μs
20x slower than RAM
HDD disk seek
~2-10 ms
100,000x slower than RAM
Send packet within same datacenter
~0.5 ms
Read 1 MB sequentially from SSD
~1 ms
Read 1 MB sequentially from HDD
~20 ms
Round trip within same datacenter
~0.5 ms
Cross-continent round trip
~150 ms
Why CDNs exist
Why this matters: These numbers are the foundation of all capacity estimation. When an interviewer asks "why use a cache?", the answer is: RAM is 100,000x faster than disk. When they ask "why use a CDN?", the answer is: cross-continent round trips take 150ms. Memorize these — they turn hand-wavy estimates into grounded calculations.
Key Takeaways
Capacity estimation avoids outages and inefficiencies.
Always plan for peak traffic and future growth.
4. What is HTTP?
HTTP (HyperText Transfer Protocol) is the foundational protocol for communication between clients (browsers) and servers on the web.
It defines how messages are formatted, transmitted, and responded to. Think of HTTP like sending a postcard — anyone who handles the postcard during delivery (like routers, ISPs, or hackers on the network) can read what's written on it because it's all in plain text.
HTTP operates on port 80 by default.
It transfers data in plain text, meaning the content is not protected.
Because it is unencrypted, attackers can intercept or modify the data easily.
It is still used for non-sensitive websites where security is not a concern (like public blogs or info pages).
Core Concepts
📬
Methods
GET, POST, PUT, DELETE, PATCH.
🔢
Status Codes
200 (OK), 404 (Not Found), 500 (Server Error).
📝
Headers
Metadata (e.g., content-type, caching rules).
🔒
HTTPS
Secure version of HTTP (TLS/SSL encryption).
Stateless: Each request is independent.
Real-World Examples
When you open Netflix.com → Browser sends HTTP GET request → Server returns HTML.
API requests in mobile apps use HTTP for data exchange.
Pros & Cons
✅ Advantages
Simple and widely supported. Human-readable and extensible.
❌ Limitations
Statelessness means extra overhead for session management. Less efficient for real-time communication (WebSockets preferred).
HTTP vs HTTPS
HTTPS stands for HyperText Transfer Protocol Secure. It is an extension of HTTP with added security through encryption. If HTTP is a postcard, HTTPS is like a locked envelope.
Aspect
HTTP
HTTPS
Full Form
HyperText Transfer Protocol
HyperText Transfer Protocol Secure
Port
80
443
Security
No encryption, plain text
Encrypted using SSL/TLS
Analogy
Like a postcard anyone can read
Like a sealed envelope only the receiver can open
User Trust
"Not Secure" warning in browsers
Padlock icon in the browser
Performance
HTTP/1.1 (basic speed)
Supports HTTP/2 for faster loading
SEO Ranking
No ranking advantage
Gets a boost in search engine ranking
Use Case
Non-sensitive websites (blogs, news)
Banking, e-commerce, login portals, sensitive data
Nuance: The table implies HTTPS is faster than HTTP — this is misleading. HTTPS actually adds latency (TLS handshake). What makes it faster in practice is HTTP/2 multiplexing, which browsers only enable over HTTPS. HTTP/2 is a protocol feature, not an encryption feature. Also note: HTTP/2 does NOT require HTTPS by specification — it's browsers that mandate TLS for HTTP/2 connections.
Request Lifecycle (Step-by-Step)
Client sends an HTTP request to server.
DNS resolves domain → IP.
Server processes request and sends response.
Client renders response (HTML, JSON, etc.).
✉️
Mail Service Analogy
HTTP is like a mail service — you send a letter (request), the server replies with a letter (response).
🎯 Interview Tip
"What happens when you type a URL in the browser?"
Mention DNS → TCP handshake → HTTP request → server response → render.
Key Takeaways
HTTP powers nearly all web communication.
It's stateless, simple, and extensible.
HTTPS is essential for security in modern systems.
5. What is the Internet TCP/IP Stack?
The TCP/IP stack is the foundational networking model that powers data communication over the internet.
It defines how data is packaged, transmitted, routed, and received between systems.
Why it matters: System design heavily relies on understanding network flow — from browser requests to backend responses — all happening through this layered model.
The 4 Layers
Layer
Name
Responsibility
Example Protocols
4
Application
User-level communication
HTTP, FTP, SMTP
3
Transport
Reliable delivery & flow control
TCP, UDP
2
Internet
Routing & addressing
IP, ICMP
1
Link/Network Access
Physical transmission
Ethernet, Wi-Fi
How the Layers Work Together During a Real Request
When you load a webpage: HTTP (Application) formats your request → TCP (Transport) breaks it into numbered segments → IP (Network) routes each segment → Ethernet (Link) physically transmits the bits on the wire. On receiving, the process reverses — bits → packets → segments → your webpage.
TCP (Transmission Control Protocol): Reliable, connection-oriented. Ensures ordered and error-free data.
UDP (User Datagram Protocol): Fast, connectionless, but no delivery guarantees.
IP (Internet Protocol): Handles addressing and routing.
Real-World Examples
HTTP over TCP/IP: Every web page request travels through these layers.
Video streaming: Uses UDP for speed in real-time delivery.
Email: SMTP runs on TCP for guaranteed delivery.
Pros & Cons
✅ Advantages
Modular and flexible. Well-tested and standardized.
❌ Limitations
Adds overhead (especially TCP). UDP trades reliability for speed.
TCP vs. UDP
Feature
TCP
UDP
Reliability
Guaranteed
Not guaranteed
Ordering
Ordered
Unordered
Use case
Web, email
Gaming, streaming
TCP's 3-Way Handshake
Before any data can flow over TCP, a connection must be established using a 3-way handshake:
(1) SYN: Client sends SYN ("I want to connect").
(2) SYN-ACK: Server replies SYN-ACK ("Got it, I'm ready too").
(3) ACK: Client sends ACK ("Great, let's go").
Only after these 3 steps can data flow. It's like calling someone: you dial (SYN), they say "hello?" (SYN-ACK), you say "hi, it's me" (ACK) — now you can talk.
HTTP Request Flow (Step-by-Step)
DNS resolves domain to IP.
TCP handshake establishes a connection.
HTTP request sent via TCP.
IP routes packets to the server.
Response travels back through layers.
📮
Postal Services Analogy
IP is the address on the envelope.
TCP ensures delivery order.
HTTP is the content inside.
🎯 Interview Tip
"Explain the layers of the TCP/IP stack." / "Why use UDP for video streaming?"
Map protocols to layers when answering.
Key Takeaways
All internet communication uses the TCP/IP stack.
TCP ensures reliability, UDP ensures speed.
Understanding these layers is vital for latency optimization.
6. What Happens When You Enter Google.com?
This classic interview question tests your understanding of networking, browser internals, DNS, caching, and backend systems. It's the end-to-end life cycle of a web request.
The steps involve DNS resolution, TCP connection, HTTP request/response, rendering, and caching.
The Complete Walkthrough
DNS Lookup
Browser checks cache → OS cache → DNS resolver → authoritative name server. Gets IP address of Google's server.
TCP Handshake
SYN → SYN-ACK → ACK establishes a connection (3-way handshake). Your computer says "Hey Google, can we talk?" Google replies "Yes!" Your computer confirms.
TLS Handshake (if HTTPS)
Encryption keys are exchanged for a secure connection. Because it's HTTPS, your browser and Google agree on encryption keys so nobody can spy on your data.
HTTP Request Sent
Browser sends a GET request to https://www.google.com.
Load Balancer & Routing
Request reaches Google's load balancer, routed to the closest, healthiest server.
Server Processing
Backend services fetch data, query databases, and render HTML.
Response Returned
HTTP response travels back via TCP/IP.
Browser Rendering
HTML parsed → CSS/JS fetched → DOM constructed → Page displayed. Your browser reads the HTML (structure), loads CSS (design), runs JavaScript (interactivity). Finally, you see the Google homepage appear.
The complete lifecycle of a web request: DNS lookup, TCP/TLS handshake, server processing, and browser rendering.
🌍 Real-World Detail
Google uses Anycast DNS, global CDNs, Edge caching, and load balancers to deliver results under ~200 ms worldwide.
💡
Simple Analogy
It's like:
Looking up a friend's address (DNS)
Calling them to confirm (TCP/TLS)
Asking for something (HTTP request)
Getting it delivered (response)
Reading or using it (rendering)
DNS finds the address → TCP connects → TLS secures → HTTP requests → Browser shows the page.
🎯 Interview Tip
"Walk me through what happens when you type a URL."
Break into 5 stages: DNS → TCP/TLS → HTTP → Backend → Rendering.
Key Takeaways
Understanding the full flow shows you grasp networking, protocols, caching, and web infrastructure.
Every system design connects back to these fundamental steps.
7. What are Relational Databases?
Relational databases (RDBMS) store data in tables with rows and columns, with relationships defined using keys.
Why it matters: They're the backbone of structured, transactional systems — banking, e-commerce, CRM, etc.
MySQL, PostgreSQL, Oracle, SQL Server used by companies like Amazon, Facebook, and banking systems.
Pros & Cons
✅ Advantages
Strong consistency and data integrity. Powerful query language (SQL).
❌ Limitations
Hard to scale horizontally. Less flexible for unstructured data.
📚
Well-Organized Library Analogy
RDBMS is like a well-organized library — every book is in a defined place and cross-referenced.
ACID Properties — Deep Dive
What is a Transaction?
A transaction is a group of database operations that must all succeed or all fail together — like transferring money requires both "deduct from Account A" and "add to Account B" to happen as one unit. If either step fails, the entire transaction is rolled back so the database stays in a valid state.
ACID stands for Atomicity, Consistency, Isolation, and Durability — the key properties that make database transactions reliable.
⚛️
Atomicity → "All or nothing."
If a transaction has multiple steps (like transferring money), either all steps happen, or none do.
✅
Consistency → "Valid data before and after."
The database must move from one valid state to another — it can't break rules or constraints.
🔒
Isolation → "No interference between transactions."
Even if many people are using the system at once, each transaction behaves as if it's the only one happening.
💎
Durability → "Data is saved permanently."
Once a transaction is complete, it's safely stored — even if the system crashes.
✅ ACID Example
When you transfer ₹500 from your account to someone else's, either both the debit and credit happen or neither — that's ACID in action.
🎯 Interview Tip
"When would you choose SQL over NoSQL?"
I'd choose SQL when my data is structured, relationships matter, and I need strong consistency. I'd choose NoSQL when my data is unstructured or changing quickly, and I need high scalability.
SQL vs NoSQL Comparison
Aspect
SQL (MySQL, PostgreSQL)
NoSQL (MongoDB, Cassandra)
Data Structure
Structured in tables (rows, columns)
Flexible (JSON, key-value, etc.)
Consistency Model
Follows ACID properties
Follows BASE (Basically Available, Soft state, Eventual consistency)
Best For
Financial or transactional apps
Real-time apps, analytics, or large-scale data
Scaling
Vertical scaling (bigger server)
Horizontal scaling (more servers)
Important correction: This is an oversimplification. Modern distributed SQL databases like Google Spanner, CockroachDB, TiDB, and Vitess scale horizontally while maintaining ACID and relational semantics. The accurate framing: SQL databases traditionally scaled vertically, but "NewSQL" databases have eliminated this limitation. Saying "SQL can only scale vertically" in an interview at Google (who built Spanner) will get you immediately challenged.
✅ Decision Example
For a banking system, I'd use SQL because consistency is critical.
For a social media app, I'd use NoSQL because data is huge and changes frequently.
ACID vs BASE
ACID → SQL (Relational Databases)
Ensures reliability and consistency of transactions. Perfect for systems where data correctness is critical (like banking). ACID systems never allow wrong data.
BASE → NoSQL Databases
More relaxed to allow scalability and availability. BASE systems accept temporary wrong data… as long as the system is fast, scalable, and always available. BASE is basically telling you: "We are okay with temporary wrong data."
BASE stands for:
Basically Available → system is mostly available even if parts fail.
Soft state → data may not be instantly consistent.
Eventual consistency → data becomes consistent after some time.
BASE Deep Dive with Examples
1️⃣ Basically Available — Instagram Feed Example
One server crashes. You still open Instagram. You still see posts. Maybe some posts are missing, some likes not updated. But app does NOT crash.
2️⃣ Soft State — WhatsApp "Last Seen" Example
You go offline. Your "last seen" doesn't update instantly everywhere. Different servers may show "online" vs "last seen 2 min ago." Data is in a temporary inconsistent state.
Think of it as "the system's data is in flux" — unlike a hard disk where data stays put, soft state means data may be changing as replicas catch up with each other.
You like a post. You see 101 likes. Your friend sees 100 likes (for a few seconds). After a few seconds → Everyone sees 101. That's eventual consistency.
Full Real-World Example: Amazon product stock
Only 1 iPhone left. 2 users try to buy at the same time.
With Strong Consistency (SQL mindset): Only 1 user succeeds. Other gets "Out of stock" instantly. Always correct, but slower + harder to scale.
With BASE (NoSQL mindset): User A buys → one server updates stock = 0. User B hits another server → still sees stock = 1 → User B also buys. Now system is temporarily wrong. But then: system fixes it later — cancels one order, sends "Out of stock" message, refunds money. Eventually everything becomes correct.
Why companies choose BASE: Because at scale (millions of users), enforcing strong consistency makes the system slow, requires too much coordination, and is hard to scale globally. With BASE: faster responses, system never goes down, easy to scale. Trade-off = temporary inconsistency.
🎯 Interview One-Liner
"BASE prioritizes availability and scalability by allowing temporary inconsistencies, where the system eventually becomes consistent over time."
CAP Theorem
What is a Network Partition?
A network partition happens when two parts of your system can't communicate — like if the cable between two data centers gets cut. It doesn't mean a server crashed; it means servers can't talk to each other temporarily. Understanding this is essential before diving into CAP, because the entire CAP trade-off revolves around what your system does during a partition.
When you talk about NoSQL, it's better to also mention the CAP theorem, which explains why BASE exists.
CAP stands for:
🔗
C — Consistency
All nodes see the same data at the same time.
🟢
A — Availability
Every request gets a response, even if it's not the latest.
🔌
P — Partition Tolerance
System continues working even if parts of it can't communicate.
According to CAP, a distributed system can guarantee only two of these three at a time.
⚠️ Important Correction (Interview Gold)
❌ It does NOT mean: "You pick any 2 and ignore the third"
✅ It actually means: "Partition tolerance is mandatory, so you choose between Consistency and Availability."
Real Scenario: Mumbai vs Delhi Servers
You have 2 servers: Server A (Mumbai) and Server B (Delhi). Both store the same data. Suddenly, network breaks between them. A and B cannot talk to each other. This is called Partition (P).
A user updates data on Server A: Balance = ₹500. But Server B still has old data → ₹1000.
Now a user hits Server B. What should it do?
Option 1: Choose Consistency (CP): Server B says "I'm not sure about latest data, so I won't respond." Some requests fail, but data is always correct. You sacrifice Availability.
Option 2: Choose Availability (AP): Server B says "I'll respond anyway." User gets a response, but data might be outdated. You sacrifice Consistency.
Real-World CAP Mapping
🛒 Amazon / Flipkart — AP System
Choose Availability + Partition tolerance. Show slightly stale data if needed. Because app should never go down.
🏦 Banking Systems — CP System
Choose Consistency + Partition tolerance. Reject request if unsure. Because wrong data = disaster.
🎯 CAP One-Line Intuition
CAP = "What do you sacrifice when the network breaks?"
"During a network partition, a system must choose between consistency and availability, because ensuring both simultaneously would require nodes to communicate, which is not possible during a partition."
In summary:
SQL databases → ACID
NoSQL databases → BASE (and designed around CAP trade-offs)
🎯 Interview Summary
"SQL databases follow ACID properties for strong consistency and reliability. NoSQL databases are often BASE-compliant — they trade strict consistency for better availability and partition tolerance as per the CAP theorem."
Key Takeaways
RDBMS ensures strong data integrity and supports complex queries.
Trade scalability for consistency and structure.
8. What are Database Indexes?
An index is a data structure that speeds up data retrieval operations on a database table.
It works like a table of contents in a book.
Core Concepts
B-Tree Index: Most common, balanced and sorted.
Hash Index: Fast lookups for equality.
Composite Index: Index on multiple columns.
B-Tree index: the root and internal nodes guide the search; leaf nodes point to actual data rows. Lookup is O(log n).
B-Tree Search Walkthrough
Searching for value 42: Start at root [30|60] → 42 is between 30 and 60, so go to the middle child → scan the internal node [40|50] → 42 is between 40 and 50, so go to the middle child → reach the leaf node containing 42 → follow the pointer to the actual data row.
Only 3 nodes visited instead of scanning millions of rows! This is why B-Tree lookups are O(log n) — each level of the tree eliminates a large fraction of the remaining data.
Real-World Examples
E-commerce product search uses indexes on product names, categories.
Social media uses indexes on user IDs for quick profile access.
Pros & Cons
✅ Advantages
Drastically improves read performance. Enables fast range queries and joins.
❌ Limitations
Slower writes due to index maintenance. Extra storage overhead.
Performance Comparison
With Index: O(log n)
Fast retrieval using balanced tree lookup.
Without Index: O(n)
Full table scan — checks every row.
💡 Implementation Insight
Use indexes selectively on frequently queried columns — not every column needs one.
🎯 Interview Tip
"How do indexes improve query performance?" / "When can indexes hurt performance?"
Key Takeaways
Indexes speed up reads but slow down writes.
Use them where read-heavy queries dominate.
9. What are NoSQL Databases?
NoSQL = "Not Only SQL" — a class of databases optimized for scalability, flexibility, and performance with unstructured or semi-structured data.
Note: See the correction above — modern distributed SQL (Spanner, CockroachDB) can also scale horizontally.
🎯 Interview Tip
"When would you prefer NoSQL over SQL?"
"I'd choose NoSQL when I need horizontal scalability, schema flexibility, and can trade strict consistency for speed and availability."
When to Choose NoSQL — 4 Reasons
1️⃣ Scalability
NoSQL databases are designed to scale horizontally, meaning we can add more servers to handle growing traffic easily. Example: Social media apps or e-commerce sites where data and users grow quickly.
2️⃣ Schema Flexibility
NoSQL allows dynamic or schema-less data, so we don't need to define strict tables and columns in advance. Example: Storing user profiles where different users may have different fields (like interests, bio, location).
3️⃣ Unstructured or Semi-Structured Data
It's ideal for JSON documents, key-value pairs, or large blobs of data that don't fit neatly into tables. Example: Chat messages, logs, sensor data, or social posts.
4️⃣ High Availability and Performance
NoSQL databases often prioritize availability and partition tolerance (based on CAP theorem), which makes them great for distributed systems where uptime is critical.
Example Summary
For a banking system, I'd use SQL for strong consistency.
For a social media or analytics app, I'd use NoSQL for scalability and flexibility.
Key Takeaways
NoSQL excels in scalability and flexibility but sacrifices consistency and complex querying.
10. What is a Cache?
A cache is a high-speed data storage layer that stores frequently accessed data to reduce latency and load on primary storage.
Cache hit returns data instantly; cache miss queries the database, stores the result in cache, then returns it to the client.
Real-World Examples
YouTube: Caches trending videos at edge servers.
Twitter: Caches user timelines for speed.
Pros & Cons
✅ Advantages
Reduces latency. Offloads database load.
❌ Limitations
Potential for stale data (the cache may contain old/outdated data instead of the latest value). Consistency challenges.
🧂
Kitchen Counter Analogy
Like storing commonly used spices on the kitchen counter instead of in the pantry.
🎯 Interview Tip
"How would you use caching in a read-heavy system?"
"In a read-heavy system, I'd use caching to reduce database load and improve response time. I'd place caches at multiple layers — from the client to the database — so frequently accessed data is served faster."
Caching Layers (Step-by-Step)
Client-Side Caching
The browser or mobile app can cache static resources like images, CSS, or JavaScript. Example: When you visit a site again, your browser doesn't re-download the logo or CSS file — it loads them from local storage.
CDN (Content Delivery Network)
A CDN caches static assets (like images, videos, or HTML) on edge servers close to users. Example: Cloudflare or Akamai delivers cached content quickly without hitting the origin server.
Application-Level Cache
The backend app can use an in-memory cache like Redis or Memcached to store frequent DB query results or computed data. Example: If a product catalog is requested often, store it in Redis so subsequent requests are served instantly.
Database Query Cache
Some databases have built-in caching (like MySQL query cache). But usually, it's better to use external caching layers for scalability and control.
🔄 Cache Invalidation
Important to mention: cached data must be updated when the underlying data changes.
Techniques: time-to-live (TTL), versioning, or manual invalidation.
🎯 Interview One-Line Summary
"I'd add caching at multiple layers — client, CDN, app, and DB — so that most reads are served from memory instead of hitting the database."
Key Takeaways
Caching is essential for performance.
Must balance freshness vs speed.
11. What is Thrashing?
Thrashing occurs when a system spends more time swapping data between memory and disk than executing tasks.
In distributed systems, it refers to excessive resource churn (e.g., cache evictions, context switching).
Thrashing at Two Levels
In system design, thrashing appears at two levels:
(1) OS-level: When RAM is full, the OS constantly swaps data between RAM and disk, spending more time swapping than working.
(2) Cache-level: When your cache is too small, items keep getting evicted and re-fetched, so the cache provides no benefit.
Both share the same root cause: too many items competing for too little space.
Interview note: In system design interviews, cache-level thrashing (endless eviction and re-fetching in your caching layer) is far more relevant than OS-level page swapping. If asked about thrashing, focus on cache sizing and eviction strategies.
🧠 What is Cache Eviction?
Cache has limited memory. When cache becomes full, it must remove some old data to make space for new data. That removal process is called Cache Eviction.
Core Causes
Insufficient memory.
Poor caching policies.
High load → frequent page faults.
Real-World Examples
💻 Web servers under heavy load swapping processes
Server RAM becomes full, so the OS keeps moving processes/data between RAM and disk instead of executing requests, causing major slowdown.
🗄️ Cache layers constantly invalidating and refilling
Cache becomes too small or overloaded, so data is repeatedly removed (evicted) and loaded again, wasting resources and increasing DB hits.
Symptoms
High latency: Requests take much longer because the system is busy managing memory/cache instead of processing work.
Low throughput: System handles fewer requests per second due to excessive swapping or cache churn.
Performance collapse: Overall system becomes extremely slow or unstable because resources are consumed by management overhead rather than useful processing.
Master line: "Thrashing occurs when the system spends more time managing resources (memory/cache/processes) than performing actual useful work."
Pros & Cons
✅ Pros
(None — it's always bad!)
❌ Cons
High latency, low throughput, performance collapse.
Technical View
Each process needs pages in RAM to execute.
When RAM is full, the OS swaps pages to disk (page faults).
If this happens excessively → system gets stuck just managing memory instead of running programs.
Simple Explanation
Imagine your computer is running too many apps at once. It keeps moving data in and out of memory to make space — that constant swapping slows everything down. That's thrashing.
How to Prevent Thrashing
Add More Physical Memory (RAM)
Reduces swapping between disk and memory.
Tune Multiprogramming Level
Limit how many processes run at once to match available memory.
Use Caching Wisely
Keep frequently used data in faster memory (RAM or cache) to reduce I/O operations.
Optimize Load / Database Queries
Reduce unnecessary memory usage in apps and queries.
Monitor System Performance
Use tools to detect high page fault rates and fix before it escalates.
📖
Book Analogy
Like flipping through a huge book repeatedly to find the same page instead of bookmarking it.
🎯 Interview Tip
"What is thrashing and how can you prevent it?"
"Thrashing happens when the system overuses virtual memory and spends more time swapping than processing. It can be prevented by adding RAM, tuning process load, and optimizing caching."
Key Takeaways
Thrashing kills performance.
Solve with better caching, memory allocation, and load control.
12. What are Threads?
A thread is the smallest unit of CPU execution within a process. Multiple threads can run concurrently within the same process space, allowing it to perform multiple tasks efficiently.
For example, in a web server, one thread can handle one user request while another thread handles a database query or file download.
Threads improve concurrency and CPU utilization, especially when some tasks are waiting for I/O operations like network or database calls.
👨🍳
Kitchen Analogy
Think of a process as a restaurant kitchen and threads as workers inside it. Multiple workers can cook different dishes simultaneously while sharing the same kitchen resources.
Think of your program as a company 🏢, and threads as employees working on different jobs at the same time — all sharing the same office (memory).
Core Concepts
Concurrency vs Parallelism: Concurrency = managing multiple tasks; parallelism = running them simultaneously.
Multithreading: Improves CPU utilization. Means running multiple parts of a program (threads) at the same time, so the program can do more than one thing simultaneously.
Context Switching: Switching between threads. CPU stops one thread temporarily, saves its current state, starts running another thread.
Concurrency interleaves tasks on one CPU; parallelism runs tasks simultaneously on multiple CPUs.
🧠 What is Context Switching?
Because CPU cannot truly run unlimited threads at the exact same instant (unless multiple cores exist), it rapidly switches between threads to give the illusion of simultaneous execution.
Suppose: Thread A → downloading file, Thread B → playing music, Thread C → updating UI. CPU does: A → B → C → A → B → C. Very fast. This switching is Context Switching.
Context = current state of thread: where execution stopped, variable values, CPU registers, memory info.
Before switching: OS saves current thread's context. When thread resumes: OS restores it.
Real-World Examples
Web servers spawn threads for each request.
Kafka uses thread pools for message handling.
🧠 What is a Thread Pool?
A thread pool is a fixed group of reusable threads kept ready to execute tasks. Instead of creating a new thread for every request, the system reuses existing threads from the pool.
Why use thread pools? Creating threads repeatedly is expensive (memory allocation, CPU overhead, context switching). Thread pools improve: performance, resource usage, scalability.
Example: A web server gets 1000 requests. Without thread pool: create 1000 new threads ❌ (very expensive). With thread pool: 50 reusable threads pick requests one by one efficiently ✅.
Analogy: Think of a restaurant. Instead of hiring a new waiter for every customer, you keep a fixed staff of waiters who handle customers continuously. That fixed staff = thread pool.
🎯 Thread Pool One-Liner
"A thread pool is a collection of pre-created reusable threads used to execute tasks efficiently without creating a new thread for every request."
Pros & Cons
✅ Advantages
Efficient use of CPU. Faster I/O operations.
❌ Limitations
Complex debugging. Risk of race conditions.
Implementation Tips
Use thread pools instead of creating threads per request.
Synchronize shared resources.
🎯 Interview Tip
"How does multithreading improve server performance?"
"Multithreading improves server performance by enabling concurrency, reducing idle CPU time during I/O waits, and efficiently sharing resources between threads."
Why Multithreading Improves Performance
⚡
Concurrency
Threads allow multiple parts of a program to run at the same time. A web server can use one thread for each incoming user request, so 100 users can be served concurrently instead of one by one.
⏳
Better CPU Utilization During I/O Wait
While one thread waits for a slow operation (like reading from disk or calling an API), other threads can continue executing — keeping the CPU busy instead of idle.
🔄
Resource Sharing
Threads within the same process share memory and resources, so they communicate faster and use less memory compared to multiple processes.
⚠️
Trade-offs
More threads = better concurrency, but also more context switching. Too many threads can reduce performance if not managed properly (use thread pools or async I/O).
Concurrency vs Parallelism
🧠 Concurrency (dealing with many things at once)
A system can manage multiple tasks at the same time, but not necessarily run them simultaneously. Tasks take turns using the CPU. It's about structure and coordination — how tasks are organized and interleaved.
Example: A single chef cooking multiple dishes by switching between them — chopping vegetables, then stirring soup, then checking the oven.
⚡ Parallelism (doing many things at the same time)
Tasks are literally executed at the same time, usually on multiple CPU cores or machines. It's about execution speed and performance.
Example: Multiple chefs each working on a different dish at the same time.
🔑 Key Difference
Concurrency = handling multiple tasks (may overlap in time)
A system can be concurrent but not parallel (one CPU switching between tasks). A system can be both concurrent and parallel (multiple cores running multiple tasks).
In programming:
Concurrency: threads, async/await, event loops
Parallelism: multiprocessing, GPU computing, distributed systems
Concurrency vs Parallelism in Practice
Python threads give you concurrency (one CPU switches between threads due to the GIL — Global Interpreter Lock, which only lets one thread run Python code at a time) but not parallelism. For true parallelism in Python, you need multiprocessing.
Java threads can give you true parallelism on multi-core CPUs — each thread can run on a separate core simultaneously.
Node.js uses concurrency via its event loop — one thread handles thousands of connections by switching between them rapidly. It achieves high throughput without parallelism for I/O-bound work.
Key Takeaways
Threads enable parallelism and high throughput.
Concurrency control is crucial to avoid race conditions.
✅ Module 1 (Basics) — Completed!
System design is about building scalable, reliable, maintainable systems.
Scaling strategies and capacity planning are foundational decisions.
Networking fundamentals (TCP/IP, HTTP) power all communication.
Databases, indexing, caching, threading are core building blocks.
Understanding trade-offs (SQL vs NoSQL, TCP vs UDP, Cache vs DB) is vital in interviews and real systems.
Module 2: LOAD BALANCING
Distributing traffic intelligently
1. What is Load Balancing?
Load balancing is the technique of distributing incoming network traffic or requests evenly across multiple servers to ensure high availability, scalability, and reliability.
It prevents any single server from being overloaded, ensuring smooth user experience and optimal resource utilization.
Why it matters: Without load balancing, traffic spikes can crash servers, cause downtime, or degrade performance — critical issues for large-scale systems.
Core Concepts
Load Balancer: A reverse proxy that sits between clients and servers, routing requests intelligently.
Health Checks: Monitors backend server status and removes unhealthy ones from the pool.
Sticky Sessions: Ensures a client's requests go to the same server (important for stateful apps).
Failover: Automatically reroutes traffic if a server goes down.
Layer 4 vs Layer 7: L4 (Transport) operates at TCP/UDP level — faster but less intelligent. L7 (Application) operates at HTTP level — content-aware routing (e.g., by URL, header).
⚡ Sticky Sessions (Session Persistence)
Suppose you log into a website. Server A stores: User logged in = true, Cart items = 3. Now next request goes to Server B. But Server B does NOT know who you are, your cart, your login session. User may get logged out or lose cart.
Solution: Load balancer says: "All requests from this user should go to the SAME server." This is called Sticky Sessions / Session Affinity.
Flow: User → Load Balancer → Server A (every time).
How load balancer remembers: Using cookies, client IP hash, or session ID.
⚠️ Why sticky sessions are NOT ideal long-term
One server may get overloaded. Poor scalability. If server crashes → session lost.
Modern systems instead: Store session in Redis, distributed session store, or database. Now ANY server can handle the request. This is called Stateless Architecture.
Modern flow: User → LB → Any Server → Redis/DB. Much more scalable.
🎯 Sticky Sessions One-Liner
"Requests from the same client are sometimes routed to the same server using sticky sessions because that server may hold user session state like login information or shopping cart data."
💡 What is a Reverse Proxy?
A reverse proxy is a server that sits in front of one or more backend servers and handles requests from clients on their behalf. So instead of clients talking directly to your servers, they talk to the reverse proxy, and the proxy forwards the requests to the right server.
How it works:
1. A user sends a request. 2. Request goes to the reverse proxy (like Nginx, HAProxy, AWS ELB). 3. The proxy decides which backend server should handle it. 4. The chosen server processes and sends response back through the proxy. The user never directly sees or connects to the backend servers.
Why use a reverse proxy?
✅ Load Balancing: Distributes traffic across multiple servers.
✅ Security: Hides internal server details (like IPs).
✅ Caching: Can store frequently requested responses.
✅ SSL Termination: Handles HTTPS encryption so backend servers don't have to.
✅ Failover: If one server goes down, redirects to a healthy one.
Analogy: Think of it like a reception desk at a company. Visitors (clients) talk only to the receptionist (reverse proxy). The receptionist decides which employee (backend server) to connect them with. The visitor never directly interacts with employees.
Forward proxy vs Reverse proxy:
Forward proxy: sits in front of clients → hides clients from servers (used for privacy or filtering). Reverse proxy: sits in front of servers → hides servers from clients (used for load balancing and security).
Real-World Examples
Google Search: Traffic distributed across thousands of servers using L7 load balancers.
Netflix: Uses load balancers in front of microservices to balance billions of daily API calls.
Prevents downtime due to overload. Improves latency and response time. Enables horizontal scaling. Provides high availability and fault tolerance.
❌ Limitations
Extra network hop (slight latency). Adds complexity and cost. Single point of failure if not highly available.
L4 vs L7 Comparison
Feature
L4 Load Balancer
L7 Load Balancer
Layer
Transport (TCP/UDP)
Application (HTTP/HTTPS)
Speed
Faster
Slightly slower
Routing
IP/Port based
Content-based
Use Case
Low-latency apps
Web apps, APIs
🚦 L4 Load Balancer (Transport Layer)
"Fast but not very smart." Only looks at IP address and Port number. Does NOT inspect actual HTTP request content. Like a traffic police officer — just directs cars quickly, doesn't know what's inside cars.
Advantages: Very fast, low latency, great for huge traffic.
Limitation: Cannot do smart routing (can't understand /api vs /images vs /payments). Used in gaming, TCP services, very high-performance systems.
Example: AWS Network Load Balancer (NLB), HAProxy in TCP mode.
🚀 L7 Load Balancer (Application Layer)
"Slower but very smart." Can inspect: URL, headers, cookies, HTTP methods. Can route /images → Image Server, /api → Backend API Server, /videos → Video Server. L4 cannot do this.
Advantages: Smart routing, SSL termination, caching/compression, better for microservices.
Disadvantage: Slightly slower because it deeply inspects requests. Used in web apps, APIs, microservices.
"L4 load balancers operate at the transport layer and route traffic using IP addresses and ports, making them very fast but less intelligent. L7 load balancers operate at the application layer and can inspect HTTP content like URLs, headers, and cookies to perform advanced routing and traffic management."
Step-by-Step Flow
Client sends request to load balancer.
Load balancer checks server health and availability.
Uses routing algorithm to pick a server.
Routes request → server processes → response sent back via load balancer.
👮
Traffic Police Officer Analogy
Think of a load balancer as a traffic police officer at a busy junction — distributing cars (requests) evenly to prevent congestion.
🎯 Interview Tip
"How does load balancing improve availability?"
"Load balancing improves availability by distributing incoming traffic across multiple servers, so no single server becomes a bottleneck or point of failure. It ensures that if one server goes down, the load balancer automatically redirects requests to healthy servers — keeping the application online."
Key Points to Mention
⚖️
Even Traffic Distribution
Prevents overloading one server while others sit idle.
🏥
Health Checks
Load balancers continuously monitor server health. If a server fails a health check, it's temporarily removed from rotation until it recovers.
🛡️
High Availability & Failover
If one server or even a region goes down, traffic automatically routes to available servers — ensuring uptime.
📈
Scalability
When traffic increases, you can add more backend servers — the load balancer will include them automatically.
✅ Example
In a web app with 5 servers, if one crashes, users won't even notice — the load balancer simply routes traffic to the other 4 servers.
Routing Algorithms (Brief)
Round Robin: Send each request to the next server in turn.
Least Connections: Send to the server with the fewest active connections.
IP Hash: Route a client's requests to the same server consistently.
🎯 One-Line Summary
"Load balancers improve availability by routing traffic intelligently and removing failed servers. L4 works at the transport layer using IP/port, while L7 works at the application layer using content-based routing."
Key Takeaways
Load balancing is essential for scaling and uptime.
L4 = speed; L7 = intelligence.
Always deploy load balancers in high-availability mode to avoid SPOF.
2. What is Consistent Hashing?
Consistent hashing is a distributed hashing technique used to evenly distribute requests or data across servers while minimizing redistribution when nodes are added or removed.
Why it matters: It solves a major load balancing problem — when servers scale up/down — without massive data reallocation.
🧠 What problem happens WITHOUT consistent hashing?
Suppose you have 3 servers: A, B, C. Users are distributed: User1 → A, User2 → B, User3 → C.
Traffic increases. You add a new server: D.
Problem with normal hashing: With traditional hashing (hash % N), almost ALL users/data may get reassigned. Before: User1 → A. After adding D: User1 → C.
What does this mean practically? Suddenly: cache data invalid, sessions moved, data copied everywhere, lots of network traffic.
Consistent hashing solves this by only reassigning a small fraction of keys when servers change.
The Hash Ring Concept
Consistent hashing is important because when servers are added or removed, only a small portion of data needs to move to different servers instead of reshuffling everything.
In consistent hashing, both servers and data keys are mapped onto a circular hash space called a hash ring. The ring represents the full range of hash values, typically from 0 to 232 - 1.
Hash Ring: All servers and keys are mapped to a circular hash space (0 to 232 - 1).
Mapping: Each request/data key hashes to a point on the ring and is served by the next node clockwise.
Minimal Disruption: Adding/removing a server only affects a small fraction of keys.
Virtual Nodes: Multiple hash points per server lead to smoother load distribution.
Consistent hashing ring: servers and keys are hashed onto a circular space. Each key is handled by the next server clockwise.
How Consistent Hashing Works
Create a hash ring (0 to 232)
The entire hash space is arranged in a circle. Every possible hash value sits somewhere on this ring.
Hash server IPs and place them on the ring
Each server's identifier (e.g., IP address) is hashed, and the resulting value determines the server's position on the ring.
Hash request key and locate next server clockwise
When a data key or request comes in, hash it. Walk clockwise on the ring until you hit the first server node — that server handles the request.
On adding/removing a node, only nearby keys remap
When a new server joins, it takes over only the keys between itself and the previous node. When a server leaves, its keys move to the next node clockwise. Most data stays put.
EXAMPLE
Before: Server A handles users 1-100.
Add Server D between A and B on the ring.
After: Only users 80-100 move to D. Remaining users stay with their original servers.
With traditional hashing, adding server D would require reassigning nearly all users — that is massive reallocation. Consistent hashing minimizes this chaos.
Clarification: When Server D is placed between A and B on the hash ring, D takes over keys that were previously assigned to B (the next clockwise server from D's position), not from A. The keys that move are the ones between A and D on the ring — these were B's responsibility and are now D's.
Virtual Nodes
Why Virtual Nodes?
Without virtual nodes, servers get unevenly placed on the hash ring — by pure chance, Server A might "own" 70% of the ring while Server B owns only 10%. Virtual nodes fix this by placing each server at multiple positions on the ring (e.g., A1, A2, A3, B1, B2, B3), so even if one position is unlucky, the overall distribution averages out. More virtual nodes = smoother distribution.
Without virtual nodes, servers are placed randomly on the hash ring. This can lead to uneven distribution — one server might get a huge arc of the ring (handling 70% of traffic) while another gets a tiny arc (handling only 10%).
Why does uneven distribution happen?
Because each server has only 1 position on the ring. Random placement can create imbalance — one server becomes overloaded while others sit mostly idle.
The solution: Virtual Nodes. Instead of giving each server a single position on the ring, we assign multiple logical positions (virtual nodes) to each physical server. These virtual nodes are spread around the ring.
A virtual node is NOT an actual smaller physical part of the server — it is a logical/virtual representation of the same server on the hash ring.
Physically, it is still one server. Logically, it appears multiple times on the ring.
For example, Physical Server A might appear on the ring as A1, A2, A3, A4. All four virtual nodes route to the same physical Server A.
Without virtual nodes: A gets huge space, B gets tiny space, C gets medium space. Result: A handles 70%, B handles 10%, C handles 20%. Uneven and unfair.
With virtual nodes: We create A1, A2, A3 / B1, B2, B3 / C1, C2, C3 — all spread around the ring.
Result: Traffic gets distributed much more evenly — A, B, C each get approximately 33%. When a server is added or removed, only a small amount of data needs to be moved.
Comparison: Consistent vs Modulo Hashing
Feature
Modulo Hashing
Consistent Hashing
Redistribution
Large (all keys)
Minimal (few keys)
Scalability
Poor
Excellent
Complexity
Simple
Moderate
Load Balance
Even (initially)
Even (with virtual nodes)
Fault Tolerance
Poor — all keys reshuffle
High — only affected keys move
Real-World Examples
CDNs: Map user requests to nearest cache node.
Distributed Caches — Memcached (a distributed in-memory cache) and Cassandra (a distributed NoSQL database): Use consistent hashing to distribute keys across cache/data nodes.
Load Balancers: Assign client sessions to backend servers predictably.
Pros
Minimal data movement during scaling.
High scalability and fault tolerance.
Even load distribution with virtual nodes.
Cons
Uneven distribution without virtual nodes.
Slightly more complex implementation than simple modulo hashing.
🍕
Pizza Delivery Analogy
You and your friends are delivering pizzas in a city. You split the city into zones using house numbers. You have 3 delivery people — A, B, and C. You assign houses: A: 1-33, B: 34-66, C: 67-100.
Now if a new friend D joins, with normal hashing you'd have to redivide all house numbers — everyone's deliveries change. Chaos!
With consistent hashing: Imagine a circular pizza with numbers 0-100 written around the crust. Place each delivery person (A, B, C) randomly on the pizza edge. Every house number is also placed on the pizza, and that house's pizza goes to the next delivery person clockwise.
If D joins and is placed between A and B, only the houses between A and D move to D. All other houses stay with their old delivery people. Very few houses are affected.
Virtual nodes: If one person gets too many deliveries, give each person multiple smaller spots on the pizza. That way, houses are spread out more evenly.
🔒
Clock with Lockers Analogy
Imagine a clock with lockers around it — each item goes to the next available locker clockwise. Adding/removing a locker only affects a few items. This is exactly how consistent hashing distributes and reassigns data on the ring.
🎯 Interview Tip
"Why is consistent hashing used in distributed systems?"
"How does consistent hashing handle node failures?"
Tip: Always mention hash ring, minimal key remapping, and virtual nodes. A strong interview statement: "Virtual nodes are logical representations of a physical server on the hash ring. They do not physically exist separately, but help distribute load more evenly."
Key Takeaways
Core to modern distributed caching and load balancing.
Avoids the "re-sharding storm" problem — only a small fraction of keys move when nodes change.
Virtual nodes are essential for even load distribution.
Moving data is expensive (network cost, cache rebuilds, DB updates, latency spikes) — consistent hashing minimizes this.
3. What is Sharding?
Sharding is the process of splitting a large database or dataset into smaller, faster, more manageable parts called shards — each stored on a separate server.
Why it matters: It is a critical scaling technique to handle massive data and query volumes beyond what a single server can handle.
Core Concepts
Shard Key: A field used to decide which shard stores a particular record (e.g., user_id).
Horizontal Partitioning: Splitting rows across multiple tables/databases. Each shard holds a subset of the data.
Shard Map / Directory: Tracks which shard holds which data, enabling the routing layer to direct queries.
Re-sharding: Redistributing data when scaling further — adding new shards and rebalancing the data.
Sharding splits a large database into smaller pieces (shards), each on its own server, based on a shard key like username range.
Comparison: Sharding vs Replication
Feature
Sharding
Replication
Purpose
Scale capacity
Improve availability
Data
Partitioned (each shard has different data)
Copied (each replica has same data)
Queries
Routed to specific shard
Served by any replica
Failure Impact
Only that shard's data unavailable
Other replicas take over
Real-World Examples
Instagram: Shards users based on user_id ranges.
YouTube: Shards videos/comments for storage and indexing.
MongoDB / Cassandra: Built-in sharding support for horizontal scaling.
Pros
Infinite horizontal scaling.
Better performance and reduced query latency.
Avoids single-node bottlenecks.
Cons
Complex query routing.
Re-sharding is expensive.
Cross-shard transactions are hard.
Correction: Nothing scales infinitely. Sharding has practical limits: shard management overhead, cross-shard query costs, hotspot risks, and network complexity. The accurate term is "near-linear horizontal scaling."
Why Are Cross-Shard Transactions Hard?
If User A (on Shard 1) sends money to User B (on Shard 2), you need both shards to agree the transaction succeeded. This requires a protocol like Two-Phase Commit (2PC): a coordinator asks both shards "can you commit?" and only if both say yes does the transaction go through. This is slow and complex — which is why sharding works best when most queries stay within a single shard.
Shard Failure + Replication
In production, each shard is also replicated (has backup copies). So if the server running Shard 1 crashes, a replica takes over. You would never run a single unreplicated shard in a real system — that would be a single point of failure for all the data on that shard.
Step-by-Step: Sharding Flow
Choose shard key (e.g., user_id)
Select a field that distributes data evenly and aligns with your query patterns.
Implement routing layer to map requests to shards
A shard map or consistent hashing directs each query to the correct shard.
Store data across multiple shard databases
Each shard holds a portion of the dataset on separate servers.
Add new shards and re-balance data if needed
As data grows, new shards are added and data is redistributed to maintain even distribution.
What Exactly is the Routing Layer?
The routing layer is typically a lightweight service or library that maps a request to the correct shard. For example, it might hash the user ID and use modulo to pick a shard: hash('user123') % 3 = Shard 2. This can be a separate proxy service (like Vitess for MySQL or ProxySQL) or built directly into your application code. The key requirement is that every client must agree on the same routing logic, otherwise requests go to the wrong shard.
📖
Phone Book Analogy
Think of sharding as splitting a phone book by city — each city is handled by a different server, but collectively they serve the whole country. No single book is too large to manage, and you always know which book to look in based on the city.
🎯 Interview Tip
"How would you shard a users table with 1 billion rows?"
Tip: Discuss shard key selection (e.g., user_id for even distribution), routing (how the application layer knows which shard to query), and re-sharding strategies (consistent hashing to minimize data movement).
Key Takeaways
Sharding is the go-to solution for massive scale.
Good shard key selection is critical for balanced distribution.
Re-sharding and cross-shard joins are key challenges.
The Hotspot Problem — Bad Shard Keys:
Choosing the wrong shard key can funnel most traffic to one shard, defeating the purpose of sharding entirely.
Classic examples:
Sharding by date: All writes go to the "today" shard. Yesterday's shard sits idle. Tomorrow's shard doesn't exist yet. One shard gets 100% of write traffic.
Sharding by celebrity user_id: Justin Bieber's shard gets 1000x the read traffic of a regular user's shard.
Solutions: (1) Composite shard keys (user_id + date). (2) Salting — add a random prefix to spread writes (e.g., append a random 0-9 to the key, spreading writes across 10 sub-shards). (3) Separate hot data into its own handling path (cache celebrities' data separately).
Interview tip: "What happens if your shard key is bad?" is one of the most common follow-up questions after any sharding discussion.
Module 2 Recap — Load Balancing
Load Balancing, Consistent Hashing, and Sharding
Module 2 Summary
Load Balancers distribute requests across servers to ensure high availability and performance.
Consistent Hashing solves the scaling problem by minimizing data redistribution when servers are added or removed.
Sharding allows databases to scale horizontally by splitting data intelligently across multiple servers.
Together, these three concepts form the foundation of scalability in large distributed systems.
Module 3: Datastores
Bloom Filters, Data Replication, NoSQL Optimization, Location-Based Databases, Database Migrations
1. What are Bloom Filters?
A Bloom filter is a probabilistic data structure that quickly checks whether an element is definitely not in a set or might be in a set — with extremely low memory usage.
It is commonly used to avoid unnecessary database lookups and speed up queries.
Why it matters: In large-scale systems (billions of records), checking existence before querying saves resources and improves performance.
Core Concepts
Bit Array: Fixed-size array of bits initialized to 0.
Hash Functions: Multiple independent hash functions map a value to multiple bit positions.
Membership Test:
If all bits at those positions = 1, the element might be present.
If any bit = 0, the element is definitely not present.
False Positives: Possible but rare (may say "present" when it is not).
False Negatives: Impossible (if Bloom filter says "not present," it is guaranteed).
Bloom Filter: The key "cat" is hashed by three independent functions, setting bits at positions 2, 5, and 8 to 1.
Comparison: Bloom Filter vs Hash Set
Feature
Bloom Filter
Hash Set
Memory Usage
Very low
High
Accuracy
Probabilistic (false positives possible)
100% accurate
Operations
Insert, Query
Insert, Query, Delete
Speed
Ultra-fast (O(k) hash lookups)
Fast (O(1) average)
Real-World Examples
Google Bigtable: Uses Bloom filters to check if a row exists before disk I/O.
Web crawlers: Avoid re-crawling already visited URLs.
Email spam filters: Check if sender exists in a known spam list.
Pros
Ultra-fast lookups.
Extremely memory-efficient.
Prevents expensive database hits.
Cons
False positives possible.
No deletion (unless using counting Bloom filters).
Cannot retrieve elements — only membership check.
Step-by-Step Example
Initialize a 1,000-bit Bloom filter
All bits start at 0.
Hash "alice@example.com" with 3 functions
Set bits at positions returned by each hash function to 1.
Query "bob@example.com"
Check the bits at the positions returned by the same 3 hash functions. If any bit is 0, bob is definitely not present. If all are 1, bob might be present (could be a false positive).
🛂
Security Checkpoint Analogy
Think of a security checkpoint with a blacklist. If your name isn't flagged, you're definitely not on the list. If it is flagged, they double-check (false positives). Quick screening before the expensive full lookup.
🎯 Interview Tip
"What is a Bloom filter, and where would you use it?"
Tip: Always mention probabilistic, false positives, and low memory. Example answer: "A Bloom filter is a space-efficient probabilistic data structure used for membership testing. It can have false positives but never false negatives, making it ideal for pre-filtering expensive database lookups."
Key Takeaways
Bloom filters are perfect for fast, memory-efficient membership checks.
They trade perfect accuracy for speed and efficiency.
2. What is Data Replication?
Data replication is the process of copying data from one database server to one or more others to ensure availability, durability, and scalability.
Why it matters: Replication is essential for disaster recovery, read scaling, and geo-distribution.
Core Concepts
Primary-Replica Model: One primary (write) node, multiple replicas (read).
Synchronous Replication: Data written to all replicas before confirming success — provides strong consistency but higher latency.
Asynchronous Replication: Write confirmed after primary success — provides eventual consistency with lower latency.
Multi-Master Replication: Multiple writable nodes — enables writes at any node but requires complex conflict resolution.
Data Replication: The Primary node receives writes and replicates to Replica 1 (synchronously) and Replica 2 (asynchronously).
Comparison: Synchronous vs Asynchronous Replication
Feature
Synchronous
Asynchronous
Consistency
Strong
Eventual
Latency
Higher
Lower
Availability
Lower (waits for all replicas)
Higher (doesn't wait)
Data Loss Risk
Minimal
Possible (during lag)
Real-World Examples
MySQL Read Replicas: Handle massive read traffic by distributing reads across replicas.
Cassandra: Data automatically replicated across multiple data centers.
MongoDB Replica Sets: For high availability and failover — if primary fails, a replica is promoted.
Pros
Improved read scalability.
High availability and fault tolerance.
Disaster recovery readiness.
Cons
Increased storage usage.
Possible data lag (async).
Complex conflict resolution (multi-master).
Step-by-Step: Primary-Replica Flow
Write request goes to Primary DB
All write operations are directed to the primary node.
Primary applies changes and sends them to replicas
The primary processes the write and forwards the changes to all replica nodes.
Replicas update their data asynchronously or synchronously
Depending on configuration, replicas may update immediately (sync) or with a slight delay (async).
Read requests are distributed across replicas
Read traffic is load-balanced across replicas, reducing the load on the primary.
📄
Google Doc Analogy
Like sharing a Google Doc with multiple devices — all copies get updated, but some might sync slightly later. The primary doc is the source of truth, and replicas eventually catch up.
🎯 Interview Tip
"How would you design a system for read-heavy workloads?"
Tip: Mention replication, read scaling, and failover. Explain how read replicas can handle massive read traffic while the primary handles writes.
Key Takeaways
Replication improves availability, read performance, and resilience.
Must balance consistency vs latency based on use case.
3. How are NoSQL Databases Optimized?
NoSQL databases are optimized for massive scale, low latency, and flexible data models. They achieve this by trading off consistency, using distributed storage, and adopting denormalization.
Core Optimization Techniques
Denormalization: Store data together to avoid joins. Pre-compute and embed related data within a single document.
Wait, isn't normalization good practice?
In relational databases, yes — normalization reduces data duplication and keeps data consistent. But NoSQL databases are designed for speed at scale, and they can't do JOINs efficiently (or at all). So instead of splitting data across tables and joining them at query time, NoSQL stores related data together in one document.
You trade storage efficiency for read speed — and at scale, storage is cheap but latency is expensive. A few extra kilobytes of duplicated data is a small price to pay for serving queries in milliseconds instead of seconds.
Partitioning / Sharding: Distribute data across multiple nodes for parallel processing.
Replication: Improve availability and performance by copying data to multiple nodes.
Indexes & Secondary Indexes: Speed up queries by creating indexes on frequently-queried fields.
Compaction & TTL: Remove stale data and optimize storage by expiring old records automatically.
Write-optimized Storage (LSM Trees): Batch writes for speed — Log-Structured Merge Trees are used by Cassandra, RocksDB, etc.
How LSM Trees Work
Instead of writing every change directly to disk (which is slow due to random I/O), an LSM Tree first writes to an in-memory buffer called a memtable. When the memtable fills up, it is flushed to disk as a sorted file (called an SSTable — Sorted String Table). Periodically, these files are merged together in a process called compaction.
This makes writes extremely fast (sequential I/O instead of random I/O) at the cost of slightly more complex reads (may need to check multiple SSTables before finding the data). This trade-off is why LSM Trees are ideal for write-heavy workloads like logging, time-series data, and messaging systems.
Comparison: Optimized NoSQL vs Traditional SQL
Feature
NoSQL
SQL
Schema
Flexible
Fixed
Scalability
Horizontal
Vertical
Performance
Optimized for specific queries
General-purpose
Consistency
Tunable (eventual to strong)
Strong (ACID)
Real-World Examples
Cassandra: Uses partition keys and LSM trees for write-heavy systems.
MongoDB: Denormalizes documents to reduce joins — embeds related data in single documents.
DynamoDB: Uses auto-sharding and on-demand scaling for seamless horizontal scaling.
Pros
High scalability and low latency.
Flexible schema design.
Great for real-time data.
Cons
Complexity in query logic.
Eventual consistency issues.
More storage consumption due to denormalization.
Step-by-Step Optimization Approach
Identify query access patterns
Understand what queries the application makes most frequently before designing the schema.
Denormalize schema around queries
Design your data model to serve the most common queries directly, even if it means duplicating data.
Add partitioning + replication
Distribute data across nodes for scalability and copy it for availability.
Tune indexes and TTL policies
Create secondary indexes for additional query patterns and set TTLs to automatically expire stale data.
Monitor and rebalance shards
Continuously monitor shard sizes and redistribute data to prevent hotspots.
🍔
Fast-Food Kitchen Analogy
Think of NoSQL like a fast-food kitchen — it pre-prepares common items (denormalization) to serve customers instantly. Instead of cooking each dish from scratch (like SQL joins), the kitchen pre-assembles popular combos so orders are fulfilled in seconds.
🎯 Interview Tip
"How does Cassandra achieve high write throughput?"
Tip: Mention partitioning, replication, and LSM trees. Cassandra writes to an in-memory memtable first (fast), then flushes to disk using LSM trees, which batch and compact writes for optimal throughput.
Key Takeaways
NoSQL optimizations are about speed, scale, and flexibility.
Trade-offs: storage and consistency vs performance.
4. What are Location-Based Databases?
A location-based database stores and queries data based on geographical coordinates (latitude, longitude).
Why it matters: Essential for applications like ride-sharing, delivery, mapping, and location search.
Core Concepts
Geospatial Indexing: Specialized indexes (R-trees, geohashes) to optimize spatial queries.
Spatial Queries: "Find all restaurants within 5 km." — range-based queries on geographic data.
Geohashing: Convert lat-long into a single key for easy lookup. A geohash encodes a location into a short string where similar strings are geographically nearby.
Proximity Search: Combines spatial and attribute-based queries — e.g., "Find all open restaurants within 2 km."
What is an R-tree?
An R-tree is like a B-tree but for spatial data. Instead of organizing numbers on a line, it organizes rectangles in 2D space. Each node contains a bounding box that encloses all its children. To find "all restaurants within 2km," the tree quickly eliminates entire regions that don't overlap with your search area — much faster than checking every restaurant individually.
(See the B-tree index diagram in Module 1 for a refresher on how tree-based indexing works.)
How Geohashing Enables Proximity Search
Nearby locations share a common geohash prefix. For example, two cafes on the same street might have geohashes "tdr1w9" and "tdr1w8" — they share the prefix "tdr1w". To find nearby places, you just query for all locations whose geohash starts with the same prefix. The longer the shared prefix, the closer the locations are.
Comparison: Geohash vs R-tree
Feature
Geohash
R-tree
Speed
Faster lookups
Better for range queries
Accuracy
Lower (grid-based approximation)
Higher (precise boundaries)
Complexity
Simple
More complex
Best For
Point lookups, nearby search
Complex spatial queries
Real-World Examples
Uber: Matches riders and drivers using geospatial indexing.
Google Maps: Uses R-trees and geohashes for nearby search.
Yelp: Location-based queries for businesses — "restaurants near me."
Pros
Optimized for spatial queries.
Enables proximity and range filtering.
High performance for location-based apps.
Cons
Complex indexing and storage.
Higher overhead on writes.
Requires specialized query support.
Step-by-Step: Location Query Flow
Convert location to geohash
Transform the latitude/longitude into a geohash string for efficient indexing.
Query database for matching geohash range
Search for all records whose geohash prefixes match, giving nearby results.
Filter results by radius/distance
Apply a distance filter to eliminate results outside the desired range.
Return sorted results based on proximity
Sort the remaining results by distance from the user's location.
🗺️
City Map Grid Analogy
Think of a city map grid — each block has a unique code (geohash), so finding nearby places becomes easy. Instead of searching the entire city, you just look at the neighboring grid cells. The more specific the code, the smaller the area — like zooming in on a map.
🎯 Interview Tip
"How would you design the 'nearby drivers' feature in Uber?"
Tip: Mention geohashing, spatial indexing, and range queries. Explain how drivers' locations are stored with geohashes, and a rider's request triggers a geohash range query to find drivers in nearby cells.
Key Takeaways
Location-based databases are key to modern apps with geospatial features.
Choosing the right indexing method is crucial for performance.
5. Database Migrations
Database migration is the process of modifying a database schema or moving data from one database/storage system to another — without downtime or data loss.
Why it matters: Systems evolve. Schema changes, scaling, or technology upgrades require careful migration strategies.
Core Concepts
Schema Migration: Adding/removing tables, columns, indexes — changing the structure of the database.
Data Migration: Moving or transforming data between systems — e.g., from one database to another.
Online vs Offline Migration: Online = zero downtime (changes applied while system runs); Offline = scheduled downtime (system taken down during migration).
Versioning Tools: Liquibase, Flyway, Alembic help manage migration scripts and track which migrations have been applied.
Migration Strategies
Strategy
Description
Blue-Green
Run new version in parallel, switch traffic once stable
Dual Writes
Write to both old and new systems temporarily
Shadow Reads
Compare reads from old vs new systems for validation
How Shadow Reads Work
During migration, every read query is sent to BOTH the old and new database simultaneously. Only the old database's result is returned to the user. Behind the scenes, engineers compare both results — if they match, the new database is working correctly. If they don't match, there's a bug to fix before switching over. The user never sees any disruption.
Real-World Examples
Instagram: Migrated from PostgreSQL monolith to sharded MySQL.
GitHub: Migrated databases live without downtime using blue-green deployments.
Fact check: Instagram actually migrated from an unsharded PostgreSQL to a sharded PostgreSQL setup (using their own sharding layer). They famously stayed on PostgreSQL. Getting this wrong at Meta is embarrassing — always verify migration claims.
Pros
Keeps schema and data updated.
Enables scaling and new features.
Allows tech stack upgrades.
Cons
Risk of data loss or corruption.
Downtime if poorly planned.
Complex rollback mechanisms.
Step-by-Step: Schema Migration
Write migration scripts
Define the exact schema changes in versioned migration files (e.g., add column, create index).
Run on staging environment
Test the migration on a staging database that mirrors production.
Deploy incrementally with backups
Take a full backup, then apply migration in stages. Monitor for errors at each stage.
Monitor and rollback if necessary
Watch for issues in production. If something goes wrong, use the backup and rollback scripts to revert.
🏠
House Renovation Analogy
It is like renovating a house while people still live inside — changes must be seamless and non-disruptive. You can't tear down a wall without first building a temporary support. Database migrations require the same careful planning to avoid "collapsing" the system.
🎯 Interview Tip
"How would you migrate a production database without downtime?"
Tip: Mention blue-green deployments, dual writes, and rollback plans. Explain how you'd run old and new systems in parallel, gradually shift traffic, and validate data consistency before cutting over.
Key Takeaways
Migrations are inevitable — plan for zero downtime and rollback safety.
Automation tools and version control are critical for reliability.
Bloom Filters prevent expensive lookups with probabilistic checks.
Replication improves availability, scalability, and disaster recovery.
NoSQL Optimization focuses on performance, partitioning, and denormalization.
Location-based Databases power proximity search and geo-based queries.
Migrations are essential for evolving systems and must be carefully managed.
Module 4: Consistency vs. Availability
Data Consistency, Data Consistency Levels, Transaction Isolation Levels
Quick Reminder from Module 3
Replicas receive updates with a slight delay (replication lag). This delay is the root cause of most consistency challenges we'll discuss below — when different replicas have different versions of the data, which version should a reader get?
1. What is Data Consistency?
Data consistency ensures that every client sees the same data view across the system — no matter which replica, server, or data center they query.
In distributed systems, consistency becomes challenging because data is replicated across multiple nodes.
Why it matters: Many applications (banking, payments, inventory) require consistent data to function correctly.
Core Concepts
Strong Consistency: All reads return the most recent write. Every node reflects the latest state.
Eventual Consistency: All nodes will eventually have the same data, but not immediately. Reads may return stale data temporarily.
Read-After-Write Consistency: Once you write data, subsequent reads reflect the change. Your own writes are always visible to you.
Monotonic Reads: Once you read a value, you will not see older values later. Reads never go backward in time.
Strong vs Eventual Consistency: With strong consistency, all nodes update before reads return. With eventual consistency, reads may temporarily return stale data.
Comparison: Strong vs Eventual Consistency
Feature
Strong
Eventual
Latency
Higher
Lower
Scalability
Harder
Easier
Use Case
Payments, Banking
Social Feeds, Analytics
Data Accuracy
Always current
May be stale briefly
Real-World Examples
Google Spanner: Provides strong global consistency using atomic clocks (TrueTime).
Amazon DynamoDB / Cassandra: Offer eventual consistency for higher availability.
Banking systems: Must use strong consistency for account balances — you cannot show stale data for a bank balance.
Pros (of strong consistency)
Predictable and correct results.
Simplifies application logic.
Cons (of strong consistency)
Slower writes and reads (coordination overhead).
Harder to scale globally.
Step-by-Step Example
User A updates their profile photo
A write request is sent to the primary node.
With strong consistency, all replicas update before any user sees it
The system waits for all replicas to acknowledge the change before returning success.
With eventual consistency, some users might see the old photo briefly
The write returns immediately after the primary confirms. Replicas catch up asynchronously — some users may see stale data for a short window.
🕐
Synchronized Clock Analogy
Strong consistency is like a synchronized clock — every device shows the same time instantly. No matter which clock you look at, you get the correct time.
Eventual consistency is like syncing photos to the cloud — they eventually all match, but there is a brief window where some devices show the old photo and others show the new one.
🎯 Interview Tip
"What is consistency, and why is it hard in distributed systems?"
Tip: Always mention replication delays, CAP theorem, and trade-offs. Explain that the CAP theorem states you can only have 2 of 3: Consistency, Availability, Partition Tolerance — and since network partitions are inevitable, you must choose between C and A.
Key Takeaways
Consistency ensures data accuracy across nodes.
There is always a trade-off with availability and latency.
Choosing the right level depends on the use case.
2. Data Consistency Levels
Consistency levels define how up-to-date and synchronized data must be when read from a distributed system. They let you balance speed, cost, and correctness based on application requirements.
Core Concepts
Strong Consistency: Reads always reflect the latest write. All nodes agree before responding.
Eventual Consistency: Reads may return stale data, but will converge eventually.
Causal Consistency: Reads respect causal relationships — if A leads to B, then B is never seen before A.
Monotonic Reads: Reads never go backward in time. Once you see a value, you won't see an older one.
Read-Your-Writes: After you write, you will always see your write. Other users may not see it immediately.
Quorum Consistency: Achieved when a majority of nodes agree (common in Cassandra/DynamoDB). Tunable via W and R parameters.
Causal Consistency in Action
You post "I won the lottery!" then immediately post "Just kidding!" Causal consistency ensures no one ever sees the second message without the first — because the second causally depends on the first. Without causal consistency, a replica could show "Just kidding!" before "I won the lottery!" causing confusion.
Monotonic Reads Scenario
Without monotonic reads, this can happen: You check your inbox and see 10 emails. You refresh the page, but this time your request hits a different replica that's slightly behind — now you see only 8 emails. Two emails seemingly "disappeared." Monotonic reads prevent this by ensuring each subsequent read returns data at least as recent as the previous one.
Read-Your-Writes in Practice
You update your profile photo on Instagram. With read-your-writes consistency, you immediately see your new photo when you view your own profile. But your friend, whose request hits a different replica, might still see your old photo for a few seconds until that replica catches up.
Comparison of Consistency Levels
Level
Latency
Accuracy
Use Case
Strong
High
Highest
Banking
Eventual
Low
Lower
Social Feeds
Quorum
Medium
Medium-High
E-commerce
Causal
Medium
Medium
Messaging Apps
Monotonic Reads
Medium
Medium
Session-based apps, user-facing dashboards
Real-World Examples
Cassandra: Offers tunable consistency — you can configure QUORUM, ONE, or ALL.
Amazon S3: Eventual consistency for high scalability (now offers strong consistency for reads-after-writes).
Google Spanner: Strong consistency globally using TrueTime atomic clocks.
Monotonic Reads: Social media feeds — once you've seen 10 posts, refreshing should never show fewer
Pros (tunable consistency)
Flexibility in performance vs. accuracy.
Application-level control over trade-offs.
Cons
Complex to reason about.
Requires careful design to avoid stale reads.
Step-by-Step: Quorum Example
Set up a 5-node cluster
You have 5 replicas of your data.
Configure W=3, R=3 for strong consistency
Write quorum (W=3) means 3 nodes must confirm a write. Read quorum (R=3) means 3 nodes are read. Since W + R > N (3+3 > 5), you are guaranteed to read the latest write.
Or configure W=1, R=1 for speed
Faster but eventually consistent — W + R is not greater than N, so stale reads are possible.
Concrete Quorum Example with 5 Nodes
When you write, the request goes to all 5 nodes. With W=3, you wait for 3 nodes to confirm before telling the client "write successful." On a read with R=3, you ask all 5 nodes but only need 3 responses — you take the one with the latest timestamp. Since W(3) + R(3) = 6 > N(5), at least one node in your read set must have the latest write. That overlap guarantees you read fresh data.
🗳️
Group Decision-Making Analogy
It is like group decision-making:
Strong = Everyone agrees before moving forward.
Eventual = Eventually, everyone agrees.
Quorum = Majority agreement is enough.
Just like in a board meeting — sometimes you need unanimous consent (strong), sometimes majority vote suffices (quorum), and sometimes you just announce and people catch up later (eventual).
🎯 Interview Tip
"Explain different consistency models."
"What's the difference between causal and eventual consistency?"
Tip: Use examples like Cassandra (tunable) or Spanner (strong global). Explain that consistency is not binary — it is a spectrum, and the right choice depends on the use case and SLAs.
Key Takeaways
Consistency is not binary — it is a spectrum.
Tunable consistency lets you choose latency vs. accuracy.
The right model depends on use case and SLAs.
3. Transaction Isolation Levels
Transaction isolation defines how transactions interact when executed concurrently. It ensures data integrity by controlling visibility of intermediate changes.
Why it matters: Concurrency issues can cause data corruption or inconsistent reads in multi-user environments.
Core Concepts (SQL Standard Levels)
Isolation Level
Anomalies Prevented
Description
Read Uncommitted
None
Transactions can read uncommitted changes (dirty reads possible)
Read Committed
Dirty Reads
Only committed data is read
Repeatable Read
Dirty + Non-Repeatable Reads
Same query always returns same result within a transaction
Serializable
All anomalies
Full isolation — transactions behave as if executed sequentially
Important: Repeatable Read does NOT prevent phantom reads — new rows can still appear between queries in the same transaction. This is the key reason Serializable exists: it's the only level that prevents all three anomalies (dirty reads, non-repeatable reads, and phantom reads), at the cost of the lowest performance.
Database-specific nuance: The above is true per the SQL standard. However, MySQL InnoDB's Repeatable Read DOES prevent phantom reads via next-key locking. PostgreSQL's Repeatable Read is actually Snapshot Isolation, which also prevents phantoms. In interviews, say: "Per the SQL standard, Repeatable Read doesn't prevent phantoms, but specific implementations like MySQL InnoDB go beyond the standard."
Transaction Isolation Spectrum: Moving from Read Uncommitted (fastest, least safe) to Serializable (slowest, safest).
Common Anomalies
Dirty Read: Reading uncommitted changes from another transaction. If that transaction rolls back, you read data that never existed.
Non-repeatable Read: Same query returns different results within a transaction because another transaction modified the data between reads.
Phantom Read: New rows appear/disappear between queries because another transaction inserted or deleted rows.
Why Are They Called "Phantom" Reads?
It is called "phantom" because new rows appear like ghosts in a result set that should not have changed. Imagine you run SELECT COUNT(*) FROM orders WHERE date = today — you get 100. Before your transaction ends, another transaction inserts 5 new orders. You run the same query again within your transaction and get 105. Those 5 new rows are "phantoms" — they appeared out of nowhere in your supposedly isolated transaction.
Comparison: Isolation Levels
Level
Performance
Consistency
Read Uncommitted
Fastest
Weakest
Read Committed
Fast
Moderate
Repeatable Read
Medium
High
Serializable
Slowest
Strongest
Real-World Examples
Banking: Uses Serializable to ensure accuracy — cannot afford dirty reads or phantom reads on account balances.
E-commerce inventory:Repeatable Read to avoid double deductions — ensures the same stock count within a transaction.
Analytics:Read Committed is often sufficient — slight inconsistencies are acceptable for reporting.
Pros (higher isolation)
Ensures data correctness and consistency.
Prevents concurrency bugs.
Cons (higher isolation)
Higher isolation = lower performance.
Serializable can reduce throughput drastically.
Step-by-Step: Transaction Flow
Transaction A updates a row but hasn't committed
The row is modified in Transaction A's context but not yet visible to others (depending on isolation level).
Transaction B tries to read the same row
With Read Committed, B waits for A's commit. With Read Uncommitted, B sees the dirty (uncommitted) data.
Transaction A commits (or rolls back)
If A commits, B now sees the new value. If A rolls back, the change is undone. With Read Uncommitted, B may have already acted on data that was rolled back — this is a dirty read bug.
🏫
Locked Classroom Analogy
It is like locking a classroom while an exam is ongoing — higher isolation means fewer people can peek inside, but it slows things down. At the highest level (Serializable), the room is completely locked and no one can peek. At the lowest level (Read Uncommitted), anyone can walk in and read the answers — but they might be wrong.
🎯 Interview Tip
"What's the difference between Repeatable Read and Serializable?"
"Which isolation level would you use for a payment system?"
Tip: Link anomalies to isolation levels in your answer. Repeatable Read prevents dirty reads and non-repeatable reads but allows phantom reads. Serializable prevents all anomalies but at the cost of throughput. For payments, use Serializable or at minimum Repeatable Read to prevent double-spending.
Key Takeaways
Isolation levels balance data integrity and performance.
Serializable is safest but slowest.
Most real-world systems use Read Committed or Repeatable Read.
Module 4 Recap — Consistency vs. Availability
Data Consistency, Consistency Levels, Transaction Isolation
Module 4 Summary
Consistency ensures a unified view of data across nodes but often sacrifices availability and latency.
Consistency levels allow tuning this trade-off based on requirements — from strong to eventual, with quorum and causal in between.
Transaction isolation is crucial for concurrency safety within databases — from Read Uncommitted (fastest, riskiest) to Serializable (safest, slowest).
Real-world systems blend these techniques based on use case: e.g., DynamoDB prioritizes availability, Spanner prioritizes consistency.
MODULE 5: MESSAGE QUEUES (4 Topics)
Message queues are at the heart of scalable, event-driven, decoupled systems — and understanding them is essential for designing apps like WhatsApp, Uber, Amazon, or Netflix.
1. What is a Message Queue?
Why Not Just Use Synchronous API Calls Everywhere?
Imagine your checkout service needs to: charge the card, send a confirmation email, update inventory, and notify the warehouse. Synchronously, the user waits for ALL four steps to finish (slow, and if the email service is down, checkout fails). With a message queue, checkout only does the payment, then drops messages for the other three tasks — the user gets a response in milliseconds, and the other services process their tasks independently.
Definition / Introduction
A message queue is a communication mechanism that allows components of a system to exchange data asynchronously — by sending messages to a queue where they are processed later. Producers publish messages into the queue, and consumers retrieve and process them at their own pace.
Why it matters: Message queues enable decoupling, reliability, scalability, and fault tolerance in distributed systems.
Critical distinction: Kafka is NOT a message queue — it's a distributed commit log / event streaming platform. Key differences: (1) Kafka retains messages after consumption (consumers track their offset); RabbitMQ deletes after ACK. (2) Kafka supports message replay; traditional queues don't. (3) Kafka is append-only and immutable. Conflating Kafka with RabbitMQ in an interview tells the interviewer you have surface-level knowledge. Use "message broker" or "event streaming platform" for Kafka, and "message queue" for RabbitMQ/SQS.
When to Choose What — Decision Guide:
Scenario
Choose
Why
Simple task queue (send email, resize image)
RabbitMQ / SQS
Lightweight, built-in retries + DLQ, message deleted after processing. No need for replay or ordering.
Event log acts as source of truth. New services can replay history to bootstrap. Ordering per partition key (e.g., order_id).
Request-reply / RPC pattern
RabbitMQ
Native support for reply queues and correlation IDs. Kafka is not designed for request-reply.
Serverless / low-ops environment (AWS)
SQS + SNS
Fully managed, zero infrastructure, pay-per-message. Good enough for most use cases that don't need replay or strict ordering.
Complex routing (route by headers, priority queues)
RabbitMQ
Rich routing via exchanges (direct, topic, fanout, headers). Kafka has no built-in message routing — you'd need separate topics.
Rule of thumb: If you need a task queue → RabbitMQ/SQS. If you need an event log → Kafka. If you're on AWS and want simplicity → SQS. When in doubt in an interview, default to Kafka for event-driven architectures and SQS/RabbitMQ for job queues.
Kafka Concepts: Partitions and Consumer Groups
Partitions: A Kafka topic is split into partitions — the fundamental unit of parallelism. Messages within a partition are strictly ordered. Messages across partitions have no ordering guarantee. Choose your partition key wisely: partition by user_id means all events for one user are ordered.
Consumer Groups: Multiple consumers form a group. Kafka assigns each partition to exactly one consumer in the group. If you have 6 partitions and 3 consumers, each consumer handles 2 partitions. Add a 4th consumer? Kafka rebalances automatically. This is how you horizontally scale consumption while maintaining per-partition ordering.
Delivery Semantics — The Three Guarantees:
At-most-once: Messages may be lost but never duplicated. Fire-and-forget. Used when speed matters more than completeness (e.g., metrics/logging).
At-least-once: Messages are never lost but may be duplicated. Consumer processes message, then ACKs. If ACK is lost, message is re-delivered. This is the DEFAULT for most systems. Requires idempotent consumers.
Exactly-once: Messages are delivered exactly once — no loss, no duplicates. Very hard to achieve in distributed systems. Kafka supports it within its ecosystem using transactional producers and idempotent consumers. Outside Kafka, you approximate it with at-least-once + deduplication.
Interview tip: When asked "how do you ensure no duplicate processing?", say: "We use at-least-once delivery with idempotent consumers. Each message has a unique ID; consumers check a deduplication table before processing."
Retry / DLQ (Dead Letter Queue): Failed messages are retried or moved to DLQ.
ACK and NACK Explained
ACK (Acknowledge) = consumer tells the queue "I processed this message successfully, you can delete it." NACK (Negative Acknowledge) = consumer tells the queue "I could not process this, put it back for retry." If a message gets NACKed too many times (exceeds the retry limit), it is moved to the Dead Letter Queue.
What Is a Dead Letter Queue (DLQ)?
A Dead Letter Queue (DLQ) is a separate queue where messages that could not be processed after multiple retries are parked. Think of it as a "failed mail" bin at a post office. Engineers regularly inspect the DLQ to debug why processing failed — maybe a message had bad data, or a downstream service was down. They can then fix the issue and manually replay those messages.
What Is Idempotency and Why Does It Matter Here?
Idempotency means performing the same operation multiple times produces the same result as doing it once. Why does this matter? Networks are unreliable — if a consumer processes a message but the ACK gets lost, the queue will re-deliver the same message. Without idempotency, a payment service might charge a customer twice. With idempotency (e.g., using a unique transaction ID to check "did I already process this?"), the second delivery is safely ignored.
Message Queue: Producers send messages to a broker-managed queue, and consumers fetch and process them independently.
Real-World Examples
EXAMPLES
Netflix: Queues encode video files asynchronously.
Uber: Rides, payments, and notifications flow through Kafka pipelines.
Amazon: SQS handles billions of asynchronous operations.
Pros & Cons
Pros
Decouples services → easier scaling.
Improves reliability with retries.
Smooths traffic spikes (backpressure).
Enables asynchronous workflows.
Cons
Adds latency (not real-time by default).
Requires monitoring and failure handling.
Increases complexity.
Comparisons: Sync vs Async Communication
Feature
Synchronous
Asynchronous
Response Time
Immediate
Delayed
Coupling
Tight
Loose
Use Case
API calls
Event pipelines, notifications
Step-by-Step Flow
Producer Sends
Producer sends a message (e.g., “Order Placed”) to the queue.
Queue Stores
Queue stores the message.
Consumer Fetches
Consumer fetches and processes it.
ACK Sent
Consumer acknowledges (ACK).
Retry or DLQ
If ACK not received → retry or DLQ.
📬
Mailbox Analogy
A message queue is like a mailbox — senders drop letters (messages) without waiting for recipients to pick them up immediately.
🎯 Interview Tip
“Why would you use a message queue in a microservices system?” Tip: Mention decoupling, reliability, and asynchronous scaling.
Key Takeaways
Message queues decouple producers and consumers.
They improve scalability, resilience, and system performance.
Core tools: RabbitMQ, Kafka, SQS, ActiveMQ.
2. What is the Publisher-Subscriber Model?
Definition / Introduction
The Publisher-Subscriber (Pub/Sub) model is a messaging pattern where publishers send messages to a topic and subscribers receive them without knowing each other directly.
Why it matters: Pub/Sub enables broadcasting events to multiple services, making it ideal for event-driven systems.
Core Concepts
Topic: Named channel where messages are published.
Publisher: Sends messages to a topic.
Subscriber: Receives messages from a topic.
Fan-out: A single event is delivered to multiple subscribers.
Durable Subscriptions: Ensures subscribers get messages even if offline.
Pub/Sub Fan-Out: A publisher sends an event to a topic, and the broker delivers it to all subscribers independently.
Google Cloud Pub/Sub: Powers event streaming for thousands of microservices.
Slack: Message event → triggers webhook, analytics, and indexing services.
Pros & Cons
Pros
Decouples senders and receivers.
Scales horizontally to many subscribers.
Easy to add/remove subscribers without code changes.
Cons
Harder to guarantee ordering.
Retry logic is more complex.
Duplicate message handling often needed.
Comparisons: Queue vs Pub/Sub
Feature
Message Queue
Pub/Sub
Consumers
Usually 1
Many
Use Case
Task processing
Event broadcasting
Delivery
Point-to-point
One-to-many
Clarification: "Usually 1 Consumer" for Queues
A queue can have many worker instances (a worker pool), but each message is processed by only ONE worker — they compete for messages. In Pub/Sub, every subscriber gets its OWN COPY of every message — they do not compete, they all get everything.
Do Queues Guarantee Message Ordering?
Standard queues (like SQS Standard) do NOT guarantee order — messages might arrive out of sequence. If ordering matters (e.g., processing bank transactions), use a FIFO (First-In-First-Out) queue, which guarantees messages are delivered in exactly the order they were sent. FIFO queues are slower but order-safe.
Step-by-Step Flow
Publish Event
Publisher sends event → Topic.
Broker Delivers
Broker stores and delivers message.
Subscribers Receive
All active subscribers receive the event.
Independent Processing
Each subscriber processes independently.
📰
Newsletter Analogy
It’s like a newsletter: one sender publishes updates, and every subscriber gets a copy.
🎯 Interview Tip
“How is Pub/Sub different from a traditional message queue?” Tip: Emphasize one-to-many delivery and loose coupling.
Netflix: “Video played” triggers analytics, recommendations, and billing.
Pros & Cons
Pros
Highly scalable and decoupled.
Easy to add new event consumers.
Real-time responsiveness.
Cons
Harder to trace data flow.
Event ordering and idempotency can be tricky.
Debugging is complex.
Comparisons: Event-Driven vs Request-Driven
Feature
Event-Driven
Request-Driven
Coupling
Loose
Tight
Scalability
High
Moderate
Response Time
Reactive
Synchronous
Step-by-Step Flow
Event Occurs
An event occurs (e.g., payment received).
Broker Captures
Event broker captures and stores the event.
Subscribers React
Subscribers consume and react (e.g., send confirmation email, update ledger).
Downstream Triggers
Downstream services trigger more events if needed.
🃏
Domino Effect Analogy
Event-driven systems are like a domino effect — one event triggers multiple reactions without the initiator knowing.
🎯 Interview Tip
“How would you design an event-driven architecture for an e-commerce platform?” Tip: Talk about event producers, brokers, consumers, and event sourcing.
Key Takeaways
Event-driven systems scale better and are more reactive.
Best suited for asynchronous, loosely-coupled workflows.
Central to microservice and serverless design patterns.
4. Database as a Message Queue
Definition / Introduction
A database can be used as a pseudo message queue by storing messages in a table and polling them for processing.
Why it matters: In simple systems or legacy architectures, this can be a temporary solution without dedicated message brokers.
Core Concepts
Message Table: Stores tasks or messages.
Polling: Workers query for new messages periodically.
Status Fields: Track whether a message is pending, processing, or complete.
Leasing: Locks a message to one worker to avoid duplication.
How leasing works: When a worker picks up a message, it sets a locked_until timestamp on that row (e.g., "locked until 10:05:30"). Other workers skip rows that are still locked. If the original worker crashes and doesn't finish before the lock expires, another worker can pick it up. This prevents both duplicate processing and lost messages.
Real-World Examples
EXAMPLES
Early-stage startups: Use PostgreSQL or MySQL for background jobs.
Airbnb (early days): Used a DB queue before moving to RabbitMQ.
ETL pipelines: Batch jobs often use DB queues for processing.
Pros & Cons
Pros
Easy to implement (no new infrastructure).
Works well for low-volume or batch workloads.
Transactions ensure message durability.
Cons
Polling causes latency and resource waste.
Poor scalability compared to real queues.
Harder to guarantee ordering and retry.
Comparisons: DB Queue vs Dedicated MQ
Feature
Database Queue
Message Queue
Setup Complexity
Low
Medium
Scalability
Limited
High
Latency
Higher
Lower
Reliability
Medium
High
Step-by-Step Flow
Insert Message
Producer inserts a message into a messages table.
Worker Polls
Worker polls table every few seconds.
Lock & Process
Worker locks and processes message.
Update Status
Updates status to “done” or “failed.”
📋
To-Do List Analogy
It’s like checking a to-do list every few minutes instead of being notified instantly.
🎯 Interview Tip
“When would you use a database as a message queue?” Tip: Say: Only for small-scale, low-throughput scenarios.
Key Takeaways
DB queues are a simple solution but not scalable long-term.
Always migrate to dedicated MQ systems as volume grows.
Useful for MVPs or batch job scheduling.
The Transactional Outbox Pattern:
Problem: You need to update a database AND publish an event to Kafka, but they're separate systems — you can't do both in one transaction. If the DB write succeeds but Kafka publish fails, your systems are out of sync.
Solution: Instead of publishing to Kafka directly, write the event to an "outbox" table in the SAME database transaction as your business data. A separate process (CDC tool like Debezium, or a poller) reads the outbox table and publishes events to Kafka.
Since the business data and the outbox entry are in the same DB transaction, they either both succeed or both fail — guaranteeing consistency. The CDC process handles reliable delivery to Kafka separately.
This is the standard pattern for reliable event publishing in microservices. Know it by name — interviewers ask about it specifically.
Final Recap — Module 5: Message Queues
Core messaging patterns every system designer must know
Event-Driven Systems are the backbone of scalable, reactive architectures.
Databases as Queues are fine for small-scale but should eventually be replaced by dedicated messaging systems.
MODULE 6: DEVOPS CONCEPTS (6 Topics)
This module covers the infrastructure-level principles that keep large-scale systems like Netflix, Google, and Uber running reliably in production — even under failures, high load, and unexpected events. These topics are highly valued in interviews because they test how you’d design for reliability, fault tolerance, and operational excellence.
1. What is a Single Point of Failure (SPOF)?
Definition / Introduction
A Single Point of Failure (SPOF) is any component in a system whose failure will cause the entire system (or a major part of it) to stop working.
Why it matters: SPOFs are the biggest enemies of availability. In large-scale systems, redundancy and failover mechanisms are designed specifically to eliminate SPOFs.
Core Concepts
Examples of SPOFs:
A single database server with no replica.
One load balancer without a backup.
A single region deployment without failover.
Fault Tolerance: System’s ability to continue functioning despite a failure.
Redundancy: Having multiple instances of a critical component.
Real-World Examples
EXAMPLES
AWS Multi-AZ RDS: Avoids database SPOFs by replicating across availability zones.
Google Cloud Load Balancing: Uses multiple regional load balancers to avoid SPOF at the network layer.
Netflix: Deploys services in multiple AWS regions to avoid single-region failures.
Continuously test failure scenarios (chaos testing).
What Is Chaos Testing?
Chaos testing means deliberately injecting failures into your system to see if it recovers gracefully. Netflix famously runs "Chaos Monkey" — a tool that randomly kills production servers during business hours. If the system handles it without users noticing, the redundancy works. If something breaks, the team fixes it before a real failure hits.
✈️
Single Engine Plane Analogy
A SPOF is like a single engine on a plane — if it fails, the whole plane goes down. Two engines mean one can fail without disaster.
🎯 Interview Tip
“What is a SPOF, and how do you design against it?” Tip: Mention redundancy, failover, and replication.
Key Takeaways
Always design systems with no single points of failure.
Use redundancy and failover to improve resilience.
Test failure scenarios proactively.
2. What are Containers?
Definition / Introduction
A container is a lightweight, portable, and isolated runtime environment that packages an application and its dependencies together.
Why it matters: Containers make systems easier to deploy, scale, and manage across different environments.
Core Concepts
Container Image: A snapshot of an app with its dependencies.
Immutability: Containers are built once and run anywhere without configuration drift.
Precision matters: Docker is a container platform (runtime + tooling). The actual container runtime underneath is containerd or runc. Since Kubernetes 1.24+, Docker was deprecated as a container runtime — Kubernetes now uses containerd directly. Knowing this distinction shows you understand the container ecosystem beyond just "docker run."
What Does Kubernetes Actually Do?
It automatically: (1) starts your containers across multiple machines, (2) restarts containers that crash, (3) scales up/down based on load (need more containers? Kubernetes spins them up), (4) routes traffic to healthy containers, and (5) rolls out updates without downtime. Think of it as a robot sysadmin that manages your containers 24/7 so you do not have to SSH into machines manually.
What Does "Immutable" Mean for Containers?
Once you build a container image, you never modify it — no SSH-ing into a running container to change code. If you need to update something, you build a NEW image and deploy it. This guarantees that what you tested is exactly what runs in production — no "it works on my machine" surprises.
Real-World Examples
EXAMPLES
Netflix: Uses containers for microservices deployment.
Spotify: Uses Docker + Kubernetes for scalable backend services.
Google: Runs billions of containers daily using Borg (internal Kubernetes ancestor).
Pros & Cons
Pros
Portability across environments.
Faster deployment and scaling.
Resource isolation.
Cons
Added complexity with orchestration.
Security risks if not properly sandboxed.
Comparisons: Containers vs Virtual Machines
Feature
Containers
VMs
Startup Time
Seconds
Minutes
Resource Usage
Lightweight
Heavy
Isolation
Process-level
Full OS-level
Portability
High
Medium
Containers vs VMs: VMs include a full Guest OS per instance (heavy), while containers share the Host OS via a container engine (lightweight).
Step-by-Step (Deployment Flow)
Build Image
Build a Docker image.
Push to Registry
Push to a container registry.
Deploy
Deploy containers on servers or Kubernetes clusters.
Scale
Scale up/down by running more instances.
🚢
Shipping Container Analogy
A container is like a shipping container — it holds everything an app needs and can be shipped anywhere without modification.
🎯 Interview Tip
“Why are containers preferred over VMs in modern deployments?” Tip: Highlight portability, immutability, and speed.
Key Takeaways
Containers are essential for microservices and cloud-native systems.
Kubernetes is the industry standard for container orchestration.
Immutability simplifies deployment pipelines.
3. What is Service Discovery and Heartbeats?
Definition / Introduction
Service Discovery is the process of automatically locating network services in a distributed system without hardcoding addresses. Heartbeats are periodic signals sent by services to indicate they are alive.
Why it matters: In dynamic environments (e.g., containers, autoscaling), services constantly change IPs — discovery ensures they can still find and talk to each other.
Core Concepts
Service Registry: A database storing the locations of active services (e.g., Consul, etcd).
Server-side Discovery: A load balancer queries the registry.
Heartbeats: Regular signals to update health status in the registry.
Why Is Client-Side Discovery More Complex?
Because every client application must contain the logic to: query the service registry, pick a healthy instance from the list, implement load balancing (round-robin, least connections, etc.), and handle failures (retry a different instance). With server-side discovery, all this complexity lives in the load balancer — clients just send requests to one address.
Real-World Examples
EXAMPLES
Netflix Eureka: Microservices use Eureka for discovery.
Kubernetes: Built-in DNS-based service discovery.
Consul / Zookeeper: Popular tools for dynamic environments.
Pros & Cons
Pros
Automatic scaling and resilience.
Simplified network configuration.
Better fault detection with heartbeats.
Cons
Additional infrastructure complexity.
Registry failure can impact the system.
Comparisons: Client vs Server Discovery
Feature
Client-side
Server-side
Routing
Done by client
Done by proxy/load balancer
Complexity
Higher (client logic)
Lower (centralized)
Flexibility
High
Medium
Step-by-Step Flow
Register
Service starts and registers itself.
Heartbeat
Heartbeats sent periodically to indicate health.
Discover
Clients query the registry to discover endpoints.
Cleanup
Dead services are removed automatically.
📞
Phone Directory Analogy
Like a phone directory where services register their numbers, and heartbeats are regular “I’m alive” check-ins.
🎯 Interview Tip
“How does service discovery work in microservices?” Tip: Mention registry, DNS, and health checks.
Key Takeaways
Service discovery enables dynamic scaling.
Heartbeats help detect and remove failed services.
Essential for containerized and microservice environments.
4. How to Avoid Cascading Failures?
Definition / Introduction
A cascading failure occurs when a single component’s failure triggers a chain reaction, causing widespread outages.
Why it matters: In large-scale systems, even minor failures can snowball into massive downtime without protective mechanisms.
Core Concepts
Circuit Breakers: Temporarily stop calls to failing services to prevent overload.
Bulkheads: Isolate components so one failure doesn’t affect others.
Rate Limiting: Prevents overload during spikes.
Graceful Degradation: Reduce functionality instead of complete failure.
How Does a Software Bulkhead Work?
Give each service its own dedicated thread pool or resource quota. Example: Your app has a Payment Service (10 threads) and a Notification Service (5 threads). If Notifications gets overwhelmed and uses all its threads, Payment still has its dedicated 10 threads untouched — the failure is contained. Without bulkheads, both services share one pool — Notifications could eat all threads and starve Payment.
Real-World Examples
EXAMPLES
Netflix Hystrix: Circuit breaker library to prevent cascading failures.
AWS API Gateway: Enforces rate limits to protect backend services.
Gmail: Disables certain features during partial outages instead of going offline.
Update (2026): Netflix Hystrix has been in maintenance mode since 2018 and is no longer recommended. Modern alternatives: Resilience4j (for Java), or native service mesh circuit breaking via Istio/Envoy sidecars. Citing deprecated libraries in interviews signals outdated knowledge.
Pros & Cons
Pros
Improves fault isolation.
Prevents total outages.
Increases resilience and stability.
Cons
Adds complexity to architecture.
Requires careful tuning and monitoring.
Strategies to Prevent Failures
Strategy
Purpose
Circuit Breaker
Stop repeated failing calls
Bulkhead
Isolate failures
Timeout
Prevent resource blocking
Backpressure
Slow down producers under load
What Is Backpressure?
Backpressure is when a downstream system signals upstream to slow down because it cannot keep up. Example: Your message queue is growing faster than consumers can drain it. With backpressure, the queue tells producers "slow down, I am at 90% capacity." Without it, the queue overflows and messages get dropped. Think of it like a traffic light at a highway on-ramp — it meters traffic so the highway does not jam.
Step-by-Step (Circuit Breaker Flow)
Failures Detected
Service A calls Service B → failures occur.
Circuit Opens
Circuit breaker opens → stops further calls.
Half-Open Retry
After cooldown, it tries again (half-open).
Resume or Stay Open
If stable, closes and resumes normal flow.
Circuit Breaker State Machine: Closed (normal) transitions to Open on repeated failures, then to Half-Open after a timeout. Success resets to Closed; failure returns to Open.
⚡
Electrical Circuit Breaker Analogy
Like an electrical circuit breaker — cutting power prevents further damage.
🎯 Interview Tip
“How would you prevent cascading failures in a payment system?” Tip: Talk about circuit breakers, bulkheads, and rate limiting.
Key Takeaways
Cascading failures are dangerous and must be contained.
Circuit breakers, bulkheads, and graceful degradation are essential defenses.
5. Anomaly Detection in Distributed Systems
Definition / Introduction
Anomaly detection is the process of automatically identifying unusual patterns or behaviors in system metrics that could indicate failures, security issues, or performance degradation.
Why it matters: Early detection prevents outages and improves reliability.
Static Thresholds: Fixed upper/lower limits for alerts.
Dynamic Baselines: Adaptive thresholds based on historical data.
Machine Learning Models: Used for predictive anomaly detection.
What is a dynamic baseline? Instead of a fixed threshold (alert if CPU > 80%), a dynamic baseline learns your system's normal behavior over time. It knows that traffic spikes 3x every weekday morning and 5x on Black Friday — so it only alerts when the pattern itself is unusual, not just when a number crosses a fixed line. This dramatically reduces false alarms compared to static thresholds.
Real-World Examples
EXAMPLES
Netflix Atlas: Real-time anomaly detection on millions of time-series metrics.
Google SRE: Uses ML for detecting abnormal latency spikes.
Datadog / Prometheus: Monitor distributed systems and trigger alerts.
Pros & Cons
Pros
Early warning for failures.
Reduces MTTR (Mean Time to Recovery).
Helps prevent cascading issues.
Cons
False positives/negatives.
Requires careful tuning and labeling.
Comparisons: Static vs Dynamic Detection
Feature
Static
Dynamic
Setup
Simple
Complex
Accuracy
Lower
Higher
Adaptability
Low
High
Step-by-Step Flow
Collect Metrics
Collect system metrics continuously.
Analyze
Analyze with thresholds or ML models.
Trigger Alert
Trigger alert if anomaly detected.
Remediate
Automate remediation or notify team.
🚨
Smoke Detector Analogy
Like a smoke detector — it notices unusual patterns (smoke) before a fire (system outage) happens.
🎯 Interview Tip
“How would you detect anomalies in a distributed system?” Tip: Mention metrics, baselines, thresholds, and ML.
Key Takeaways
Early detection saves systems from major outages.
ML-driven detection is powerful but complex.
Monitoring + alerting is critical in production.
6. Distributed Rate Limiting
Definition / Introduction
Rate limiting controls the number of requests a client or service can make within a time window. Distributed rate limiting ensures this control works across multiple servers or regions in a cluster.
Why it matters: It protects systems from abuse, spikes, and DoS attacks.
Core Concepts
Token Bucket / Leaky Bucket: Popular algorithms for rate limiting.
Sliding Window: Tracks requests in a rolling time frame.
Distributed Counters: Use Redis or a shared store to track usage.
Client-side vs Server-side: Rate limits can be enforced on either end.
Real-World Examples
EXAMPLES
GitHub API: Limits requests per hour per token.
Stripe: Enforces per-user rate limits to protect payment APIs.
Cloudflare: Distributed rate limiting across global edge nodes.
Pros & Cons
Pros
Protects against abuse and denial-of-service.
Smooths traffic spikes.
Prevents resource starvation.
Cons
Slight performance overhead.
Harder to coordinate globally.
Comparison: Algorithms
Algorithm
Behavior
Use Case
Token Bucket
Bursts allowed
APIs with flexible burstiness
Leaky Bucket
Steady flow
Payment gateways
Sliding Window
Accurate rate tracking
Login attempts, security
How Does a Leaky Bucket Work?
Imagine a bucket with a small hole at the bottom. Requests pour into the top. They "leak out" (get processed) at a constant rate through the hole, regardless of how fast they arrive. If requests arrive faster than they leak out, the bucket fills up. Once full, new requests are rejected. Unlike Token Bucket (which allows short bursts), Leaky Bucket enforces a perfectly steady output rate.
Step-by-Step (Token Bucket)
Refill Tokens
Bucket refills with tokens at a fixed rate.
Consume Token
Each request consumes a token.
Reject or Delay
If no tokens → request rejected or delayed.
Token Bucket: Tokens refill at a fixed rate. Each request consumes a token; when the bucket is empty, requests are rejected or delayed.
🚧
Turnstile Analogy
Think of a turnstile at a subway — it lets people through at a fixed pace, preventing overcrowding.
🎯 Interview Tip
“How would you implement global API rate limiting?” Tip: Mention distributed counters, Redis, token bucket, and sliding window.
Key Takeaways
Rate limiting is vital for API stability and security.
Distributed coordination ensures consistency across nodes.
Combine with backpressure and throttling for maximum safety.
The Three Pillars of Observability:
Before you can detect anomalies, you need observability — the ability to understand your system's internal state from its external outputs.
Logs: Timestamped records of discrete events. Use structured logging (JSON) so logs are machine-parseable. Tools: ELK Stack (Elasticsearch, Logstash, Kibana), Loki, Splunk.
Metrics: Numeric measurements aggregated over time. CPU usage, request count, error rate, p99 latency. Tools: Prometheus + Grafana, Datadog, CloudWatch.
Traces: End-to-end tracking of a single request as it flows through multiple services. Each service adds a "span" to the trace. If a request is slow, the trace shows exactly which service took too long. Tools: Jaeger, Zipkin, OpenTelemetry.
Correlation IDs: Every incoming request gets a unique ID (e.g., UUID in the X-Request-ID header) that's passed to every downstream service call. All logs, metrics, and traces for that request share this ID — so you can reconstruct the entire journey of a single request across 15 services.
SLOs, SLIs, SLAs: SLI (Service Level Indicator) = a metric you measure (e.g., p99 latency). SLO (Service Level Objective) = the target you set (e.g., p99 < 200ms). SLA (Service Level Agreement) = the contractual promise to customers (e.g., 99.9% uptime or we pay penalties). Error budgets: if your SLO is 99.9%, you have a 0.1% error budget — that's ~43 minutes of downtime per month.
Final Recap — Module 6: DevOps Concepts
Infrastructure-level principles for production reliability
Module 6 Summary
SPOFs must be eliminated with redundancy and failover.
Containers and orchestration simplify deployment and scaling.
Service Discovery and Heartbeats keep microservices connected dynamically.
Cascading Failures are prevented with circuit breakers, bulkheads, and rate limiting.
Distributed Rate Limiting protects APIs from abuse and overload.
MODULE 7: CACHING (4 Topics)
Caching is everywhere — from database queries and API responses to CDN edges and browser memory. It’s one of the most powerful ways to reduce latency, scale systems, and improve user experience. Mastering it is crucial for both interviews and real-world architecture.
1. What is Distributed Caching?
Definition / Introduction
Distributed caching is a technique where cache data is shared across multiple servers or nodes, allowing many clients and services to access the same cached data at scale.
Why it matters: In large systems, a single cache node becomes a bottleneck. A distributed cache ensures scalability, fault tolerance, and high availability.
Core Concepts
Cache Cluster: Multiple cache nodes working together.
Data Partitioning: Keys are distributed among nodes (e.g., via consistent hashing). (Consistent hashing was covered in detail in Module 2 — it distributes data across cache nodes so that adding or removing a node only remaps a small fraction of keys.)
Replication: Cache data can be duplicated for redundancy.
Eviction Policies: Decide which items to remove when memory is full (e.g., LRU, LFU).
Coherency: Ensuring cache remains updated when the source data changes.
How Is Cache Coherency Maintained?
Common approaches: (1) TTL-based: each cached item has an expiration time — after it expires, the next request fetches fresh data from the database. (2) Event-based invalidation: when data changes in the database, it publishes an event (via a message queue) that tells all cache nodes to delete or refresh that item. (3) Write-through: every write goes to both cache and database simultaneously, so they’re always in sync. Each approach trades freshness for performance.
Real-World Examples
EXAMPLES
Netflix: Uses EVCache (a distributed Memcached layer) to serve personalized recommendations.
Twitter: Redis clusters handle timeline caching.
Amazon: DynamoDB Accelerator (DAX) caches queries across distributed nodes.
Pros & Cons
Pros
Reduces database load significantly.
Speeds up reads and improves latency.
Scales horizontally as load increases.
Cons
Cache consistency is complex.
Network hops can increase latency slightly.
Requires careful partitioning and failover handling.
Comparisons: Local vs Distributed Cache
Feature
Local Cache
Distributed Cache
Scope
Single server
Cluster-wide
Scalability
Limited
High
Consistency
Easy
Complex
Latency
Lowest
Low (but network cost)
Step-by-Step (Read Path)
Client Request
Client requests data.
Check Cache
System checks distributed cache cluster.
Cache Hit
If cache hit → data returned immediately.
Cache Miss
If cache miss → fetch from DB → write to cache → return to client.
📚
Shared Library System Analogy
Think of distributed caching as a shared library system — many branches store books (data), so users across cities can access them quickly without going to the central archive (database).
🎯 Interview Tip
“How would you implement a distributed cache for a read-heavy system?” Tip: Discuss partitioning, replication, eviction policy, and consistency.
Key Takeaways
Distributed caching is essential for scaling read-heavy systems.
Trade-offs involve consistency, eviction, and network latency.
Common tools: Redis Cluster, Memcached, Hazelcast.
2. What are Content Delivery Networks (CDNs)?
Definition / Introduction
A Content Delivery Network (CDN) is a globally distributed network of servers that caches and serves static or semi-static content from locations geographically close to users.
Why it matters: CDNs significantly reduce latency, bandwidth usage, and server load, improving performance and reliability for users worldwide.
Core Concepts
Edge Servers: CDN servers located near users.
Origin Server: The main server with original content.
Cache Invalidation: Removing or refreshing stale data.
Time-to-Live (TTL): How long cached data stays before refresh.
Geo DNS / Anycast: Routes requests to nearest edge location.
What Happens When TTL Expires?
The cached item is marked as “stale.” On the next user request for that item, the edge server sends a conditional request to the origin (“I have version X — is it still current?”). If the origin says yes (304 Not Modified), the edge refreshes the TTL without re-downloading. If the data has changed, the origin sends the new version. The user who triggers this re-validation experiences slightly higher latency, but all subsequent users get the fresh cached copy.
How Do You Invalidate CDN Caches?
Three common methods: (1) Purge: explicitly tell the CDN “delete this URL from all edge servers” (e.g., Cloudflare purge API). (2) Versioned URLs: instead of updating /image.jpg, deploy /image-v2.jpg — since the URL is different, the old cache is irrelevant. (3) Short TTL: set a low TTL (e.g., 60 seconds) so stale data auto-expires quickly. Most production systems use versioned URLs for static assets and purge APIs for emergency updates.
Geo-DNS vs Anycast
Geo-DNS routes users to the nearest server based on their geographic location — your DNS query from Mumbai returns the IP of an Indian data center, not one in the US. Anycast goes further: the same IP address is announced from multiple locations worldwide, and the internet’s routing protocol automatically directs your packets to the nearest one. Think of it as: Geo-DNS = choosing which phone number to call based on your area code; Anycast = one phone number that automatically connects you to the nearest office.
CDN: Edge servers cache content close to users. A cache hit serves instantly; a cache miss fetches from the origin server.
Real-World Examples
EXAMPLES
Netflix: Uses Open Connect CDN to deliver video content globally.
YouTube: Serves cached videos from edge POPs (Points of Presence).
Cloudflare / Akamai / Fastly: Offer global CDN services for static assets, APIs, and HTML pages.
Pros & Cons
Pros
Drastically reduces latency.
Offloads origin servers.
Improves availability during traffic spikes.
Cons
Slight propagation delay for updates.
Costs for large-scale CDN services.
Dynamic content caching is complex.
Comparisons: CDN vs Traditional Caching
Feature
CDN
Application Cache
Scope
Global
Local/Cluster
Target
Static/semi-static
Any data
TTL
Long
Short
Latency Reduction
Geographical
Logical (memory/database)
Step-by-Step Flow
User Request
User requests image.jpg.
DNS Routes
DNS routes request to nearest CDN edge.
Cache Hit
If cache hit → serve instantly.
Cache Miss
If cache miss → fetch from origin → cache it → serve.
🏪
Franchise Network Analogy
A CDN is like a franchise network — instead of shipping every product from the main warehouse, local branches (edges) serve nearby customers instantly.
🎯 Interview Tip
“How would you design a CDN for a video platform?” Tip: Mention edge caching, TTL, cache invalidation, and geo-routing.
Key Takeaways
CDNs are crucial for reducing latency in global systems.
Best suited for static, media, and precomputed content.
TTL and invalidation policies determine freshness.
3. Write Policies
Definition / Introduction
Write policies define how data is written to cache and the underlying data store. They directly affect data consistency, latency, and durability.
Why it matters: Write strategies balance between performance and reliability in cache-backed architectures.
Core Concepts
Write-Through: Data written to cache and database simultaneously.
Write-Back (Write-Behind): Data written to cache first, and database is updated asynchronously.
Write-Around: Data written directly to the database, skipping the cache.
Write Policies: Write-Through writes to cache and DB simultaneously; Write-Back writes to cache first, then DB asynchronously; Write-Around writes directly to DB, bypassing the cache.
Write-Back: High-write systems like analytics pipelines.
Write-Around: Large batch inserts (avoids polluting cache).
Pros & Cons
Policy
Pros
Cons
Write-Through
Strong consistency
Slower writes
Write-Back
Fast writes, reduced DB load
Risk of data loss if cache fails
Write-Around
Avoids cache pollution
Higher read latency on next access
Write Policies Comparison
Feature
Write-Through
Write-Back
Write-Around
Write Speed
Slow
Fast
Medium
Consistency
High
Eventually consistent
High
Risk
Low
Medium-High
Low
Why is Write-Around "Medium" speed? Write-Around skips the cache and writes directly to the database — so it's roughly as fast as a normal DB write. It's labeled "Medium" (not "Fast") because it doesn't benefit from the cache absorbing writes like Write-Back does, but it's not as slow as Write-Through which must update both cache and DB synchronously.
Step-by-Step (Write-Through Flow)
App Writes to Cache
App writes to cache.
Sync to DB
Cache writes to DB synchronously.
Confirmation
Confirmation returned to client.
Analogies
📝
Ledger Analogies
Write-Through: Writing a cheque and updating the ledger immediately. Write-Back: Writing a note and updating the ledger later. Write-Around: Writing directly to the ledger without the note.
🎯 Interview Tip
“When would you choose write-back over write-through?” Tip: Discuss performance vs durability trade-offs.
Key Takeaways
Write policy choice depends on workload patterns.
Write-back is fast but risky; write-through is safe but slow.
Write-around is ideal when write frequency is high but read frequency is low.
4. Replacement Policies
Definition / Introduction
Replacement policies determine which cached data should be removed when the cache is full.
Why it matters: Proper eviction ensures the cache stores the most valuable data, improving hit rates and performance.
Core Policies
LRU (Least Recently Used): Removes data that hasn’t been accessed recently.
LFU (Least Frequently Used): Removes data with the fewest accesses.
FIFO (First In, First Out): Removes oldest data.
Random: Removes a random entry (simple but unpredictable).
ARC (Adaptive Replacement Cache): Balances recency and frequency dynamically.
How ARC Works
ARC keeps two lists: one for recently accessed items (like LRU) and one for frequently accessed items (like LFU). It dynamically adjusts the size of each list based on workload. If the system notices it’s evicting recently-used items that are needed again, it grows the recency list. If frequently-used items keep getting evicted, it grows the frequency list. This self-tuning makes ARC better than either LRU or LFU alone.
ARC's secret weapon — ghost lists: ARC also tracks "ghost entries" — items that were recently evicted from each list. If a ghost entry is requested again, ARC knows that list was too small and grows it. This self-tuning based on eviction patterns is what makes ARC smarter than simply combining LRU and LFU.
Real-World Examples
EXAMPLES
Redis: Supports LRU, LFU, and random eviction.
Web browsers: Use LRU for caching pages.
Databases: Use ARC to optimize cache pages.
Pros & Cons
Policy
Pros
Cons
LRU
Easy, effective for temporal locality
Might evict frequently used but recently idle data
LFU
Good for long-term popular items
Complex to implement
FIFO
Simple
Ignores usage patterns
ARC
Adaptive and efficient
Complex, more memory
LRU vs LFU Comparison
Feature
LRU
LFU
Based On
Recency
Frequency
Complexity
Low
Higher
Best For
Temporal workloads
Repetitive workloads
Step-by-Step (LRU Flow)
Add Data
New data added to cache.
Evict LRU
Cache full → evict least recently used item.
Update Order
Update usage order on every access.
🧊
Fridge Cleaning Analogy
LRU is like cleaning out your fridge — you throw away the items you haven’t touched in the longest time.
🎯 Interview Tip
“Which cache eviction policy would you use for a news feed service?” Tip: Pick LFU for stable hot data or LRU for rapidly changing data.
Key Takeaways
Replacement policy impacts cache hit ratio significantly.
LRU is the most commonly used due to simplicity and effectiveness.
Advanced systems use adaptive approaches like ARC for mixed workloads.
Cache Stampede (Thundering Herd Problem):
When a popular cache key expires, hundreds of requests simultaneously hit the database — overwhelming it. This is called a cache stampede or thundering herd.
Solutions:
Request coalescing (single-flight): Only one request goes to the DB; all others wait for that result. The first request "locks" the key; when it returns, all waiting requests get the cached value.
Probabilistic early expiration: Instead of all copies expiring at exactly the same time, each copy randomly expires slightly before the TTL. This staggers the re-fetches.
Background refresh: A background job refreshes popular keys before they expire, so the cache never goes cold.
Locking: Use a distributed lock (Redis SETNX) so only one process can refresh a key at a time.
Interview tip: If an interviewer asks "what happens when a hot key expires?", this is what they're probing for. Cache stampede is one of the top 5 caching interview questions.
Final Recap — Module 7: Caching
Caching strategies that power every high-performance system
Module 7 Summary
Distributed Caching shares data across nodes for scalability and performance.
CDNs cache content at the edge to minimize latency for global users.
Write Policies balance consistency, durability, and speed.
Replacement Policies ensure optimal use of cache space and maximize hit rates.
Module 8: Microservices
2 Topics — Microservices vs Monoliths & Migration Strategies
1. Microservices vs. Monoliths
Definition / Introduction
A monolith is a single, unified codebase where all features and services run together as one application.
Microservices are a collection of independent, loosely coupled services, each responsible for a specific business function, communicating via APIs or message queues.
Why it matters: Choosing between these two architectures defines how your system scales, deploys, and evolves over time.
Monolith vs Microservices Comparison
Feature
Monolith
Microservices
Architecture
Single unit
Multiple small services
Deployment
All-or-nothing
Independent
Scalability
Vertical
Horizontal (per service)
Communication
In-process calls
Network (API/Queue)
Coupling
Tightly coupled
Loosely coupled
Correction: Monoliths CAN scale horizontally — you can run multiple instances behind a load balancer. The real limitation is that you must scale the entire application even if only one feature needs more capacity. It's not that horizontal scaling is impossible — it's that it's less efficient than scaling individual microservices independently.
Monolith: all layers in one unit. Microservices: independent services communicating via APIs, each owning its own data store.
Core Concepts
Service Independence: Microservices can be built, deployed, and scaled individually.
Bounded Context: Each microservice owns a specific domain (e.g., Auth, Payment, Notifications).
Communication: REST, gRPC, Kafka, or internal message buses.
Polyglot Persistence: Each service can choose its own database type and schema.
Quick definitions:
gRPC: A high-performance RPC (Remote Procedure Call) framework by Google that uses binary serialization (Protocol Buffers) instead of JSON. Much faster than REST for service-to-service communication. Covered in detail in Module 9.
Kafka: A distributed event streaming platform. Think of it as a super-powered message queue that can handle millions of events per second, stores them durably, and lets multiple consumers read independently. Used when services need to communicate asynchronously at massive scale.
Polyglot Persistence: Proceed with Caution
Polyglot persistence sounds great in theory, but it creates real challenges: if your Auth service uses PostgreSQL and your Orders service uses MongoDB, how do you query “all orders by verified users”? You can’t just JOIN across databases. You’ll need API calls between services, data duplication, or event-driven sync — all of which add complexity. Use different databases only when the benefit clearly outweighs this cost.
Bounded Context Explained
A bounded context means each microservice owns a specific “domain” with its own data, rules, and terminology — and those rules don’t leak into other services. Example: in an e-commerce system, the “Shipping” service has its own definition of an “Order” (tracking number, weight, delivery address), while the “Billing” service has a different definition (amount, payment method, invoice). They don’t share a single “Order” table — each maintains its own view. This prevents one service’s changes from breaking another.
The Hardest Part of Microservices: Data
When Service A needs data that Service B owns, you have three options: (1) API call: A calls B’s API at runtime (adds latency, creates coupling). (2) Data duplication: A keeps its own copy of B’s data, updated via events (faster reads, but now you have eventual consistency). (3) Composite API: a new service aggregates data from A and B (adds another service to manage). There’s no perfect answer — each option has trade-offs.
Rule of thumb: Use API calls when data freshness is critical and latency is acceptable. Use data duplication (via events) when read latency matters most and eventual consistency is okay. Use a composite/aggregator service when you need to join data from many services for a specific use case (like a dashboard).
The Saga Pattern — Distributed Transactions Without 2PC:
When a business operation spans multiple microservices (e.g., Order → Payment → Inventory → Shipping), you can't use a traditional database transaction. The Saga pattern breaks it into a sequence of local transactions, each with a compensating action if something fails.
Two flavors:
Choreography: Each service publishes an event after its local transaction. The next service listens and acts. No central coordinator. Simple but hard to track. Example: Order Service publishes "OrderCreated" → Payment Service listens, charges card, publishes "PaymentCompleted" → Inventory Service listens, reserves stock.
Orchestration: A central Saga Orchestrator tells each service what to do and when. If Payment fails, the orchestrator tells Order Service to run its compensating action (cancel order). Easier to understand and debug, but the orchestrator is a single point of coordination.
Compensating actions: Unlike rollback in a DB transaction, compensating actions are business-level undos. "Refund payment" compensates "charge card." "Release inventory" compensates "reserve stock." These must be idempotent.
Interview tip: The Saga pattern is one of the most asked microservices questions. Know both choreography and orchestration, and be ready to explain when you'd choose each (choreography for simple flows, orchestration for complex multi-step workflows).
CQRS — Command Query Responsibility Segregation:
Separate your read path and write path into different models (and potentially different databases). Writes go to a normalized write store optimized for consistency. Reads go to a denormalized read store optimized for query speed.
Why? Most systems are read-heavy (100:1 read-to-write ratio). By separating the paths, you can scale reads independently (add read replicas, caches) without affecting write performance.
Example: Instagram's feed. Writes go to a relational database (new post created). A separate process fan-outs the post to followers' pre-computed feed caches (read-optimized). When you open Instagram, the app reads from the cache — it never queries the write database directly.
Trade-off: CQRS adds complexity (two models to maintain, eventual consistency between them). Only use it when read and write patterns are significantly different.
Real-World Examples
Netflix: Microservices architecture powers content delivery, recommendations, billing, etc.
Amazon: Transitioned from a monolithic codebase to hundreds of independent microservices.
Uber: Broke its original monolith into domain-based services like Driver, Payment, and Ride-Matching.
Pros & Cons
Pros of Microservices
Independent scaling and deployments.
Technology diversity per service.
Fault isolation (one service failing doesn't crash the system).
Faster development by independent teams.
Cons of Microservices
Operational complexity (networking, monitoring, discovery).
Data consistency challenges across services.
Higher latency due to network calls.
Pros of Monoliths
Simple to develop, test, and deploy.
Easier to debug and monitor.
No inter-service communication latency.
Cons of Monoliths
Harder to scale parts of the system individually.
A single bug can bring down the whole application.
Slower development velocity in large teams.
Use Cases Comparison
Use Case
Recommended Approach
Early-stage startup
Monolith (simplicity > scalability)
Rapidly scaling product
Microservices (scalability > simplicity)
Large enterprise app
Microservices (independent teams)
MVP or prototype
Monolith (faster development)
Step-by-Step Design Approach (Microservices)
Break Down Domains
Break down the system into domains.
Define APIs
Define APIs and contracts between services.
Choose Protocols
Choose communication protocols (REST/gRPC/Kafka).
Deploy Independently
Deploy each service independently.
Add Observability
Add monitoring, service discovery, and logging layers.
🏭
Factory vs. Workshops
A monolith is like a single giant factory — one failure shuts everything down. Microservices are like independent workshops — each can work (and fail) separately.
🎯 Interview Tip
“Monolith vs Microservices: Which would you choose and why?”
Always answer based on project scale, team size, and performance requirements.
Key Takeaways
Microservices = scalability, flexibility, and team autonomy.
Monolith = simplicity and speed for small-scale apps.
The best systems often evolve from monolith → microservices as they scale.
2. How Monoliths are Migrated
Definition / Introduction
Monolith-to-microservices migration is the process of gradually decomposing a large monolithic application into smaller, independent services — without disrupting business operations.
Why it matters: Most companies start with monoliths and migrate as they grow. A well-planned migration ensures scalability, maintainability, and minimal downtime.
Core Concepts
Strangler Fig Pattern: Build new services around the monolith and slowly replace old functionality.
API Gateway: Acts as a unified entry point while routing to microservices or the monolith.
Event-Driven Extraction: Gradually shift functionalities using event streams.
Service Mesh: Helps manage communication and observability during migration.
How Event-Driven Extraction Works
Instead of directly calling the monolith’s database, you make the monolith publish events (e.g., “order_created”, “user_registered”) to a message queue whenever something happens. New microservices subscribe to these events and build their own data stores. Gradually, you shift more functionality from the monolith to event-consuming microservices. The monolith shrinks as services take over its responsibilities.
What Is a Service Mesh?
A service mesh (like Istio or Linkerd) is an infrastructure layer that handles all communication between microservices. It automatically does: encrypted traffic between services (mTLS), retries on failed calls, circuit breaking, load balancing, and provides a unified dashboard showing which service talks to which and how often. You don’t write this logic in your application code — the mesh handles it transparently.
Strangler Fig migration: an API Gateway incrementally routes traffic to new microservices while the monolith shrinks over time.
Real-World Examples
Amazon: Migrated from a monolithic retail application to 100+ microservices over several years.
Netflix: Started as a monolith DVD system → migrated to microservices for streaming scalability.
Airbnb: Used the strangler pattern to split its monolith into modular services without downtime.
Pros & Cons
Pros
Enables scalability and faster releases.
Easier to maintain and evolve codebases.
Reduced deployment risk with gradual rollout.
Cons
Complex to coordinate across teams.
Requires heavy DevOps investment.
Increased latency during transition period.
Migration Strategies
Strategy
Description
Strangler Fig
Gradually replace parts of the monolith with microservices.
Feature Extraction
Extract specific features into standalone services.
Event Sourcing
Capture and replay events to rebuild state in new services.
API Gateway Fronting
Route traffic selectively to new microservices.
Step-by-Step Migration Plan
Assessment
Identify tightly and loosely coupled components.
Boundaries
Define clear service boundaries (e.g., auth, billing, notifications).
Strangler Pattern
Start routing specific features to new services.
Incremental Migration
Gradually move logic and data.
Full Transition
Once all parts are split, retire the monolith.
🏠
Renovating a House
It's like renovating a house one room at a time while still living in it — you keep the house functional while upgrading it piece by piece.
🎯 Interview Tip
“How would you break a monolithic system into microservices?”
Always mention the strangler pattern, API gateway, and incremental approach.
Key Takeaways
Migration should be gradual and carefully planned.
Use an API Gateway and event-driven design to ease the transition.
Focus on modular boundaries to avoid tight coupling in new services.
Module 8 Recap — Microservices
Key concepts at a glance
Module 8 Summary
Monoliths are simple and fast to build but hard to scale.
Microservices offer scalability, flexibility, and team autonomy — but require more operational complexity.
Migration from monolith to microservices is best done incrementally, using patterns like Strangler Fig and API Gateways.
Almost every major tech company has followed this evolution path as they scaled.
Module 9: API Gateways
2 Topics — API Design & Asynchronous APIs
Module Scope
Note: This module covers API Design principles and communication patterns. An API Gateway (the infrastructure component that sits in front of your services to handle routing, authentication, and rate limiting) is a related concept often used alongside these patterns — we touch on it in the recap. The module is titled 'API Gateways' to match the original curriculum, but the primary focus is on API design principles and communication patterns.
1. How Are APIs Designed?
Definition / Introduction
An API (Application Programming Interface) is a set of defined rules and protocols that allows different software components to communicate.
API design is the process of defining endpoints, request/response structures, authentication, versioning, and error handling in a way that is scalable, intuitive, and maintainable.
Why it matters: A poorly designed API leads to tight coupling, poor developer experience, and scalability issues. A well-designed API is the backbone of any distributed or microservices system.
Core Concepts — Key Principles of Good API Design
Consistency: Endpoints, naming, and responses should follow a clear convention.
Simplicity: APIs should be easy to understand and use without deep documentation.
Statelessness: Each request should contain all necessary context (REST principle).
Versioning: Maintain backward compatibility with /v1, /v2, etc.
Security: Implement authentication (OAuth, tokens) and authorization.
Pagination & Filtering: Use parameters for scalable data access.
Each API request must carry ALL the information the server needs — the server doesn’t remember anything from previous requests. Example: instead of “continue my session” (stateful), you send your auth token with every request (stateless). This is why REST APIs use tokens instead of sessions — any server can handle any request because everything needed is in the request itself.
Types of APIs
REST (HTTP/JSON): Most common, stateless, resource-based.
GraphQL: Flexible queries, client-controlled data shape.
gRPC: High-performance, binary protocol for service-to-service communication.
WebSocket is technically a communication protocol (persistent bidirectional connection), not an API architectural style like REST or GraphQL. But it’s listed here because it’s a common way to build real-time APIs (chat, live updates, gaming).
Real-Time Communication: The Full Spectrum
Method
How It Works
Direction
Best For
Short Polling
Client asks server repeatedly ("Any updates?") every N seconds
Client → Server
Simple status checks, low-frequency updates
Long Polling
Client asks server; server HOLDS the connection open until there's new data (or timeout), then responds. Client immediately reconnects.
Client → Server
Chat apps (before WebSocket), notifications. Used by early Facebook Messenger.
Server-Sent Events (SSE)
Server pushes updates to client over a persistent HTTP connection. One-directional. Auto-reconnects on failure.
Server → Client
Live dashboards, stock tickers, news feeds, LLM streaming responses
WebSocket
Full-duplex persistent connection. Both sides can send anytime.
Interview tip: Don't default to WebSocket for everything. SSE is simpler, uses standard HTTP (works through proxies/firewalls), and auto-reconnects — it's the right choice for server-to-client streaming. Use WebSocket only when you need bidirectional communication.
REST vs GraphQL: A Concrete Example
REST:GET /api/users/123 → returns ALL 20 fields (name, email, address, phone, bio, avatar, created_at...) GraphQL:query { user(id: 123) { name, email } } → returns ONLY the 2 fields you asked for.
This is especially useful for mobile apps where bandwidth is limited — fetch only what you need.
GraphQL's N+1 Problem:
The flexibility of GraphQL comes with a trap. If you query { users { posts { comments } } }, a naive implementation will: (1) fetch all users (1 query), (2) for each user, fetch their posts (N queries), (3) for each post, fetch comments (N×M queries). This is the N+1 problem — it can generate hundreds of database queries from a single GraphQL query.
Solution: DataLoader pattern. Instead of fetching one-by-one, DataLoader batches requests. When 50 users need their posts, DataLoader collects all 50 user IDs and makes a single query: SELECT * FROM posts WHERE user_id IN (id1, id2, ...id50). This collapses N queries into 1.
Also needed: Query complexity analysis — limit the depth and breadth of queries to prevent clients from requesting the entire database graph in one call.
REST API flow: the client sends a request to the API Gateway, which routes it to the User Service. The response returns as structured JSON.
Real-World Examples
Stripe API: Known for its simplicity and developer-friendly documentation.
Twitter API: RESTful with pagination, rate limiting, and versioning.
Google Maps API: Offers both REST and WebSocket endpoints for real-time updates.
Pros & Cons
Pros of Well-Designed APIs
Easy integration for clients.
Better scalability and maintainability.
Easier versioning and evolution.
Cons / Risks
Poorly designed APIs increase coupling.
Harder to deprecate old versions if not planned properly.
Security vulnerabilities if not designed carefully.
REST vs gRPC
Feature
REST
gRPC
Format
JSON/Text
Binary (Protocol Buffers)
Performance
Slower
Faster
Best For
External/public APIs
Internal microservice communication
Streaming
Limited
Full duplex supported
Step-by-Step API Design Process
Define Use Cases
What problems does the API solve?
Design Resources
Identify key entities (/users, /orders).
Decide Methods
GET, POST, PUT, DELETE.
Define Schemas
Define request/response schemas — keep them predictable.
Implement Auth & Rate Limiting
Implement authentication and rate limiting.
Document & Version
Document and version your API.
Pagination: Cursor-Based vs Offset-Based
Offset-based (?page=5&limit=20): Simple but broken at scale. If new items are inserted while you're paginating, you'll see duplicates or skip items. Also, OFFSET 10000 forces the database to scan and discard 10,000 rows — O(n) performance.
Cursor-based (?after=eyJpZCI6MTIzfQ&limit=20): Uses an opaque cursor (usually a base64-encoded ID or timestamp) pointing to the last item you saw. The query becomes WHERE id > cursor LIMIT 20 — O(1) using an index, stable under concurrent inserts.
Rule of thumb: Use offset-based for admin dashboards with small datasets. Use cursor-based for feeds, timelines, search results — anything user-facing or high-volume. Every modern API (Twitter, GitHub, Stripe) uses cursor-based pagination.
🍽
Restaurant Menu
Think of an API like a restaurant menu — it tells customers (clients) what they can order (endpoints) and how to order it (HTTP methods).
🎯 Interview Tip
“How would you design an API for a ride-hailing app?”
Talk about resources, endpoints, methods, authentication, versioning, and pagination.
Key Takeaways
API design should be simple, predictable, and consistent.
Always plan for versioning, security, and scalability.
REST is most common, but gRPC and GraphQL are growing rapidly in distributed systems.
2. What Are Asynchronous APIs?
Definition / Introduction
Asynchronous APIs allow clients and servers to communicate without blocking, meaning the client does not wait for the server to complete a request before continuing.
Why it matters: Asynchronous communication improves performance, reduces latency, and is essential for event-driven systems and long-running tasks.
Core Concepts
Polling: Client periodically checks if a task is complete.
Webhooks: Server pushes updates to a client endpoint.
WebSockets: Persistent, bidirectional communication channel.
Event Streams: Real-time streams of events (e.g., Kafka topics).
Callbacks: Server notifies the client once a task is done.
Callbacks vs Webhooks: A callback is a broad term — any function or URL that gets called when an async operation completes. A webhook is a specific type of callback: an HTTP POST to a URL you registered. In practice, "webhook" and "callback URL" are often used interchangeably in API design.
Synchronous vs Asynchronous
Feature
Synchronous
Asynchronous
Flow
Request → Wait → Response
Request → Continue → Notify Later
Scalability
Limited (blocking)
High (non-blocking)
Use Case
CRUD APIs, immediate responses
Notifications, long-running jobs
Synchronous: client blocks waiting for the response. Asynchronous: client gets an immediate 202 acknowledgment, continues working, and receives a webhook callback when processing completes.
Real-World Examples
Stripe Webhooks: Payment events trigger real-time notifications to client systems.
GitHub Webhooks: Trigger CI/CD pipelines when code is pushed.
Slack Events API: Uses event-driven asynchronous architecture for real-time messaging.
Pros & Cons
Pros
Scales better under high load.
Ideal for long-running or background tasks.
Improves responsiveness and user experience.
Cons
Harder to debug and test.
Increased complexity in error handling and retries.
Requires idempotency and tracking for reliability.
What Is Idempotency?
Idempotency means performing the same operation multiple times produces the same result as doing it once. This is critical for async systems: if a webhook delivery fails and retries, your handler must not process the same event twice. Example: use a unique event ID — when you receive an event, check “have I already processed event #abc123?” If yes, skip it.
Webhooks vs Polling
Feature
Webhooks
Polling
Communication
Push (server → client)
Pull (client → server)
Latency
Low
Higher
Resource Usage
Efficient
Resource heavy
Reliability
Requires retry handling
Simple but slower
Step-by-Step Flow (Webhook Example)
Submit Request
Client submits a request (e.g., payment processing).
Background Processing
Server starts processing in the background.
Webhook Notification
Once complete, server sends a webhook notification to a client URL.
Client Updates
Client updates its system asynchronously.
🍕
Pizza Ordering
It's like ordering a pizza and leaving the restaurant — they'll call you when it's ready instead of you waiting there.
🎯 Interview Tip
“How would you implement asynchronous APIs for a video encoding service?”
Mention webhooks, callbacks, message queues, or event streams.
Key Takeaways
Asynchronous APIs are essential for event-driven and real-time systems.
They improve scalability and user experience but add complexity.
Best for background jobs, notifications, and streaming use cases.
Module 9 Recap — API Gateways
Key concepts at a glance
Module 9 Summary
Good API design is crucial for scalability, developer experience, and maintainability.
API Gateways often handle routing, authentication, rate limiting, logging, and versioning across microservices.
Asynchronous APIs are a must for event-driven systems, real-time applications, and long-running tasks.
Real-world systems often combine synchronous and asynchronous APIs depending on use cases.
Two terms you’ll see everywhere in this module — Authentication (AuthN): proving WHO you are (e.g., entering your username and password). Authorization (AuthZ): proving WHAT you’re allowed to do (e.g., can this user delete this file?). Authentication comes first — you can’t authorize someone whose identity you haven’t verified.
1. OAuth (Open Authorization)
Definition / Introduction
OAuth is an open standard for access delegation — it allows applications to access resources on behalf of a user without sharing the user's credentials.
Why it matters: OAuth enables secure third-party access (like “Login with Google”) while keeping user passwords safe. It's a foundational protocol for modern API security.
Core Concepts — Key Roles
Role
Description
Resource Owner
The user (you) who owns the data.
Client
The application requesting access (e.g., Slack).
Resource Server
API server hosting protected resources (e.g., Google Calendar API).
Authorization Server
Issues tokens (e.g., accounts.google.com).
OAuth Flow (Authorization Code Grant)
1. User Authorization: User is redirected to the authorization server (e.g., Google) to grant access.
2. Authorization Code: Once the user approves, the server sends a short-lived authorization code to the client.
3. Token Exchange: The client exchanges the code for an access token.
4. API Access: The client uses this token to access protected resources from the resource server.
OAuth Authorization Code flow: user authenticates with the auth server, the app receives an auth code, exchanges it for a token, and uses the token to access protected resources.
OAuth 2.0 Grant Types
Note: OAuth 2.0 has several “grant types” for different scenarios. This section covers Authorization Code Grant (the most common for web apps). Others include: Client Credentials (server-to-server, no user involved), Device Code (smart TVs with no keyboard), and Implicit (deprecated, was used for browser-only apps). For interviews, Authorization Code is the one to know in depth.
PKCE — Required for Modern OAuth (since 2021):
The Authorization Code flow described above is the pre-PKCE version. Since OAuth 2.1, PKCE (Proof Key for Code Exchange) is mandatory for ALL clients — not just mobile apps.
Why? Without PKCE, if an attacker intercepts the authorization code (e.g., via a malicious browser extension), they can exchange it for a token. PKCE prevents this by adding a one-time proof.
How it works: (1) Client generates a random code_verifier (a long random string). (2) Client hashes it to create a code_challenge and sends the challenge with the auth request. (3) When exchanging the code for a token, the client sends the original code_verifier. (4) The auth server hashes the verifier and checks it matches the challenge. An attacker with just the code can't generate the verifier.
Interview tip: If you describe OAuth without mentioning PKCE, a security-aware interviewer will notice. Always mention it — it shows you know the current standard, not just the textbook version.
Real-World Examples
“Login with Google/Facebook” on websites and apps.
Slack → Google Calendar integration: Slack doesn't need your password, just an access token.
GitHub OAuth Apps: 3rd-party tools use OAuth tokens to access repositories.
Pros & Cons
Pros
Highly secure (no password sharing).
Granular, scoped permissions.
Revocable tokens and short expiry reduce risk.
Cons
Complex to implement correctly.
Misconfiguration can lead to security vulnerabilities.
Requires HTTPS and secure storage of tokens.
OAuth vs Basic Auth
Feature
OAuth
Basic Auth
Credentials
Never shared
Sent on every request
Security
High
Low
Use Case
Third-party access
Simple internal APIs
Step-by-Step (Authorization Code Flow)
User Clicks Login
User clicks “Login with Google.”
Redirect to OAuth Server
Redirects to Google's OAuth server.
Authenticate & Grant
User authenticates and grants permission.
Authorization Code Returned
Google returns an authorization code.
Exchange for Token
Client exchanges code for access token.
API Calls
Token used to call APIs securely.
🚗
Valet Parking
Think of OAuth like valet parking — you give the valet a key (token) to move your car, but they never get full access to your house (password).
🎯 Interview Tip
“Explain the OAuth flow.” / “How is OAuth different from token-based authentication?”
Always mention scopes, tokens, and authorization server.
Key Takeaways
OAuth is about delegating access securely.
Access tokens are temporary and scoped.
It's the backbone of secure third-party integrations.
OpenID Connect (OIDC)
OpenID Connect (OIDC) is built on top of OAuth 2.0 and adds an identity layer. OAuth answers “what is this app allowed to access?” (authorization). OIDC answers “who is this user?” (authentication). When you click “Sign in with Google,” that’s OIDC — Google tells the app your name and email (identity), not just that you’ve granted calendar access.
OAuth Scopes
OAuth Scopes define what a token is allowed to do. When an app requests access, it specifies scopes like read:email or write:calendar. The user sees these during the consent screen (“This app wants to read your email — allow?”). The resulting token only works for those specific scopes — it can’t access anything else.
2. Token-Based Authentication
Definition / Introduction
Token-based authentication is a stateless authentication mechanism where the server issues a token after a successful login. This token is sent with every request to verify identity.
Why it matters: It allows scalable, stateless, and secure user sessions — essential for APIs, microservices, and distributed systems.
Core Concepts
Access Token: Proof of authentication (often JWT).
JWT (JSON Web Token): Compact, self-contained token with user data and signature.
Statelessness: No session data stored on the server — the token carries all info.
Refresh Token: Used to obtain a new access token after expiration.
JWT Structure
header.payload.signature
Header: Algorithm & token type Payload: Claims (user ID, roles, expiry) Signature: Ensures integrity and authenticity
A JWT is three Base64-encoded segments separated by dots: the Header (algorithm), Payload (user claims), and Signature (verification hash).
Real-World Examples
REST APIs: Clients include JWTs in Authorization: Bearer <token> headers.
Mobile apps: Authenticate once, store the token locally, and reuse it.
Microservices: Authenticate inter-service calls with signed tokens.
Pros & Cons
Pros
Stateless → scalable and easy to distribute.
Tokens can include custom claims (e.g., roles, permissions).
Faster than database-backed sessions.
Cons
Token revocation is difficult (especially with JWT).
Larger tokens = more network overhead.
Must be stored securely (e.g., not in localStorage).
Why Is JWT Revocation Hard?
A JWT is self-contained — the server doesn’t store it or track active sessions. Once issued, it’s valid until it expires. If a user logs out or their account is compromised, you can’t “un-issue” the JWT. The workarounds: (1) Short expiry times (e.g., 15 minutes) with refresh tokens. (2) Maintain a server-side blocklist of revoked tokens — but this reintroduces the statefulness that JWTs were supposed to eliminate.
Refresh Token Rotation:
Best practice: every time a refresh token is used, issue a NEW refresh token and invalidate the old one. If an attacker steals a refresh token and the legitimate user also tries to use it, the server detects the reuse (the old token was already rotated) and invalidates ALL tokens for that user — forcing re-login.
This limits the damage window of a stolen refresh token to a single use.
mTLS — Service-to-Service Authentication:
OAuth and JWT work great for user authentication. But how do microservices authenticate each other? Mutual TLS (mTLS): both the client service and server service present certificates. The server verifies the client's certificate, and the client verifies the server's. This ensures both sides are who they claim to be.
Service meshes like Istio handle mTLS automatically — every service-to-service call is encrypted and authenticated without changing application code.
JWT vs Session-Based Auth
Feature
JWT
Session
Storage
Client-side
Server-side
Scalability
High
Low
Revocation
Complex
Easy
Stateless
Yes
No
Step-by-Step Flow
Send Credentials
Client sends username/password to server.
Validate & Issue JWT
Server validates and issues a JWT.
Store & Send Token
Client stores token and sends it with each request.
Verify & Grant Access
Server verifies signature and grants access.
🆔
Signed ID Card
A JWT is like a signed ID card — once issued, it's enough to prove your identity without checking the database every time.
🎯 Interview Tip
“How does JWT authentication work?” / “What's the difference between access and refresh tokens?”
Highlight statelessness and scalability.
Key Takeaways
Token-based auth is scalable and ideal for APIs.
JWT is the most common implementation.
Always secure tokens and use refresh tokens for long-lived sessions.
3. Access Control Lists (ACLs) and Rule Engines
Definition / Introduction
Access Control Lists (ACLs) define which users or roles can access which resources.
Rule engines go a step further — dynamically enforcing policies based on conditions, context, or attributes.
Why it matters: Authentication proves who you are, but authorization decides what you can do.
Access Control Models
Model
Description
Example
ACL (Access Control List)
Resource lists who can access it
“user123 can read /orders”
RBAC (Role-Based Access Control)
Access based on roles
“Admins can delete users”
ABAC (Attribute-Based Access Control)
Access based on attributes (time, IP, device)
“Access only during office hours”
PBAC (Policy-Based Access Control)
Complex policies with conditions
“Allow API access if role=manager AND region=US”
What Does an ABAC Policy Look Like?
Example: IF user.role == 'employee' AND user.department == 'engineering' AND current_time BETWEEN 9:00-18:00 AND resource.classification != 'top-secret' THEN ALLOW. ABAC evaluates multiple attributes (user, resource, environment) for each request — more flexible than RBAC but more complex to manage.
RBAC: a User is assigned a Role (Admin, Editor, Viewer), which maps to specific Permissions, granting or denying access to Resources.
Rule Engine: Evaluates access based on custom logic or business policies.
Policy-as-Code: Using languages like OPA/Rego for dynamic policy enforcement.
Real-World Examples
AWS IAM: Combines RBAC, ABAC, and policy-based control.
Kubernetes RBAC: Controls who can access resources in a cluster.
Google Drive: Uses ACLs for document sharing.
OPA (Open Policy Agent): A rule engine for API authorization.
Pros & Cons
Pros
Fine-grained and flexible access control.
Improves security and compliance.
Easy to audit and update policies.
Cons
ACLs become complex at scale.
Dynamic policies require careful testing.
Performance overhead for complex rule evaluations.
RBAC vs ABAC
Feature
RBAC
ABAC
Simplicity
Simple
Complex
Flexibility
Limited
High
Performance
Fast
Slower
Use Case
Enterprise apps
Large, dynamic environments
Step-by-Step (RBAC Flow)
Login & Assign Role
User logs in and is assigned a role.
Check Policy
Application checks policy: “Does this role have permission?”
Grant or Deny
If yes → grant access; else → deny.
📝
Guest List + Bouncers
ACLs are like a guest list — only listed people can enter. Rule engines are like bouncers checking ID, time, and dress code — more context-aware.
🎯 Interview Tip
“Explain RBAC vs ABAC.” / “How would you design an authorization layer for a SaaS platform?”
Discuss ACL → RBAC → ABAC evolution.
Key Takeaways
ACLs define who can do what.
Rule engines bring context and dynamic policies.
Policy-as-code is now a best practice in large systems.
BONUS: Keycloak and Modern Identity Management
What is Keycloak?
Keycloak is an open-source Identity and Access Management (IAM) solution built by Red Hat. It handles authentication, authorization, SSO, identity brokering, and OAuth2/OIDC flows out of the box.
It's often used as a central auth server in microservice systems — replacing the need to implement authentication from scratch.
Why It Matters
Keycloak abstracts away complexity like token issuance, OAuth flows, SSO, LDAP/Active Directory integration, and MFA — allowing you to focus on application logic.
It implements standards like OAuth 2.0, OpenID Connect, and SAML 2.0.
What is SAML? SAML (Security Assertion Markup Language) is an older, XML-based protocol primarily used for enterprise Single Sign-On (SSO). Think: logging into your company's internal apps with one corporate login. OIDC (JSON-based, modern, mobile-friendly) is gradually replacing SAML for new applications, but SAML remains dominant in enterprise environments.
Key Features
SSO & SLO: Single sign-on and sign-out across apps.
Clients redirect users to Keycloak for login.
Keycloak issues tokens.
APIs verify tokens using Keycloak's public keys.
Policies and roles are managed centrally.
Real-World Examples
Enterprises use Keycloak as a self-hosted alternative to Auth0 or Okta.
Often used in Kubernetes microservices for unified authentication.
Pros & Cons
Pros
Standards-compliant, production-ready.
Reduces security risk by centralizing auth.
Extensible via plugins and custom providers.
Cons
Can be overkill for small apps.
Needs dedicated infrastructure and maintenance.
Limited UI customization without extensions.
🎯 Interview Tip
“How would you integrate Keycloak in a microservices architecture?”
Mention it as a central auth provider handling tokens and policies.
Key Takeaways
Keycloak is a powerful, enterprise-grade IAM solution.
Great for large, distributed systems needing centralized security.
Replaces the need for building authentication/authorization logic manually.
Module 10 Recap — Authentication & Authorization
Key concepts at a glance
Module 10 Summary
OAuth is the industry standard for secure delegated access.
ACLs, RBAC, ABAC, and rule engines form the backbone of authorization.
Keycloak simplifies identity, token, and policy management across complex systems.
Together, these tools ensure your system is secure, scalable, and compliant — from login to granular access control.
Module 11: System Design Trade-Offs
6 Topics — How to think like a system designer
About This Module
This chapter is not about “new components” — it's about how to think like a system designer. Every real-world decision — from choosing a database to deciding API patterns — involves trade-offs. Interviewers love testing how you reason through them.
1. Pull vs. Push
Definition / Introduction
Push model: The server initiates communication and “pushes” data to clients as soon as it's available.
Pull model: The client requests (pulls) data from the server when it wants or needs it.
Why it matters: The right choice affects scalability, latency, bandwidth usage, and freshness.
Push vs Pull Comparison
Feature
Push
Pull
Initiator
Server
Client
Latency
Low (instant updates)
Higher (polling delay)
Control
Server decides
Client decides
Scalability
Harder (server tracks many clients)
Easier (clients control load)
Push: server sends data to clients instantly via WebSocket. Pull: clients periodically poll the server for new data via HTTP requests.
Real-World Examples
Push: WebSockets in chat apps (WhatsApp, Slack), push notifications.
Real-time data delivery.
Lower latency and better user experience.
Push Cons
Complex to scale with many clients.
Harder to handle offline clients.
Pull — Pros & Cons
Pull Pros
Easier to scale and cache.
Client controls frequency.
Pull Cons
Higher latency.
More bandwidth if polling frequently.
Step-by-Step Example (Push)
New Data Arrives
New message arrives on server.
Server Pushes
Server immediately pushes it to client over WebSocket.
Client Updates
Client updates UI instantly.
📠
News Alerts vs. Checking
Push is like news alerts sent directly to your phone. Pull is like checking the news site manually.
🎯 Interview Tip
“Would you use push or pull for a stock ticker?”
Choose push for real-time, pull for predictable traffic.
Key Takeaways
Push = real-time but harder to scale.
Pull = simpler but slower.
Many systems use a hybrid (e.g., push for critical events, pull for bulk updates).
2. Memory vs. Latency
Definition / Introduction
Memory and latency are often in conflict: more data cached in memory reduces latency but consumes more memory.
Why it matters: Optimizing this trade-off ensures cost-effective performance.
Core Concepts
In-memory caching: Low latency but memory-intensive.
Disk-based systems: Slower access but cheaper and scalable.
Pre-computation: Uses more memory to avoid expensive calculations.
Real-World Examples
Redis / Memcached: Use memory for ultra-low-latency reads.
Elasticsearch: Stores large indexes on disk to handle scale.
Netflix: Precomputes recommendations to trade memory for speed.
More Memory — Pros & Cons
More Memory (Pros)
Faster access and responses.
Better user experience.
More Memory (Cons)
Expensive at scale.
Harder to maintain cache coherence.
Less Memory — Pros & Cons
Less Memory (Pros)
Cheaper and easier to scale.
Less risk of cache thrashing. (Cache thrashing = when your cache is too small for your workload, items keep getting evicted and re-fetched endlessly. The cache churns but provides no benefit — like a revolving door where nothing stays inside.)
Less Memory (Cons)
Slower responses.
More CPU cycles for computation.
Comparison
Approach
Memory Usage
Latency
Caching
High
Low
On-demand Computation
Low
High
💼
Desk vs. Storage Room
It's like keeping frequently used tools on your desk (fast but space-heavy) vs. in a storage room (cheap but slower to fetch).
🎯 Interview Tip
“Would you precompute results or compute on the fly?”
Mention access frequency, latency budget, and memory cost.
Key Takeaways
High memory reduces latency but increases cost.
Use hybrid strategies (cache hot data, compute cold data).
3. Throughput vs. Latency
Definition / Introduction
Throughput: Total work done per unit time.
Latency: Time to process a single request.
Why it matters: High-throughput systems may sacrifice latency, and vice versa.
Core Concepts
Batch Processing: High throughput but higher latency.
Real-Time Processing: Low latency but lower throughput.
Queue-based systems: Smooth bursts but add latency.
Real-World Examples
Kafka: Optimized for high throughput (millions of messages/sec).
HFT Trading: Optimized for microsecond latency.
MapReduce: Batches jobs for maximum throughput.
High Throughput — Pros & Cons
High Throughput (Pros)
Better resource utilization.
Lower cost per request.
High Throughput (Cons)
Slower per-request response.
Low Latency — Pros & Cons
Low Latency (Pros)
Better UX.
Suitable for interactive apps.
Low Latency (Cons)
May reduce system capacity.
Comparison Table
Approach
Throughput
Latency
Batch
High
High
Stream
Medium
Medium
Real-time
Low
Low
Reading this table: For throughput, HIGH is good (more work done). For latency, LOW is good (faster response). So "Batch = High throughput, High latency" means it processes lots of data but takes a long time. "Real-time = Low throughput, Low latency" means it processes less data at once but responds instantly. There is always a trade-off — you cannot maximize both.
Why "low throughput" for real-time? Real-time systems process each item individually as it arrives — no batching. A batch system can process 1 million records per hour by reading them all at once. A real-time system processes them one-by-one as they stream in. The per-item latency is much lower, but the total volume processed per hour may also be lower due to per-item overhead. It's a trade-off, not a deficiency.
“Would you prioritize latency or throughput for a payment processor?”
Choose latency for real-time UX, throughput for analytics.
Key Takeaways
Trade-offs depend on use case.
Batch jobs → throughput. Real-time → latency.
4. Consistency vs. Availability (CAP Theorem)
Definition / Introduction
CAP Theorem: In distributed systems, you can guarantee only two out of three: Consistency, Availability, Partition Tolerance.
Since partition tolerance is a must, the trade-off is between consistency and availability.
Why it matters: It guides database and system design choices.
CAP Terms
Term
Meaning
Consistency
All nodes return the same data.
Availability
System responds to requests even during failures.
Partition Tolerance
System continues despite network splits.
CAP Theorem: in a distributed system you can only guarantee two of three properties. Since Partition Tolerance is mandatory, the real choice is between Consistency (CP) and Availability (AP).
AP (Availability + Partition Tolerance): Cassandra, DynamoDB.
Consistency — Pros & Cons
Consistency (Pros)
Predictable results.
Easier to reason about.
Consistency (Cons)
May reject requests during partition.
Availability — Pros & Cons
Availability (Pros)
Always responds.
Availability (Cons)
May serve stale data.
CP vs AP
Approach
Strength
Weakness
CP
Strong data integrity
Lower uptime under partition
AP
High uptime
Eventual consistency
⚖️
Same Answer vs. Always an Answer
It's like choosing between always getting the same answer (consistency) or always getting an answer (availability).
🎯 Interview Tip
“Would you choose CP or AP for a messaging system?”
CP for banking, AP for social feeds.
Key Takeaways
Partition tolerance is non-negotiable.
Choose based on use case (critical data → CP, availability → AP).
Why can't you just choose to not have partition tolerance? Because network failures are a fact of life, not a design choice. Cables get cut, switches fail, data centers lose connectivity. If your system cannot handle a network partition, it simply stops working when one occurs — which is unacceptable for any real distributed system. Since P is mandatory, CAP really becomes a choice between C and A.
PACELC — CAP's Practical Extension:
CAP only describes behavior during a network partition. But partitions are rare — most of the time your system is running normally. PACELC asks: During a Partition, choose Availability or Consistency; Else (normal operation), choose Latency or Consistency.
Real-world examples:
DynamoDB (PA/EL): During partition → Available. Normal operation → Low Latency. Sacrifices consistency in both cases for speed.
Google Spanner (PC/EC): During partition → Consistent. Normal operation → Consistent. Sacrifices latency for correctness in both cases (uses TrueTime GPS clocks to achieve this).
MongoDB (PA/EC): During partition → Available. Normal operation → Consistent. A balanced approach — fast reads when things are fine, available when things break.
Why this matters: PACELC gives you a more practical framework than CAP because it addresses the common case (no partition), not just the edge case. Mentioning PACELC in an interview shows depth beyond the standard CAP answer.
5. Latency vs. Accuracy
Definition / Introduction
Latency: Speed of response.
Accuracy: Completeness or correctness of results.
Why it matters: Many systems trade accuracy for speed (e.g., search engines, recommendations).
Core Concepts
Approximation algorithms: Sacrifice accuracy for speed.
Sampling: Analyze partial data for faster insights.
Eventually consistent systems: Prioritize low latency over strict accuracy.
Real-world example: A bank's fraud detection system has two options. Option A: Run a complex ML model that analyzes 50 features in 2 seconds — 99.5% accuracy but the customer waits. Option B: Run a lightweight rule-check (amount > $10K? foreign country? new device?) in 20ms — 95% accuracy but the card swipe approves instantly. Most banks use Option B for real-time decisions and run Option A in the background to catch what B missed.
What does sampling mean in practice? Instead of analyzing all 1 billion log entries to find your error rate, randomly pick 1% (10 million entries), calculate the error rate from that sample, and extrapolate. The result will not be exact, but it will be close enough — and you get it in seconds instead of hours. Tools like Datadog and New Relic use sampling to provide real-time dashboards without processing every single event.
Real-World Examples
Google Search: Returns approximate results in milliseconds.
Fraud detection: Early alerts may be less accurate but faster.
Analytics dashboards: Use sampled data for near real-time updates.
Pros & Cons
Low Latency (Pros)
Better user experience.
Real-time decisions.
Low Latency (Cons)
May miss edge cases or details.
High Accuracy (Pros)
More reliable results.
High Accuracy (Cons)
Slower and more expensive.
Comparison
Approach
Latency
Accuracy
Real-time alerts
Low
Medium
Offline reports
High
High
🤔
Guessing vs. Solving
Fast does not equal perfect. Like guessing an answer quickly vs. solving it thoroughly.
🎯 Interview Tip
“Would you prefer latency or accuracy for a recommendation engine?”
Choose latency — users prefer fast suggestions over perfect ones.
Key Takeaways
Trade accuracy for latency in real-time systems.
Choose accuracy when correctness is non-negotiable (e.g., billing).
NoSQL: Schema-less, flexible, scalable, BASE (eventually consistent).
Why it matters: Choosing the right database affects performance, flexibility, and scalability.
SQL vs NoSQL Feature Comparison
Feature
SQL
NoSQL
Schema
Fixed
Flexible
Scalability
Vertical
Horizontal
Consistency
Strong (ACID)
Eventual (BASE)
Querying
Rich (JOINs, complex queries)
Limited but fast
ACID = Atomicity (all-or-nothing), Consistency (valid state always), Isolation (transactions don't interfere), Durability (committed data survives crashes). BASE = Basically Available (system always responds, maybe stale), Soft State (data may be in flux as replicas sync), Eventually Consistent (all replicas converge given time). ACID prioritizes correctness; BASE prioritizes availability.
Real-World Examples
SQL: MySQL, PostgreSQL, Oracle.
NoSQL: MongoDB, DynamoDB, Cassandra, Redis.
SQL — Pros & Cons
SQL Pros
Strong consistency and integrity.
Rich querying and relationships.
SQL Cons
Harder to scale horizontally.
Schema migrations can be slow.
NoSQL — Pros & Cons
NoSQL Pros
Highly scalable and flexible.
Handles large, unstructured data.
SQL is like a structured filing cabinet — everything has its place. NoSQL is like a big bucket — flexible, but less organized.
🎯 Interview Tip
“Would you use SQL or NoSQL for a chat app?”
NoSQL for messages (scalability), SQL for user accounts.
Key Takeaways
SQL = consistency, structure, complex queries.
NoSQL = scalability, flexibility, and speed.
NewSQL — The Best of Both Worlds?
NewSQL databases provide horizontal scalability (like NoSQL) with full ACID transactions and SQL interfaces (like traditional SQL). They challenge the old "SQL vs NoSQL" binary.
Key players:
Google Spanner: Globally distributed, strongly consistent, uses GPS/atomic clocks (TrueTime) for transaction ordering. The "impossible" database.
CockroachDB: Open-source Spanner-inspired database. Horizontal scaling with serializable transactions.
TiDB: MySQL-compatible distributed database. Drop-in replacement that scales horizontally.
Vitess: MySQL sharding middleware (used by YouTube, Slack). Not a new DB — it makes MySQL horizontally scalable.
When to use: When you need both relational semantics (JOINs, transactions) AND horizontal scale. The trade-off: higher operational complexity and latency compared to a single-node PostgreSQL.
Module 11 Recap — System Design Trade-Offs
The art of making the right choice for the right context
Module 11 Summary
Push vs Pull: Real-time vs scalability.
Memory vs Latency: Faster access vs resource cost.
Throughput vs Latency: Bulk performance vs single-request speed.
Consistency vs Availability: Data integrity vs uptime.
Latency vs Accuracy: Speed vs precision.
SQL vs NoSQL: Structure & reliability vs scalability & flexibility.
Trade-off
If you need...
Then choose...
Consistency vs. Availability
100% Accuracy (Banking)
CP (Consistency/Partition)
Consistency vs. Availability
100% Uptime (Social Media)
AP (Availability/Partition)
Latency vs. Throughput
Fast User Response
Low Latency
Latency vs. Throughput
High Data Volume (Logs)
High Throughput (Batching)
Pull vs. Push
Real-time Updates (Chat)
Push (WebSockets)
The Golden Rule
“There's no ‘best’ choice — only the best choice for a given context.” Great system designers justify decisions based on business and technical constraints.
Module 12: Practice Problems
17 flagship system design problems -- end-to-end designs framed for interviews + real-world architecture
Common terms you will see across these practice problems:
ABR (Adaptive Bitrate): Automatically adjusting video quality based on your internet speed.
HLS/DASH: Streaming protocols that break video into small chunks for adaptive delivery.
RTMP: A protocol for sending live video from a camera/encoder to a server.
CDN: Content Delivery Network — edge servers worldwide that cache content close to users.
E2EE: End-to-End Encryption — only sender and receiver can read the messages.
DRM: Digital Rights Management — prevents unauthorized copying of content (e.g., Widevine for Chrome, FairPlay for Safari).
Fan-out: Distributing one event/message to multiple recipients.
FIFO: First-In-First-Out — processing items in the order they arrived.
Part 1: Problems 1 -- 5
1) System Design of a Live-Streaming App (Twitch / YouTube Live)
A platform where creators stream live video/audio to many concurrent viewers with chat, reactions, and DVR. Why it matters: Combines real-time ingestion, fan-out at scale, low latency, and edge delivery.
Live streaming pipeline: Creator streams to an ingest server, which transcodes to multiple bitrates, distributes via CDN to viewers. Chat uses a separate WebSocket/Pub-Sub path.
Player requests manifest from closest edge; ABR selects bitrate.
Realtime Chat
Client opens WebSocket to chat tier (regionally sharded).
DVR
Segments also persisted to object storage (S3/GCS) with lifecycle rules.
Observability
Edge/player metrics → stream health dashboards, autoscale.
Moderation
Stream flags → ML + human review; chat rate-limits, bans.
📺
TV Broadcast Analogy
TV broadcast trucks (ingest), studio converting to multiple formats (transcode), cable network (CDN), and a live call-in line (chat).
🎯 Interview Angle
How to achieve <5 s latency? (LL-HLS, chunked CMAF, shorter segments, prefetch)
How to scale chat to 1M concurrents? (sharded rooms, pub-sub, local fan-out gateways)
Cost controls? (multi-CDN, smart ABR ladders, just-in-time transcoding)
Key Takeaways
Separate data plane (video) from control plane (metadata/chat).
Choose LL-HLS for scale; WebRTC for sub-second niches.
Chat uses pub-sub + WebSockets with sharding and rate limiting.
2) System Design of Instagram (Feed + Media)
Social photo/video sharing with feeds, explore, stories, likes, comments, notifications. Why it matters: Mix of hot media reads, fan-out notifications, ranking, and write-heavy social graph.
Core Concepts
Write Path: Upload → object store (images/video), thumbnails, metadata (Postgres/MySQL).
Feed Generation: Fan-out-on-write (precompute followers' feeds) vs fan-out-on-read (compute on demand). Often hybrid.
Ranking: Signals (freshness, interactions) → feature store → ML rankers.
Caches: User timelines (Redis), profile cache, counts.
Instagram architecture: Write path stores media and fans out post IDs to followers' Redis caches. Read path fetches from cache, applies ML ranking, and returns a personalized feed.
Real-World Examples
Instagram: Hybrid feed fan-out; heavy cache; ML-based ranking; reel pipeline.
Matches: Only when both "likes" exist → generate a match row + notify.
Chat: WebSocket real-time + persistent store (NoSQL/append logs).
Real-World Examples
Uses geohash bucketing for nearby search; feature stores for ranking; spam detection via heuristics + ML.
Pros & Cons / Trade-offs
Trade-off Considerations
Strict filters vs discovery: Over-filter reduces matches; relax increases noise.
Strong consistency on matches vs eventual on counters.
Scaling Challenges
Hot regions require shard rebalancing. Geo-dense cities produce hotspots.
Geohash vs R-tree
Aspect
Geohash
R-tree
Simplicity
Simple, string-prefix based
More complex tree structure
Speed
Faster lookups
Slightly slower
Precision
Grid boundary issues
More precise for range queries
Best For
Simple nearby search
Complex spatial queries
Geohash boundary problem: Two people standing 10 meters apart might fall in different geohash cells (like being on opposite sides of a zip code boundary). A query for cell "abc123" would miss the person in adjacent cell "abc124." The fix: always query the target cell AND its 8 neighboring cells, then filter by actual distance.
Step-by-Step Walkthrough
Login & Location
Update geohash; cache user features.
Candidate Fetch
Query geohash buckets ± neighbors; apply filters.
Ranking
Score candidates; return page.
Swipe
Record swipe (idempotent), detect mutual like → create match.
Notify
Pub-sub → push notifications.
Chat
Create room; WebSocket; store messages; typing indicators.
📌
Neighborhood Notice Board
A neighborhood notice board filtered to your preferences; a mutual pin = match; then private chat.
🎯 Interview Angle
Prevent duplicate matches? (unique constraints on pair unordered key)
Scale hotspots? (shard by region/time, precompute candidates)
Key Takeaways
Geospatial + ranking + strict idempotency for swipes/matches.
WebSockets for chat; pub-sub for notifications.
Anti-abuse and rate-limit everywhere.
4) System Design of WhatsApp (Messaging)
End-to-end encrypted (E2EE) messaging with 1:1, groups, media, presence, and delivery receipts. Why it matters:Billions of devices, mobile intermittency, E2EE, store-and-forward.
Core Concepts
E2EE: Signal protocol; server never sees plaintext.
Message Routing: Store-and-forward on server queues until device ACKs.
Device Sync: Multi-device sessions with per-device keys.
Presence/Typing: Ephemeral via pub-sub; rate limited.
Media: Encrypted media to object storage + CDN; keys sent via message.
Receipts: ✓ (sent), ✓✓ (delivered), ✓✓ blue (read).
WhatsApp E2EE: the sender encrypts the message client-side; the server queues the encrypted blob without reading it; the recipient decrypts upon delivery.
Real-World Examples
Partition by user ID; queues per recipient; offline delivery; exponential backoff.
Pros & Cons / Trade-offs
Strong Privacy
E2EE ensures server blindness -- user data stays private on device.
Limited Server-side Features
No server-side indexing/search. Eventual sync across devices; conflict resolution with timestamps.
Push vs Pull & 1:1 vs Group
Dimension
Option A
Option B
Delivery
Push (FCM/APNs) wakes client
Then long-poll / WebSocket
1:1 vs Group
1:1 -- direct queue
Group fan-out to N members; server scales via batch fan-out
Step-by-Step Walkthrough
Auth & Key Setup
Register device keys; establish sessions.
Send
Client encrypts message; sends to server.
Persist & Fan-out
Enqueue per-recipient; deliver when online.
ACKs & Receipts
Update sender; update ticks.
Media
Upload encrypted media; send only URL + key to recipient(s).
Backup/Restore
Client-side encrypted backups (optional).
📮
Post Office with Sealed Letters
A post office with sealed letters it can't read; holds mail until recipient picks up.
🎯 Interview Angle
How to scale groups with 1M members? (topic shards, partial delivery, digest)
E2EE + search? (on-device index only)
Reliability with flaky networks? (retries, store-and-forward, idempotent message IDs)
Key Takeaways
E2EE dictates server blindness; design shifts to client.
2B users, 100B messages/day ≈ ~1.15M messages/sec. Average message size: ~100 bytes (text). Daily storage: 100B × 100 bytes = 10 TB/day for text alone. With media: ~50 TB/day. Connection state: 500M concurrent WebSocket connections, each consuming ~10KB memory = ~5 TB of connection state across servers. This is why WhatsApp needs thousands of Erlang servers optimized for concurrent connections.
5) System Design of TikTok (Short-Video Feed)
Endless personalized short-video feed with creation, editing, sound tracks, comments, and shares. Why it matters:High-throughput video CDN, real-time personalization, heavy ML ranking, creator pipeline.
Core Concepts
Upload Pipeline: Transcode to ABR, thumbnail sprites, audio track extraction.
Feature/Embedding Store: Viewer & video embeddings; real-time features (watch time, replays).
Multi-armed bandits is a strategy from probability theory (named after slot machines). The idea: when you do not know which content will perform best (cold start), show different options to different users and gradually learn which ones get more engagement. It balances "exploiting" what you already know works vs. "exploring" new content that might work better — perfect for a recommendation system handling new videos with no watch history.
Key Takeaways
Multi-stage ranking with fast feedback is the core.
CDN + prefetch deliver smooth playback.
Safety & moderation pipelines are first-class.
Part 2: Problems 6 -- 17
6) Online Coding Judge -- Part 1 (Submissions & Execution)
An online judge executes user code in sandboxes, validates against test cases, and reports verdicts (AC/WA/TLE/MLE/RE). Why it matters: Mixes secure sandboxing, queueing, isolation, autoscaling, and result streaming.
Core Concepts
Submission API → Queue (Kafka/SQS) → Executor Pool (containers/VMs)
Test Runners: Hidden & sample tests; deterministic environments; language runtimes.
Artifacts: Compiler output, logs, runtime stats.
What are seccomp, cgroups, namespaces, and AppArmor? These are Linux security features: seccomp restricts which system calls a process can make (e.g., block network access so submitted code cannot make HTTP requests). cgroups (control groups) limit how much CPU, memory, and disk a process can use (e.g., max 256MB RAM, 2 seconds CPU time). Namespaces isolate processes so they cannot see each other's files or processes. AppArmor adds additional access control rules. Together, they create a secure sandbox — the submitted code runs in a locked-down environment.
The Sandbox Execution Challenge (Defense-in-Depth): The core challenge of a coding judge is running untrusted code safely. Every submission could be malicious — it could try to access the filesystem, make network calls, fork-bomb the CPU, or allocate infinite memory. The solution is defense-in-depth: Linux namespaces isolate the process (it cannot see other processes), cgroups limit resources (max 256MB RAM, 2s CPU), seccomp blocks dangerous system calls (no network, no filesystem writes outside the sandbox). Each layer catches what the others might miss. Think of it as a series of locked doors — even if an attacker picks one lock, the next door stops them. This layered approach is why platforms like LeetCode and Codeforces can safely run millions of untrusted code submissions per day without compromising their infrastructure.
Real-World Examples
Codeforces/LeetCode/HackerRank use container pools; pre-warm popular language images.
How plagiarism detection works:AST (Abstract Syntax Tree) similarity compares the structure of two programs, ignoring variable names and formatting — so renaming variables will not fool it. Token shingling breaks code into overlapping groups of N tokens (like a sliding window) and compares the fingerprints. Minhash is a technique that efficiently estimates how similar two sets of fingerprints are without comparing every pair. MOSS (Measure Of Software Similarity) is Stanford's widely-used system that combines these techniques.
The Auto-Scaling Challenge (Contest Spikes): The tricky part of a coding judge at scale: contest spikes. A normal day might see 1,000 submissions/hour, but when a contest starts, you get 50,000 in the first 5 minutes. Pre-provisioning for peak wastes money 99% of the time. The solution: a message queue (SQS/Kafka) buffers submissions, auto-scaling spins up judge workers based on queue depth, and each worker pulls jobs independently. Users see their submission enter a “Queued” state and get results as workers drain the backlog. The key metric to watch is queue depth — when it crosses a threshold, the scaling policy launches new executor containers. When the contest ends and the queue drains, workers scale back down. This elastic approach keeps costs proportional to actual demand rather than peak capacity.
Real-World Examples
Live scoreboards with incremental updates via WebSockets/SSE.
Pros & Cons
Streaming Boards
Real-time experience but requires idempotent updates.
Batch Boards
Simple implementation but laggy user experience.
Exact vs Approximate Similarity
Approach
Method
Best For
Exact
AST comparison
Detecting structural plagiarism
Approximate
N-gram shingling / minhash
Scalable fuzzy matching
Step-by-Step Walkthrough
Verdict Events
Submission verdicts → event bus.
Leaderboard Update
Leaderboard aggregator updates ranks & penalties.
Live View
Web clients consume topic updates → live view.
Plagiarism Check
Offline plagiarism jobs run post-contest.
🏆
Sports Scoreboard
A sports scoreboard updating live after every play.
🎯 Interview Angle
Rejudging impact? (immutable history + compensating updates)
Real-time bank-to-bank payments via VPA/QR on NPCI rail (UPI). Why it matters:Low-latency, idempotency, security, reconciliation, high availability.
Core Concepts
Participants: PSP app, Payer Bank, Payee Bank, NPCI Switch.
Flows: Collect (pull) & Pay (push).
Security: Device binding, PIN, TPIN, tokenization, PSP + bank certs.
IDs: TxnId, RRN; strict idempotency.
UPI Terminology:
VPA (Virtual Payment Address): Your UPI ID (e.g., yourname@upi) — replaces bank account numbers for payments
PSP (Payment Service Provider): The app you use (Google Pay, PhonePe, Paytm) that connects to UPI
NPCI (National Payments Corporation of India): The central switch that routes all UPI transactions between banks
TPIN (Transaction PIN): The 4/6-digit PIN you enter to authorize a payment
RRN (Retrieval Reference Number): A unique ID for each transaction, used for tracking and dispute resolution
The Two-Phase Transaction Challenge (Atomicity Across Banks): UPI's core design challenge is atomicity across two banks. When you send money, your bank must debit AND the receiver's bank must credit — but they are separate systems. UPI uses a two-phase approach: Phase 1 (Collect): NPCI asks the sender's bank to place a hold on the funds. Phase 2 (Pay): NPCI tells the receiver's bank to credit. If Phase 2 fails, Phase 1 is rolled back. The entire flow happens in under 2 seconds across 4 entities (your app → your bank → NPCI → receiver's bank). This is conceptually similar to the two-phase commit protocol in distributed databases, but adapted for the financial domain with strict timeout-based rollback guarantees and reconciliation jobs that run periodically to catch any edge cases.
Concurrency Control: Pessimistic locks or logical reservation tokens.
Queueing: Virtual waiting room for surges.
Payments: Multi-PG fallback, time-bound holds.
The Inventory Reservation Challenge (Tatkal Booking): IRCTC's hardest problem: 10 million users trying to book the same 500 seats simultaneously during Tatkal booking (opens at 10:00 AM sharp). The core challenge is preventing overselling while maintaining responsiveness. IRCTC uses a soft-hold mechanism: when you select a seat, it is temporarily reserved (soft hold) for ~10 minutes. If you complete payment, the hold becomes a confirmed booking (hard lock). If you don't pay, the hold expires and the seat returns to the pool. This prevents both overselling and permanent seat-blocking by indecisive users. The virtual waiting room acts as a first gate — only a controlled number of users are let into the actual booking flow at a time, reducing contention on the inventory database. Think of it as a bouncer at a nightclub: the venue (database) has a fixed capacity, so the bouncer (waiting room) meters the inflow to prevent chaos inside.
Ingest, transcode, package, QC, DRM, and publish videos to the catalogue.
Core Concepts
Transcode Farm: ABR ladder; per-title encode optimization.
Packaging: HLS/DASH CMAF.
DRM: Widevine / FairPlay / PlayReady.
QC: Automated (PSNR/VMAF) + manual.
Metadata: Title, audio/subs locales.
The Encoding Challenge (1,200+ Versions per Movie): When Netflix receives a new movie, it does not just store one file — it creates 1,200+ versions. Each version is a different combination of: resolution (480p, 720p, 1080p, 4K), bitrate (for different internet speeds), codec (H.264 for old devices, H.265/AV1 for new), audio track (English, Spanish, Hindi…), and subtitle track. A single 2-hour movie can generate 8+ TB of encoded files. This encoding farm runs on thousands of parallel workers, splitting the movie into chunks that are encoded independently and reassembled — turning a 10-hour encoding job into a 15-minute one. Netflix's per-title encoding optimization further analyzes each movie's visual complexity: an animated film can look great at a lower bitrate than a fast-action scene, so the encode ladder is customized per title to minimize bandwidth without sacrificing perceived quality (measured by VMAF scores).
Order Orchestration: Payment, KDS integration, prep time prediction.
Dispatch: Driver ETA, batching, assignment optimization.
Tracking: GPS streams; map-matching.
The Real-Time Matching Challenge (Order-to-Driver Assignment): DoorDash's core algorithm challenge: matching orders to drivers optimally. It is not just “assign the nearest driver.” The system considers: (1) food preparation time (no point sending a driver if the food is not ready for 20 mins), (2) driver's current delivery route (a driver already heading to that restaurant area is better than a closer idle driver), (3) batching opportunities (can this driver pick up two nearby orders?), (4) estimated delivery time promise (can we still meet the customer's expected time?). This is solved as a real-time optimization problem, re-evaluated every few seconds as conditions change. The dispatch engine collects all unassigned orders and available drivers in a short batch window (typically 2–4 seconds), then runs a global optimization that considers all these factors simultaneously — producing assignments that are significantly better than a simple greedy nearest-driver approach.
Real-World Examples
ElasticSearch for search; Redis for hot store; Kafka for events.
Pros & Cons
Fast ETAs
Real-time ETA predictions improve customer trust and experience.
Batching Trade-off
Batching reduces cost but risks freshness of assignments.
Greedy vs Global Optimizer
Approach
Description
Trade-off
Greedy
Assign nearest available driver immediately
Fast but sub-optimal globally
Global Optimizer
Batch window + optimization across all orders
Better utilization, higher latency
Step-by-Step Walkthrough
Order
Search restaurant → place order → auth/capture payment.
Merchant Confirms
Merchant confirms; prep time predicts ready-by.
Dispatch
Dispatch selects driver; live tracking.
Delivery
Delivery + rating + refunds workflow.
👨🍳
Restaurant Manager During a Dinner Rush
Another way to think about DoorDash: it is like a Restaurant Manager during a dinner rush — simultaneously tracking which tables (orders) are waiting, which servers (drivers) are available, which kitchen stations (restaurants) are backed up, and dynamically reassigning staff to keep wait times low for every table.
🎯 Interview Angle
ETA accuracy? (historical features + real-time traffic & prep time)
Key Takeaways
Reliable orchestration, accurate ETAs, resilient payments & support flows.
12) Amazon Online Shops (Marketplace)
Multi-tenant e-commerce platform with catalog, search, carts, payments, fulfillment.
Strong consistency for payments/inventory prevents overselling.
Read Scalability
Read scalability for browse requires heavy caching and eventual consistency.
Event Sourcing vs CRUD
Approach
Description
Trade-off
Event Sourcing
Append-only log of state changes
Full audit trail, replay; higher complexity
CRUD
Direct update of current state
Simpler; loses history
Event Sourcing explained: Instead of storing just the current state (like a CRUD database that overwrites the old value), you store every event that led to the current state. For example, instead of "balance = $500," you store: "deposited $1000," "withdrew $300," "deposited $200," "withdrew $400." To get the current balance, replay all events. This gives you a complete audit trail, the ability to rebuild state from any point in time, and undo capability — critical for financial systems.
The Inventory & Ordering Challenge: Amazon processes ~66,000 orders per hour. The design challenge: showing accurate inventory counts across millions of products while thousands of people try to buy the same item simultaneously. Amazon uses eventual consistency for the product catalog (a slight delay in updating "only 3 left" is acceptable) but strong consistency for the actual purchase transaction (you must never sell more than you have). The shopping cart is stored per-user and persisted even across devices — add something on your phone, see it on your laptop.
Typical Amazon Order Flow
Browse/Search: User queries hit the search service (Elasticsearch), which returns products ranked by relevance, price, and reviews
Product Page: Data assembled from multiple services — catalog (details), pricing (current price), inventory (availability), reviews (ratings)
Add to Cart: Cart service stores items with a TTL; prices are re-validated at checkout to prevent stale-price exploits
Checkout: Order service reserves inventory (soft lock), processes payment, and only on payment success converts to a confirmed order
Fulfillment: Order event triggers warehouse assignment (nearest warehouse with stock), picking, packing, and carrier handoff
Delivery Tracking: Real-time location updates streamed from carrier APIs, displayed to user via polling or push notifications
Step-by-Step Walkthrough
Browse & Search
Browse/search → PDP → Add to cart.
Checkout
Checkout: address, shipping, payment auth.
Reserve & Capture
Reserve inventory → capture → create shipment.
Track & Support
Track & notify; returns/refunds.
🏬
Giant Mall
A giant mall with unified checkout and logistics brain.
Routing: Graph (road network), Dijkstra/A*; live traffic speed overlays.
Traffic: Probe data ingestion; speed estimation; incidents.
The Map Tile & Routing Challenge: Google Maps serves billions of map tile requests daily. The world map is divided into tiles at multiple zoom levels — zoom level 0 is 1 tile (the whole world), zoom level 20 is ~1 trillion tiles (individual buildings). When you pan or zoom, your app requests only the tiles visible on screen. These tiles are pre-rendered and cached on CDNs worldwide. For routing, Maps uses a modified Dijkstra's algorithm with precomputed "highway hierarchies" — it doesn't explore every street; it quickly jumps to major roads for long-distance routes and only explores local streets near the start and destination.
Real-World Examples
CDN tiles; incremental map updates; mobile offline packs.
Pros & Cons
Vector Tiles
Reduce bandwidth significantly with client-side rendering.
Client Power
Vector tiles need client rendering power; older devices struggle.
Raster vs Vector Tiles
Aspect
Raster Tiles
Vector Tiles
Rendering
Server-side (pre-rendered images)
Client-side (GPU accelerated)
Bandwidth
Higher (image data)
Lower (geometry data)
Flexibility
Fixed style
Dynamic styling on client
Offline
Large cache needed
Compact offline packs
Step-by-Step Walkthrough
Pan/Zoom
Pan/zoom → tile requests to CDN.
Search
Search → geocoder → candidates.
Route
Route → path computation with weights (traffic).
Live Rerouting
Live rerouting via probe data streams.
🧅
Layered Onion
A layered onion: base map, roads, POIs, then live traffic on top.
The Email Storage & Search Challenge: Gmail stores 1.8 billion users' emails — that's petabytes of data. The design challenges: (1) Storage: emails are stored in a distributed file system (like Bigtable), sharded by user ID. Attachments are stored separately in blob storage and referenced by pointer. (2) Search: full-text search across years of email in under a second requires an inverted index — every word maps to the list of emails containing it, pre-built and updated on every incoming email. (3) Spam filtering: ML models score every incoming email in real-time, using signals like sender reputation, content analysis, and user behavior (what you've marked as spam before).
Real-World Examples
Priority Inbox; Smart Reply; offline via local index.
Pros & Cons
Heavy Indexing
Improves search speed and relevance dramatically.
Cost
Indexing costs storage/CPU. Maintaining real-time index on write path adds latency.
Folders vs Labels
Approach
Description
Trade-off
Folders
Single location per message
Simple but rigid; one category only
Labels
Multi-classification per message
Flexible; allows message in multiple categories
Step-by-Step Walkthrough
SMTP Receive
SMTP receive → store message once; link to recipients.
Index & Classify
Index text/metadata; classify spam.
Thread & Notify
Update threads & labels; notify clients.
Search
Search retrieves via inverted index; snippets cached.
📚
Library
A library where a single book can be shelved in multiple sections (labels).
🎯 Interview Angle
Defend against spam floods? (rate limits, greylisting, reputation)
The Real-Time Game State Challenge: The hardest part of online chess isn't the game logic — it's real-time state synchronization with fairness. Each move must be: (1) validated server-side (a client could send illegal moves), (2) broadcast to the opponent in under 100ms (any higher and the game feels laggy), (3) accurately timed (chess clocks must be fair — network latency can't steal a player's time). The solution: moves are sent via WebSocket, validated by the server, and the server acts as the source of truth for the clock. The server adjusts clock deductions to compensate for measured network round-trip time, so a player with higher latency isn't penalized.
Real-World Examples
Spectator mode via pub-sub; tournaments with brackets.
Pros & Cons
Low Latency
Critical for competitive play -- moves must feel instant.
Fairness
Lag compensation is hard; network advantage is real.
P2P vs Server Authoritative
Approach
Description
Trade-off
P2P
Direct connection between players
Lower latency, but vulnerable to cheating
Server Authoritative
Server validates all moves
Prevents cheating; slight added latency
Step-by-Step Walkthrough
Queue & Match
Queue for match → matchmaking → start room.
Play
Moves validated → broadcast to both; update clocks.
Result
Result stored; analysis job enqueued.
Ratings
Ratings updated; leaderboards.
♚
Arbiter
An arbiter supervising many boards simultaneously.
🎯 Interview Angle
Prevent move spoofing? (signed events, server validation)
The Location Matching Challenge: Uber's core technical challenge: matching riders to drivers in real-time across millions of concurrent users. Drivers send GPS updates every 4 seconds. These locations are stored in a spatial index (using geohashing or an in-memory grid). When a rider requests a ride, the system: (1) queries the spatial index for nearby available drivers, (2) estimates ETA for each using real-time traffic data, (3) ranks candidates by ETA + driver rating + acceptance probability, (4) sends a request to the top candidate with a 15-second timeout. If declined, it moves to the next candidate. The entire matching happens in under 5 seconds.
Typical Uber Trip Flow
Request: Rider enters destination; the app calculates fare estimate using distance, time, demand (surge pricing), and route
Match: Dispatch service queries the spatial index for nearby drivers, ranks them, and sends the trip offer
Accept: Driver accepts within the timeout; both rider and driver see each other's location in real-time via WebSocket
Navigate: Turn-by-turn navigation with real-time traffic; ETA updates streamed to the rider
Complete: Trip ends; fare calculated from actual route/time; payment processed asynchronously
Rate: Both parties rate each other; ratings feed back into the matching algorithm for future trips
Real-World Examples
Kafka for telemetry; Redis for hot geo-indexes; ML for ETA & surge.
Pros & Cons
Fast Matching
Quick driver assignment improves rider experience and utilization.
Fairness
Balancing fast matching vs fairness/driver utilization is an ongoing challenge.
20M rides/day ≈ ~230 rides/sec. 5M active drivers sending GPS every 4 seconds = ~1.25M location updates/sec. Each update: ~100 bytes (lat, lng, timestamp, driver_id). Daily location data: 1.25M × 100 bytes × 86,400 sec ≈ ~10 TB/day. The spatial index must handle 1.25M writes/sec and support radius queries completing in <50ms. This is why Uber uses an in-memory geospatial index, not a traditional database.
17) Google Docs (Realtime Collaborative Editing)
Multi-user collaborative editor with live cursors, comments, and version history.
Core Concepts
Concurrency Control: OT (Operational Transform) or CRDTs (Conflict-free Replicated Data Types).
Storage: Document snapshots + op logs; compaction.
Permissions: ACL/RBAC with fine-grained sharing.
The Collaboration Challenge: Google Docs supports 100+ simultaneous editors on the same document. The core challenge: when two people type at the same position at the same time, whose edit wins? Neither — both edits are preserved using Operational Transformation (OT). Every keystroke is sent to the server as an "operation" (e.g., "insert X at position 5"). The server transforms conflicting operations so they can be applied in any order and produce the same result. The key insight: instead of locking the document or forcing users to take turns, OT makes all edits compatible by mathematically adjusting positions. Google Docs uses OT; newer tools like Figma use CRDTs which achieve the same goal without a central server.
Real-World Examples
Google Docs uses OT lineage; others use CRDTs (Yjs/Automerge).
Pros & Cons
OT
Mature, server-centric; complex transforms but battle-tested at Google scale.
CRDT
Offline-friendly, eventual convergence; larger payloads and metadata overhead.
OT vs CRDT
Aspect
OT
CRDT
Architecture
Server-centric
Peer-to-peer / decentralized
Offline Support
Limited
Strong (merge on reconnect)
Complexity
Transform functions
Data structure design
Payload Size
Smaller ops
Larger metadata
Maturity
Battle-tested (Google)
Growing (Yjs, Automerge)
How OT and CRDT handle the same conflict: Two users are editing "HELLO". User A inserts "X" at position 2 (producing "HXELLO"). User B deletes the character at position 4 (producing "HELL"). With OT: A central server receives both operations. It transforms B's "delete at position 4" to account for A's insertion — since A added a character before position 4, B's operation becomes "delete at position 5." Both operations are applied in order. With CRDT: Each character has a unique ID (not just a position). User A's insert and User B's delete reference specific character IDs, so they can be applied in any order on any device and converge to the same result — no central server needed.
No matter the domain — streaming, messaging, payments, ride-hailing, or collaborative editing — the same building blocks appear again and again. Master these patterns and you can design any system:
Twitch QoS, Netflix QC, UPI reconciliation, all systems
The Golden Rule
There's no "best" choice — only the best choice for a given context. Great system designers justify decisions based on business and technical constraints.
Module 13
AI/ML System Design
10 Topics — Essential for 2026 AI/ML Interviews
13-1: ML Model Serving Architecture
How do you take a trained ML model and serve predictions to millions of users in real-time?
Analogy — Restaurant Kitchen: Training a model is like developing a recipe (months of experimentation). Serving is like running a restaurant kitchen — you need to cook that recipe thousands of times per minute, consistently, without burning anything, and handle dinner rush spikes.
Core Concepts
Online Inference (Real-time): Model serves predictions on-demand, typically < 100ms. Used for recommendations, search ranking, fraud detection. Requires low-latency infrastructure.
Batch Inference (Offline): Model processes large datasets periodically (hourly/daily). Used for generating email recommendations, computing embeddings for a catalog, pre-ranking. Cheaper but stale.
Model Server: Specialized software that loads a trained model, accepts prediction requests via API, handles batching/preprocessing. Examples: TensorFlow Serving, Triton Inference Server, vLLM (for LLMs), TorchServe.
GPU vs CPU Inference: Deep learning models (neural networks) run 10-100x faster on GPUs. Simple models (XGBoost, logistic regression) run fine on CPUs. GPU instances cost 5-10x more — use them only when needed.
Quantization (INT8/INT4): Model weights are typically stored as 32-bit floats (FP32). Quantization converts them to lower precision — INT8 (8-bit integers) or even INT4 (4-bit). This shrinks model size by 4-8x, speeds up inference by 2-4x, and reduces GPU memory usage dramatically. The trade-off: a small accuracy drop (usually < 1%). For LLM serving, quantization is now standard — it's the difference between needing 4 GPUs vs 1 GPU to serve a model. Tools: GPTQ, AWQ, bitsandbytes, ONNX Runtime. In interviews, mentioning quantization shows you understand the practical economics of model deployment, not just the ML theory.
Request Batching: Instead of processing one prediction at a time, wait a few milliseconds to collect multiple requests and process them as a batch. GPUs are most efficient with batches (e.g., batch size 32). Trade-off: slightly higher latency for much higher throughput.
Model Warm-up: First prediction after loading a model is slow (loading weights into GPU memory). Pre-warm models by sending dummy requests before routing real traffic.
Deployment Strategies for ML Models:
Shadow Mode: New model runs alongside the old one. Both process every request, but only the old model's predictions are served to users. Compare results to validate the new model before switching.
Canary Deployment: Route 5% of traffic to the new model, 95% to the old. Monitor key metrics (accuracy, latency, error rate). Gradually increase if metrics are healthy.
A/B Testing: Split users into two groups. Group A gets old model, Group B gets new. Run for days/weeks, measure business metrics (CTR, revenue, engagement). Requires statistical significance.
Interview Tip: When asked "how would you deploy a new model?", always mention shadow mode → canary → gradual rollout. Never say "just swap the model" — that's how production breaks.
Key Takeaway: Model serving is not just "load model, call predict()." It requires batching, GPU management, model versioning, deployment strategies, autoscaling, and monitoring — essentially, it's a microservice with ML-specific challenges.
13-2: Feature Stores
The bridge between data engineering and ML — ensuring consistent features across training and serving.
Analogy — Prep Kitchen: A feature store is like a restaurant's prep kitchen. Instead of each chef (model) individually chopping onions and marinating meat, the prep kitchen does it once and serves ready-to-use ingredients. Every chef uses the same prepped ingredients — ensuring consistency.
Core Concepts
Feature: A measurable property used as input to an ML model. Examples: "user's average session duration (last 7 days)", "number of items in cart", "time since last login."
Training-Serving Skew: The #1 problem feature stores solve. If your training pipeline computes "average order value" differently than your serving pipeline, the model sees different data in production than it was trained on — predictions degrade silently. A feature store ensures the SAME computation is used everywhere.
Online Store: Low-latency key-value store (Redis, DynamoDB) for real-time serving. Query: "give me user_123's features" → response in <10ms.
Offline Store: Data warehouse (S3, BigQuery) for training. Contains historical feature values. Query: "give me all users' features as of 2024-01-15" for training a model on historical data.
Point-in-Time Correctness: When training, you must use features as they existed WHEN the event happened — not current values. Example: predicting whether a user will click an ad on Jan 15th must use the user's features from Jan 15th, not today. Using today's features would be "data leakage" and the model would fail in production.
Popular Feature Store Tools:
Feast: Open-source, integrates with any data source. Good starting point.
Hopsworks: Open-source with both online and offline stores.
Vertex AI Feature Store: Google Cloud managed feature store.
Interview Tip: When designing any ML system, always mention feature stores and training-serving skew. It shows you understand the operational reality of ML, not just the modeling.
Key Takeaway: Feature stores centralize feature computation, prevent training-serving skew, and provide both low-latency online serving and historical offline access. They are the unsung hero of production ML.
13-3: Training Pipelines & MLOps
The end-to-end lifecycle of an ML model — from raw data to production deployment.
The ML Lifecycle
Data Ingestion: Collect raw data from databases, event streams, APIs, and logs. Store in a data lake (S3, GCS).
Data Validation: Check for schema changes, missing values, distribution shifts. Tools: Great Expectations, TensorFlow Data Validation.
Feature Engineering: Transform raw data into features. Compute aggregates, embeddings, encodings. Store in the feature store.
Model Training: Train model on historical features + labels. Track experiments (hyperparameters, metrics, artifacts). Tools: MLflow, Weights & Biases.
Model Evaluation: Validate on held-out test set. Check for bias, fairness, performance across segments. Compare against the current production model.
Model Registry: Version and store the trained model artifact. Tag as "staging" or "production." Tools: MLflow Model Registry, Sagemaker Model Registry.
Deployment: Deploy to model serving infrastructure. Shadow mode → canary → full rollout.
Monitoring: Track prediction quality, latency, data drift. Alert on degradation. Trigger retraining if needed.
Automated Retraining:
Models degrade over time as the world changes (concept drift). A retraining pipeline automatically: (1) Detects drift via monitoring alerts, (2) Pulls fresh training data, (3) Retrains the model, (4) Validates against the current model, (5) Deploys if metrics improve. This can run on a schedule (weekly) or be triggered by drift detection. Tools: Kubeflow Pipelines, Apache Airflow, Vertex AI Pipelines.
Distributed Training:
Large models (LLMs, vision transformers) can't fit on one GPU. Two approaches:
Data Parallelism: Same model on multiple GPUs, each processing a different batch. Gradients are averaged across GPUs. Scales well up to ~64 GPUs.
Model Parallelism: Model is split across GPUs — each GPU holds a portion of the model. Needed when the model itself doesn't fit in one GPU's memory (e.g., large LLMs with hundreds of billions+ parameters that exceed a single GPU's memory). Two sub-types:
Tensor Parallelism: A single layer's weight matrix is split across GPUs. Each GPU computes part of the same layer simultaneously. Requires fast GPU-to-GPU communication (NVLink). Best for very wide layers.
Pipeline Parallelism: Different layers are assigned to different GPUs. GPU 1 processes layers 1-10, GPU 2 processes layers 11-20, etc. Data flows through GPUs like an assembly line. Simpler communication but can have "bubble" idle time between micro-batches.
In practice, large-scale training (GPT-class models) combines all three: data parallelism across clusters, pipeline parallelism across GPUs within a node, and tensor parallelism within each layer. This is called 3D parallelism.
Interview Tip: When asked about ML pipelines, walk through all 8 steps. Most candidates jump straight to "train the model" — showing you know the full lifecycle (especially data validation, experiment tracking, and monitoring) demonstrates real-world ML experience.
13-4: RAG — Retrieval-Augmented Generation
The most asked AI system design question in 2026. How to make LLMs answer questions using YOUR data.
Analogy — Open-Book Exam: A plain LLM is like taking a closed-book exam — it can only answer from what it memorized during training. RAG is like an open-book exam — the LLM can look up relevant information from your documents before answering. The quality of the answer depends on how good your "book index" is.
The RAG Pipeline
Document Ingestion: Load your documents (PDFs, web pages, internal wikis, code repos).
Chunking: Split documents into smaller pieces (chunks). Typical size: 200-500 tokens. Overlap chunks by 10-20% so context isn't lost at boundaries. Chunking strategy significantly affects retrieval quality.
Embedding: Convert each chunk into a vector (a list of numbers that captures meaning) using an embedding model (OpenAI ada-002, Cohere embed, open-source models like BGE). Similar content produces similar vectors.
Vector Storage: Store embeddings in a vector database (Pinecone, Weaviate, Qdrant, pgvector, Milvus). Indexed for fast similarity search.
Query: User asks a question. Convert the question into a vector using the SAME embedding model.
Retrieval: Search the vector database for the top-K most similar chunks to the question vector. This is Approximate Nearest Neighbor (ANN) search.
Re-ranking (optional): Use a cross-encoder model to re-score the retrieved chunks for more accurate relevance ranking.
Generation: Pass the retrieved chunks as context to the LLM along with the user's question. The LLM generates an answer grounded in the provided context.
Hybrid Search — Best of Both Worlds:
Vector search is great for semantic similarity ("What's our refund policy?" matches "return guidelines"). But it can miss exact keyword matches. Hybrid search combines:
BM25 / keyword search: Exact term matching (finds specific product names, error codes)
Results from both are combined using Reciprocal Rank Fusion (RRF) or learned scoring. Most production RAG systems use hybrid search.
Evaluating RAG Quality:
How do you know your RAG system is working well? Key metrics:
Faithfulness: Does the generated answer actually match the retrieved context? (No hallucinations)
Answer Relevancy: Does the answer address the user's question?
Context Recall: Did the retrieval step find all the relevant documents?
Context Precision: Were the retrieved documents actually relevant (no noise)?
Tools: RAGAS (open-source RAG evaluation framework), LangSmith, Phoenix by Arize. Always set up automated evaluation before deploying a RAG system — "it looks right" is not a metric.
Common RAG Pitfalls:
Bad chunking: Chunks too small = lost context. Chunks too large = noisy retrieval. Experiment with chunk size for your data.
Embedding model mismatch: Using different embedding models for indexing vs querying produces garbage results.
No citation/grounding: The LLM can still hallucinate even with context. Always return source references so users can verify.
Stale index: Documents change but the vector index isn't updated. Implement incremental re-indexing.
Interview Tip: RAG is THE most common AI system design question in 2026. Know the full pipeline cold. When asked "Design a Q&A system over internal documents," this is your answer. Key differentiators: mention hybrid search, re-ranking, and chunking strategy.
Key Takeaway: RAG = Retrieve relevant context from your data + Generate answer using an LLM with that context. It's how you make LLMs useful for domain-specific applications without expensive fine-tuning.
13-5: Vector Databases & Similarity Search
The infrastructure that makes RAG, recommendations, and image search possible.
Core Concepts
Vector Embedding: A numerical representation of data (text, images, audio) as a list of floating-point numbers (e.g., 768 or 1536 dimensions). Similar items have vectors that are close in this high-dimensional space.
Similarity Search: Given a query vector, find the K nearest vectors in the database. Distance metrics: cosine similarity (most common for text), Euclidean distance, dot product.
Approximate Nearest Neighbor (ANN): Exact nearest neighbor search is O(n) — too slow for millions of vectors. ANN algorithms trade a small amount of accuracy for massive speed gains (1000x faster).
ANN Indexing Algorithms
Algorithm
How It Works
Strengths
Used By
HNSW
Builds a multi-layer graph where each node connects to its nearest neighbors. Search starts at the top layer (coarse) and drills down (fine).
Best recall, fast search, good for dynamic data
Pinecone, Weaviate, pgvector
IVF
Clusters vectors into groups (Voronoi cells). Search only looks in the nearest clusters instead of all vectors.
Memory efficient, good for large datasets
FAISS (Meta), Milvus
Product Quantization
Compresses vectors by splitting them into sub-vectors and replacing each with a codebook entry. Reduces memory 10-50x.
Not a database — a library. No persistence, but fastest raw search.
Metadata Filtering:
Vector search alone isn't enough. You often need: "find similar products BUT only in the 'Electronics' category AND priced under $100." This is metadata filtering — combining vector similarity with traditional filters. Most vector DBs support this natively, but the implementation matters for performance (pre-filtering vs post-filtering).
Interview Tip: When discussing vector databases, always mention the trade-off between recall (accuracy) and latency. HNSW has ~95-99% recall with sub-millisecond latency. Exact search has 100% recall but 1000x slower. In production, 95% recall is almost always acceptable.
Key Takeaway: Vector databases are to AI applications what relational databases are to CRUD apps — the foundational storage layer. Understanding HNSW, the recall/latency trade-off, and when to use pgvector vs a dedicated vector DB is essential for 2026 interviews.
13-6: A/B Testing & Experimentation Platforms
How do you know if your new model or feature actually improves things?
Analogy — Clinical Trial: A/B testing is like a drug trial. Group A gets the existing treatment (control). Group B gets the new drug (variant). You measure outcomes over time and use statistics to determine if the new drug actually works — not just random chance.
Core Concepts
Control vs Variant: Control = existing experience. Variant = new model/feature/UI. Users are randomly assigned to groups.
Statistical Significance: How confident are you that the observed difference isn't random noise? Typically require p-value < 0.05 (95% confidence). Avoid "peeking" at results early — this inflates false positives.
Sample Size: Need enough users in each group to detect meaningful differences. Small effect sizes need larger samples. Use power analysis to calculate upfront.
Guardrail Metrics: While testing a new recommendation model for engagement, also monitor: latency, crash rate, revenue, user complaints. A model that increases clicks but increases load time by 2x is not a win.
Feature Flags: Toggle features on/off without deploying new code. Tools: LaunchDarkly, Unleash, Statsig. Essential for gradual rollouts and instant kill-switches.
Multi-Armed Bandits: Instead of fixed 50/50 splits, dynamically allocate more traffic to the winning variant as data comes in. Faster convergence but harder to achieve statistical rigor (traffic allocation keeps shifting, so you never have a clean, stable control group to compare against). Good for short-lived optimizations (ad selection, headline testing).
Interleaving: For ranking systems (search, recommendations), show results from BOTH models mixed together in one feed. Measure which model's results get more clicks. More sensitive than traditional A/B tests — needs fewer users. Used by Netflix and Spotify.
Interview Tip: When designing any ML system, always include an experimentation layer. "How do you know the new model is better?" is a guaranteed follow-up question. Mention A/B testing, guardrail metrics, and statistical significance.
13-7: ML Monitoring & Drift Detection
Models degrade silently. Without monitoring, you won't know until users complain.
Types of Drift
Type
What Changes
Example
Detection
Data Drift
Input distribution shifts
COVID changed user behavior — model trained on pre-COVID data sees completely different patterns
Statistical tests (KS test, PSI) comparing training vs live input distributions
Concept Drift
Relationship between inputs and outputs changes
A fraud model trained when "international purchases" were rare. Now they're common — the same signal means something different.
Monitor prediction accuracy over time. Sliding window metrics.
Prediction Drift
Model output distribution changes
Model suddenly predicts 80% "spam" instead of usual 30%
Track prediction distribution histograms over time
What to Monitor
Model Performance: Accuracy, precision, recall, F1 — but measured on LIVE data, not just test sets. Requires delayed ground truth labels (did the user actually click? did the transaction turn out to be fraud?).
Feature Distributions: Are input features still in the range the model was trained on? If a feature goes from mean=50 to mean=500, something is wrong upstream.
Business Metrics: CTR, conversion rate, revenue per user. The ultimate measure — a model can have great accuracy but hurt business metrics.
Automated Remediation:
When drift is detected: (1) Alert the ML team. (2) Optionally auto-rollback to the previous model version. (3) Trigger an automated retraining pipeline with fresh data. (4) Deploy retrained model via the standard shadow → canary → rollout process.
Interview Tip: "How do you handle model degradation?" is a senior-level question. Mention data drift, concept drift, monitoring dashboards, and automated retraining. This separates L4 from L5 candidates.
13-8: Recommendation System Design
The system behind "You might also like..." — powering Netflix, YouTube, TikTok, Amazon, and Spotify.
The Recommendation Pipeline
Candidate Generation (~1000 items): From millions of items, narrow down to ~1000 candidates using fast, approximate methods. Techniques: collaborative filtering ("users like you also watched X"), content-based filtering ("similar to what you watched"), embedding-based retrieval (ANN search in a vector index).
Scoring/Ranking (~100 items): A more precise ML model scores each candidate. Uses rich features: user history, item metadata, context (time of day, device). This is typically a deep learning model.
Re-Ranking (~50 items): Apply business rules and diversity filters. Remove duplicates, enforce freshness (don't show only old content), add diversity (don't show 10 action movies in a row), filter explicit content.
Serving: Return the final ranked list to the user. Log the impressions for future training data (feedback loop).
Two-Tower Model (Industry Standard):
Separate neural networks for users and items. The User Tower encodes user features into an embedding. The Item Tower encodes item features into an embedding. Relevance = dot product of both embeddings (multiply corresponding numbers and sum them — higher score means more similar, like measuring how much two arrows point in the same direction).
Why two towers? Item embeddings can be precomputed and indexed in a vector DB. At serving time, you only compute the user embedding (fast), then do ANN search against precomputed item embeddings. This makes candidate generation feasible at scale.
Cold Start Problem:
New user: No history to personalize from. Solutions: popularity-based recommendations, ask preferences during onboarding, use demographic features.
New item: No engagement data. Solutions: content-based features (genre, description), multi-armed bandit exploration (show to some users and learn from engagement), editorial curation.
Interview Tip: Always structure your answer as the 4-stage pipeline (Candidate Generation → Scoring → Re-Ranking → Serving). Mention the feedback loop. This shows you understand the engineering, not just the ML.
13-9: LLM-Powered Application Architecture
How to build production systems around Large Language Models — the defining challenge of 2025-2026.
Core Architecture Components
Prompt Management: System prompts, templates, and version control for prompts. Small prompt changes can dramatically affect output quality. Treat prompts like code — version them, test them, review them.
Context Window Management: LLMs have limited context windows (8K-128K+ tokens). When your RAG retrieves 50 relevant chunks but the context window fits 10, you need smart selection, summarization, or hierarchical retrieval.
Guardrails: Input validation (block prompt injection attempts, PII detection), output validation (check for harmful content, verify factual claims against sources, format enforcement).
Streaming Responses: LLMs generate tokens one at a time. Use Server-Sent Events (SSE) to stream tokens to the client as they're generated — the user sees the response building in real-time instead of waiting 5-10 seconds for the full response.
Model Routing: Not every query needs GPT-4. Route simple queries to smaller/cheaper models (GPT-3.5, Haiku) and complex queries to powerful models (GPT-4, Opus). Use a classifier or heuristic to decide. This can cut costs 5-10x.
Semantic Caching: If two users ask semantically similar questions ("What's the return policy?" and "How do I return an item?"), cache the first response and return it for the second. Use embedding similarity to match queries. Saves LLM costs and reduces latency.
LLM Safety & Security:
Prompt Injection: Malicious users try to override your system prompt. Defense: input sanitization, separate system/user prompts, output validation.
PII Leakage: The model might output sensitive data from context. Defense: redact PII before passing to the LLM, filter outputs.
Cost Control: LLM API calls are expensive at scale. Defense: caching, model routing, token limits, rate limiting per user.
Multi-Agent Architecture:
For complex tasks, use multiple specialized LLM agents that collaborate. Example: A customer support system might have: (1) Router Agent — classifies the query type, (2) FAQ Agent — handles common questions using RAG, (3) Order Agent — looks up order status via API calls, (4) Escalation Agent — determines when to transfer to a human.
Each agent has its own system prompt, tools, and context. An orchestrator decides which agent handles each query. This is more maintainable and accurate than one monolithic prompt trying to do everything.
Interview Tip: For "Design an AI chatbot/assistant" questions: RAG for knowledge, model routing for cost, streaming for UX, guardrails for safety, semantic caching for efficiency. Hit all five and you'll stand out.
13-10: AI/ML Practice Problems
Common AI/ML system design interview questions with solution sketches.
Problem 1: Design a Recommendation System (Netflix/YouTube)
Key components: Candidate generation (collaborative filtering + embedding ANN search) → ML ranking model (user features + item features + context) → Re-ranking (diversity, freshness, business rules) → Feature store (online: Redis, offline: data warehouse) → A/B testing (interleaving for ranking) → Feedback loop (log impressions + clicks → retrain).
Scale: 200M users, 50K items, ~10K candidates per user per request. Real-time latency: <200ms end-to-end.
Problem 2: Design a RAG-Based Enterprise Q&A System
Critical considerations: Access control (user can only retrieve docs they have permission to see), incremental re-indexing (as docs change), semantic caching, model routing (simple factual → small model, complex reasoning → large model).
Problem 3: Design a Real-Time Fraud Detection System
Key components: Event stream (Kafka) → Real-time feature computation (transaction amount, velocity, geo distance from last transaction, device fingerprint) → Feature store (online: <10ms lookup) → ML model (gradient boosted trees for speed, ~5ms inference on CPU) → Decision engine (auto-block if score > 0.95, manual review if 0.7-0.95, approve if < 0.7) → Feedback loop (analyst labels → retrain weekly).
Latency budget: Total <100ms from transaction initiation to approve/deny. The ML model gets ~5ms; the rest is feature lookup and network.
Problem 4: Design a Content Moderation Pipeline
Key components: Upload triggers moderation → Multi-modal analysis: image classification (NSFW, violence, hate symbols), text analysis (toxicity scoring, PII detection), video analysis (sample frames + audio transcript) → ML models run in parallel → Aggregation: combine scores with weighted rules → Action: auto-remove (high confidence), queue for human review (medium confidence), approve (low score) → Human review interface → Feedback loop for model improvement.
Scale challenge: Instagram gets 100M+ photos/day. Each needs moderation in <2 seconds after upload. Use GPU inference clusters with auto-scaling.
Problem 5: Design a Search Ranking System
Key components: Query understanding (spell correction, query expansion, intent classification) → Candidate retrieval (inverted index + BM25 for text, ANN for semantic) → ML ranking (Learning-to-Rank model using features: query-document relevance, document quality, freshness, user personalization, click history) → Business rules (promoted results, diversity) → A/B testing with interleaving.
Key insight: Search ranking is a recommendation problem where the "user intent" changes with every query. The ranking model must balance relevance (does this result match the query?) with quality (is this a good result in general?) and personalization (does this user prefer certain types of results?).
Module 13 Recap: AI/ML System Design
Model Serving: Online vs batch, GPU batching, shadow → canary → rollout deployment
THE entry-level system design question. If you can only prepare one problem, make it this one.
Core Concepts
Base62 Encoding: Convert a numeric ID to a short string using [a-zA-Z0-9]. ID 12345 → "dnh". 6 characters = 62^6 = ~56 billion unique URLs.
ID Generation: Auto-incrementing DB ID, or a distributed ID generator (Snowflake). Avoid hash collisions by using unique IDs rather than hashing the URL.
Read-Heavy: 100:1 read-to-write ratio. Reads served from cache (Redis). Writes go to DB.
301 vs 302 Redirect: 301 (permanent) = browser caches, less server load but no analytics. 302 (temporary) = every click hits your server, enabling click tracking.
Expiration: Optional TTL on short URLs. Background job cleans expired entries.
Create: User submits long URL → Generate unique ID → Base62 encode → Store mapping in DB → Return short URL
Redirect: User clicks short URL → Check cache (Redis) → Cache miss: query DB → 302 redirect to long URL → Log click for analytics
Analytics: Log click events to Kafka → Process with Spark/Flink → Dashboard shows clicks over time, geographic distribution, referrers
Back-of-Envelope: 100M new URLs/month = ~40 writes/sec. 10B redirects/month = ~3,800 reads/sec. Storage: 100M × 500 bytes = 50 GB/month. 5 years = 3 TB. Easily fits in a single sharded database + Redis cache layer.
Interview Tip: This problem tests fundamentals: hashing, caching, database design, and capacity estimation. Nail these basics and the interviewer knows you're ready for harder questions.
Design a Rate Limiter
Protect your API from abuse. A standalone system design question AND a component in every other design.
Algorithms
Algorithm
How It Works
Pros
Cons
Fixed Window
Count requests in fixed time windows (e.g., 100 req per minute). Reset counter at window boundary.
Simple, low memory
Burst at window edges (200 req in 2 seconds spanning a boundary)
Sliding Window Log
Store timestamp of each request. Count requests in the last N seconds. Remove old entries.
Precise, no boundary issues
High memory (stores every timestamp)
Sliding Window Counter
Combine current + previous window counts weighted by overlap. E.g., 70% through current window: rate = current_count + prev_count × 0.3
Low memory, smooth
Approximate (but good enough)
Token Bucket
Bucket fills at fixed rate. Each request consumes a token. No tokens = rejected.
Allows bursts, simple
Tuning bucket size + refill rate
Leaky Bucket
Requests enter a queue. Queue drains at fixed rate. Full queue = rejected.
Smooth output rate
No bursts, even for legitimate spikes
Distributed Rate Limiting:
With multiple API servers, each server can't track limits independently (user sends 100 req to server A and 100 to server B = 200 total). Solutions:
Centralized counter (Redis): All servers increment a shared counter in Redis. INCR + EXPIRE are atomic. Adds ~1ms latency per request.
Sticky sessions: Route each user to the same server. Simpler but less resilient.
Local + sync: Each server tracks locally and periodically syncs with a central store. Allows slight over-limit but reduces Redis load.
Where to place the rate limiter: API Gateway (most common — centralized, applies to all services), per-service middleware, or as a sidecar proxy in a service mesh. Return HTTP 429 (Too Many Requests) with a Retry-After header.
Interview Tip: Know at least 3 algorithms and when to use each. Token Bucket is the most common default (used by AWS, Stripe). Sliding Window Counter is the best balance of precision and efficiency.
Design a Notification System
Push notifications, email, SMS, in-app — reaching users across every channel.
Core Components
Notification Service: Receives notification requests from other services (order confirmed, new message, etc.). Validates, enriches with user preferences, and routes to the correct channel.
User Preferences: Users control what notifications they receive and on which channels. Store in a preferences DB. Always check before sending — spam kills trust.
Template Engine: Notifications use templates with variables: "Hi {name}, your order #{order_id} has shipped!" Templates are versioned and A/B testable.
Channel Adapters: Separate adapters for each delivery channel: Push (APNs for iOS, FCM for Android), Email (SendGrid, SES), SMS (Twilio), In-App (WebSocket or SSE).
Priority Queue: Not all notifications are equal. Payment failures = urgent (high priority). Weekly digest = low priority. Use priority queues to ensure critical notifications aren't stuck behind bulk sends.
Rate Limiting: Never send more than N notifications per user per hour. Notification fatigue causes users to disable notifications entirely.
Track: Log delivery status. Retry on transient failures. DLQ for permanent failures.
Interview Tip: Key differentiators: mention user preferences check, rate limiting per user, priority queues, and idempotency (same event shouldn't trigger duplicate notifications). These show you think about the user experience, not just the infrastructure.