A distributed system is a collection of autonomous computing nodes connected over a network that coordinate actions by passing messages, presenting themselves to end users as a unified, coherent system.
Unlike a monolithic program running on a single multi-core server with shared physical RAM, a distributed system operates across independent failure domains with no shared physical memory, no synchronized physical clock, and non-deterministic network latency.
1. The Core Characteristics of Distributed Systems
Every distributed system exhibits three fundamental physical realities:
A. Concurrency and Independent Execution
Each node executes its own instructions at its own CPU clock frequency. Operations on Node A happen concurrently with operations on Node B without global coordination unless explicitly synchronized via consensus protocols.
B. Lack of a Global Shared Clock
Because physical quartz clocks drift due to temperature variations and hardware imperfections, two servers in the same datacenter can drift by milliseconds per day. You cannot determine the exact global chronological order of two events simply by inspecting local system timestamps (Date.now() or System.currentTimeMillis()).
C. Independent Partial Failures
In a single-process application, if the CPU or memory bus fails, the entire process crashes (total failure). In a distributed system, Node B can crash while Node A continues running normally, or the network switch between them can fail while both nodes remain fully operational.
[!IMPORTANT] Leslie Lamport's Definition:
"A distributed system is one in which the failure of a computer you didn't even know existed can render your own computer unusable."
2. Monolith vs. Distributed: The Architectural Shift
When a system transitions from a single-node process to a distributed architecture, function calls transform into network remote procedure calls (RPCs):
| Dimension | Single-Node In-Memory Program | Distributed System |
|---|---|---|
| Communication Mechanism | Direct Memory Pointer / Stack Call | Serialization over TCP/UDP Network Sockets |
| Call Latency | ( slower) | |
| Failure Modes | Success or Process Crash | Success, Explicit Failure, or Unknown State (Timeout) |
| Memory Isolation | Shared Address Space / L1-L3 Cache | Completely Isolated Independent Address Spaces |
| Concurrency Model | Mutexes, Semaphores, Atomic CAS | Distributed Locks, Raft Consensus, CRDTs |
3. The Partial Failure State Machine
When a client invokes a function locally in memory, the invocation is deterministic: either the function returns a value, or it throws an exception.
When Node A invokes an RPC on Node B over the network, three distinct outcomes are possible:
If Node A experiences a network timeout after 5,000ms, Node A cannot distinguish between:
- The request packet never reached Node B (the transfer was never executed).
- Node B crashed mid-execution (partial execution).
- Node B successfully processed the transfer, but the return confirmation packet was dropped by a router on the way back.
This is the Uncertainty Principle of Distributed Computing: A timeout does not indicate failure—it indicates an unknown state.
4. Code Deep-Dive: In-Memory vs. Network-Bound Execution
Consider how error handling and state management change when transitioning from a local function call to an RPC over an asynchronous network.
In-Memory Invocation (Deterministic)
Distributed RPC Invocation (Handling Partial Failure & Timeouts)
5. Production Failure Mode: The Cascading Timeout Storm
A common failure mode in newly distributed systems is the Cascading Timeout Storm.
The Anatomy of the Outage:
- Service C (Payment Gateway) experiences a database lock, causing its latency to increase from to .
- Service B (Order Service) calls Service C synchronously with a timeout. As incoming requests pile up, Service B's thread pool and connection pool are exhausted waiting for Service C.
- Service A (API Gateway) calls Service B with a timeout. Service A immediately times out and begins retrying requests aggressively with zero backoff.
- The retry traffic amplifies incoming request volume by , completely overwhelming Service B and Service C and knocking the entire platform offline.
Prevention Architecture:
- Client-Side Deadlines / Context Propagation: Pass strict decreasing deadlines across service hops ().
- Circuit Breakers: Immediately trip and fast-fail calls to Service C when error rates exceed 10%.
- Exponential Backoff with Full Jitter: Randomize retry delays to prevent synchronized request thundering herds.