Rate Limiting and Traffic Shedding at Scale

In a distributed microservices ecosystem, no backend service has infinite capacity.

Without protective boundaries, unexpected traffic spikes, runaway client retry loops, scraping bots, and DDoS attacks will saturate database connection pools and CPU threads, triggering cascading collapse across the entire platform.

To protect system stability, high-scale platforms (Stripe, Cloudflare, Netflix, AWS) implement two complementary layers of defense:

1. Distributed Rate Limiting: Restricting the rate of requests per client/API key.
2. Adaptive Load Shedding: Proactively dropping low-priority background work when servers are under severe CPU/memory strain to protect mission-critical transactions.

1. Rate Limiting Algorithms Compared

2. The Sliding Window Counter Algorithm

To avoid the $2\times$ boundary burst of fixed windows without paying the high memory penalty of sorted set logs, modern gateways use the Weighted Sliding Window Counter:

$\text{Estimated Requests}=(\text{Previous Window Count}\times \text{Overlap Percentage})+\text{Current Window Count}$

If $86>\text{Limit }(100)$, the request is allowed. If $86\ge 100$, it is rejected with HTTP 429 Too Many Requests.

3. Distributed Atomic Execution with Redis Lua Scripts

When running 100 API gateway pods in Kubernetes, tracking limits in local process memory is useless because requests from the same user land on different pods.

To execute rate limiting atomically across servers in $<1\text{ms}$ without distributed locks, we use an Atomic Redis Lua Script:

4. Adaptive Load Shedding: Protecting Saturated Systems

When a massive surge occurs, Rate Limiting alone cannot save your system if the total volume of legitimate users exceeds capacity.

Load Shedding monitors the health of the host machine and proactively rejects low-priority work before the server runs out of memory or CPU threads:

5. Fail-Open vs Fail-Closed Design

What happens if the Redis rate-limiting cluster itself goes down?

6. Production Failure Postmortem: The Fail-Closed Redis Gateway Outage

Incident Overview:

In 2020, a top payment gateway experienced a 22-minute total global outage when a single Redis cluster hosting rate limiter keys suffered memory fragmentation and crashed.

What Happened:

1. The API Gateway was configured with a Fail-Closed policy: if (redisError) throw new RateLimitUnavailableException(500).
2. When Redis crashed during a routine memory defragmentation cycle, the gateway began rejecting 100% of all incoming merchant payment transactions globally.
3. Over \4,!000,!000$ in transactions failed in 22 minutes, even though the primary PostgreSQL database and Stripe upstream integrations were completely healthy and idle at 2% CPU utilization.

Remediation:

• Converted all public API gateway rate limiters to Fail-Open with a 3-second circuit breaker.
• Added a local in-memory fallback token bucket on each pod to catch extreme bursts during Redis downtime.

7. Landmark Capstone #7: Build a Distributed Sliding Window Rate Limiter ⚔️

Celebrate the grand finale of Track 1 by implementing an industrial-strength Distributed Sliding Window Rate Limiter:

👉 Launch Capstone: Build a Distributed Sliding Window Rate Limiter

• Supported Languages: TypeScript & Python 3
• Challenge Focus:
  • Implement a microsecond-accurate Sliding Window Log / Counter.
  • Enforce atomic capacity checks and expiration window management.
  • Return deterministic X-RateLimit-Limit, X-RateLimit-Remaining, and Retry-After headers.
  • Support multi-tier priority load shedding under simulated queue pressure.

8. Chapter Summary & Rate Limiting Rules