Batch vs Stream Processing: The Lambda Architecture vs The Kappa Architecture

For decades, enterprise data engineering was dominated by Batch Processing: daily, hourly, or nightly scheduled jobs (MapReduce, Apache Spark, Snowflake ELT) that processed bounded slices of historical data at high latency.

In modern distributed computing, the demand for instant fraud detection, real-time personalization, and live anomaly monitoring has driven the shift to Continuous Stream Processing.

Understanding this evolution requires examining the two defining paradigms of modern big data architecture: The Lambda Architecture and The Kappa Architecture.

---

1. Bounded vs Unbounded Datasets

• Batch processing is merely a special, artificial subset of stream processing: taking an infinite stream of records and carving it into arbitrary 24-hour static files.
• Stream processing treats data as it naturally exists in the physical world: a continuous, unending timeline of cause-and-effect events.

---

2. The Legacy Lambda Architecture: The Dual-Pipeline Nightmare

Formulated by Nathan Marz in 2011, the Lambda Architecture attempted to balance low-latency insights with historical data accuracy by splitting data processing into two separate pipelines:

The Inherent Flaws of the Lambda Architecture:

1. Dual Codebase Maintenance: Engineers must write and maintain identical business transformation logic in two completely different programming paradigms (e.g., Scala/Java in Spark for batch, and Storm/Flink in streaming).
2. Data Discrepancies and Reconciliations: Because the two engines have different timestamp semantics, windowing logic, and floating-point rounding, the real-time dashboard and the morning batch report constantly disagree, causing operational panic.
3. Operational Overhead: Running, monitoring, and debugging two distinct distributed clusters (e.g., Spark on EMR + Kafka/Flink on Kubernetes) doubles infrastructure and maintenance costs.

---

3. The Modern Kappa Architecture: A Unified Event Streaming Engine

Proposed by Jay Kreps (co-creator of Apache Kafka), the Kappa Architecture eliminates the batch layer entirely:

All data processing—both real-time streaming and historical backfilling—is handled by a single, unified stream processing engine consuming from an append-only distributed commit log.

How Historical Reprocessing Works in Kappa Architecture:

When business logic changes or a bug is fixed in your stream processor:

1. You deploy a new version of your streaming application (Job V2).
2. Job V2 starts reading from events.v1 starting at Offset 0 (the beginning of recorded time).
3. The stream engine processes historical events at maximum hardware speed ($500,000\text{ msg/sec}$) into a new staging database table.
4. Once Job V2 catches up to the real-time high watermark, you switch your application query routing pointer to the new table and decommission Job V1.

---

4. Architectural Comparison: Lambda vs Kappa

---

5. Summary & Key Takeaways

1. Batch is an Illusion: Real-world events happen continuously; batch processing is an artifact of legacy computational constraints.
2. Embrace the Kappa Paradigm: Build pipelines where stream processors like Apache Flink or Kafka Streams handle live incoming traffic and historical backfills identically.
3. Leverage Tiered Storage: Pair Kafka with Tiered Storage (AWS S3 / GCS) to make long-term historical log retention cost-effective for instant Kappa replayability.