Windowing Operations and Stateful Stream Joins: Tumbling, Sliding, and Session Windows

In streaming architectures, unbounded streams cannot be processed with naive global SELECT COUNT(*) queries because the stream never ends.

To compute meaningful metrics (such as "transactions per minute" or "fraud risk over the last 15 minutes"), stream processing engines carve unbounded data into discrete temporal chunks called Windows, storing intermediate state inside high-performance embedded key-value engines like RocksDB.

1. The Three Fundamental Window Types

A. Tumbling Windows (Fixed, Disjoint)

• Definition: Defined by a fixed duration (e.g. 5 minutes). Windows are contiguous and do not overlap.
• Invariant: Every event belongs to exactly one tumbling window instance.
• Formula: $\text{WindowStart}(t)=t-(t(\mathrm{mod}D))$ $\text{WindowEnd}(t)=\text{WindowStart}(t)+D$

B. Sliding Windows (Overlapping, Hopping)

• Definition: Defined by Window Size $S$ and Slide Step $L$ (where $L<S$).
• Invariant: If $S=10\text{ min}$ and $L=2\text{ min}$, an incoming event belongs simultaneously to $\frac{S}{L}=5$ active overlapping windows.
• Best Used For: Moving averages, rolling fraud rate detection, and stock trend momentum calculations.

C. Session Windows (Data-Driven Dynamic Duration)

• Definition: Defined by an Inactivity Gap (e.g. 30 minutes). Rather than having fixed start/end times, a session window remains open as long as events for that key continue to arrive within the gap threshold.
• Best Used For: Web user session analytics, gaming telemetry, and shopping cart lifecycle tracking.

2. Stateful Stream-Stream Joins (Interval Joins)

Joining two infinite streams (e.g., OrderPlacedStream and PaymentReceivedStream) requires bounding the join condition within a temporal window called an Interval Join:

The Interval Join Condition:

$t_{\text{order}}-{\Delta }_{\text{lower}}\le t_{\text{payment}}\le t_{\text{order}}+{\Delta }_{\text{upper}}$ Both streams buffer un-matched events in state memory until the counterpart arrives or the temporal window expires.

3. State Storage Architecture: Embedded RocksDB & Changelog Backups

In high-throughput stream processing, keeping millions of active windows in JVM heap memory causes fatal GC pauses and out-of-memory crashes.

Modern engines (Kafka Streams, Flink) store state in an embedded RocksDB instance running off-heap on local NVMe SSDs:

Why This Architecture is Resilient:

1. Zero Garbage Collection Overhead: Billions of state entries reside in off-heap memory and local SSD storage.
2. Instant Crash Recovery: If a worker pod crashes, its replacement pod downloads the state snapshot from S3 or replays the compacted Kafka changelog topic to reconstruct its exact window state.

4. Landmark Global Arena Capstone #6 Integration

In this module's connected Landmark Arena challenge, global-kafka-sliding-window-aggregator, you will implement a stateful Sliding Window Aggregation Engine:

1. Stateful Record Ingestion: Ingest events stamping (timestamp, key, value) into active overlapping window slices.
2. Deterministic Watermark Triggers: Advance the watermark $W(t)$ and compute closed window aggregates (SUM, AVG, COUNT).
3. State Eviction: Prune expired window buckets from memory once watermarks pass the window boundary.

5. Multi-Language Implementation: Stateful Sliding Window

A. Java (Kafka Streams Sliding Window Aggregation)

B. Go (Stateful In-Memory Sliding Window Bucket Engine)

6. Summary & Best Practices

1. Size Sliding Steps Carefully: A 1-hour window sliding every 1 second creates $3600\times$ state multiplication. Keep the slide ratio ($\frac{S}{L}$) reasonable ($5\times -20\times$).
2. Always Use RocksDB for Production State: Avoid in-memory hash maps for stateful joins to eliminate JVM Garbage Collection pauses.
3. Enforce State Retention TTLs: Set clear expiration horizons on stateful interval joins to prevent RocksDB storage growth.