Kafka Transactions: Atomic Multi-Topic and Multi-Partition Writes

While the Idempotent Producer prevents duplicates within a single partition, real-world stream processing (such as a Consume-Transform-Produce loop) requires atomic writes spanning multiple topics, multiple partitions, and consumer offset commits.

If a processing node reads a message from orders.v1, updates an aggregation in analytics.v1, publishes an invoice to invoices.v1, and commits its consumer offset to __consumer_offsets, all four actions must succeed together or fail together atomically.

This is achieved using Kafka Transactions and the Two-Phase Commit (2PC) Protocol.

1. The Transaction Coordinator & __transaction_state Topic

Transactions are orchestrated by a dedicated broker role called the Transaction Coordinator:

The Transaction State Machine:

1. transactional.id: A persistent, user-configured identifier that survives client restarts (e.g. order-processor-node-0).
2. Epoch-Based Zombie Fencing: When a producer starts with a transactional.id, the coordinator increments the producer's Epoch Number. If an older "zombie" instance of the container (stalled during a GC pause) wakes up and attempts to write, the broker rejects it with ProducerFencedException.

2. The 2-Phase Commit Transactional Lifecycle

3. Consumer Isolation Levels: read_uncommitted vs read_committed

Consumers configure their visibility into in-flight and aborted transactions using isolation.level:

• Last Stable Offset (LSO): The offset of the first ongoing (uncommitted) transaction. read_committed consumers never advance past the LSO until the in-flight transaction explicitly commits or aborts.

4. Production Code: The Consume-Transform-Produce Loop

A. Java Transactional Processing

5. Landmark Global Arena Capstone #7 Integration

In this module's connected Landmark Arena challenge, global-kafka-transactional-producer-coordinator, you will implement the Transaction Coordinator and 2PC Commit Protocol:

1. State Machine Transitions: Coordinate states [EMPTY -> ONGOING -> PREPARE_COMMIT -> COMPLETE_COMMIT].
2. Atomic Control Marker Appends: Inject COMMIT and ABORT control frames into partition logs.
3. Read-Committed Consumer Filtering: Filter aborted record spans and compute the Last Stable Offset (LSO).

6. Summary Checklist

• Always assign a unique, stable transactional.id per worker node to enable zombie fencing.
• Always configure isolation.level = read_committed on all downstream consumers.
• Combine sendOffsetsToTransaction() with commitTransaction() to eliminate state drift between processing and offset checkpoints.