/ The Arena

Problem Overview

Static batching causes severe head-of-line blocking in LLM servers. Continuous Batching (Cellular Scheduling) operates at the iteration level: admitting new requests and evicting completed sequences at every single token step.

In this capstone, you will implement an Iteration-Level Continuous Batching Scheduler that handles:

1. Priority waiting queues and prefill slot admission.
2. Step-by-step token generation across heterogeneous requests.
3. Immediate sequence eviction upon reaching max tokens or stop conditions.
4. Backfilling empty slots in the active batch dynamically.