Preprint 2026
We introduce the first efficient end-to-end inference method for depth-adaptive looped language models, which are taking less compute for "easy" tokens and more for "hard" tokens, translating per-token compute savings into practical inference speedups through continuous depth batching.
A main promise of looped language models is depth-adaptive inference. By looping a block of shared layers a variable number of times, the model can use less compute for “easy” tokens and more for “hard” ones. However, tokens with different numbers of loops cannot share a uniform forward pass and therefore cannot be handled by standard batching systems such as vLLM. The practical value of depth-adaptive inference thus hinges on whether batching can be made efficient.
We introduce the first efficient method for depth-adaptive looped LMs via continuous depth batching (CDB), which forms new batches between loop steps. Our method dynamically schedules looped and non-looped parts of the architecture, manages looped KV-caching, and predicts which tokens will exit the loop in advance so it can prepare batches asynchronously.
Experiments on Ouro 1.4B and Huginn 3.5B show that fully looped architectures are best suited to depth-adaptive inference, as large non-looped layers outside the recurrent core (e.g., token embedding, LM head, and unshared transformer blocks) slow down and complicate scheduling. Overall, CDB realizes up to 99% of the estimated maximum speedup available, leaving further gains primarily dependent on model architecture and exit behavior.
Looped LMs repeatedly execute a block of shared layers, called the recurrent core, to scale effective computation with loop steps without increasing parameter count. Outside the recurrent core, unshared layers in the prelude and coda process each token before it enters the loop and predicts the next token after exiting the loop, respectively.
Depth-adaptive decoding allows tokens to exit the recurrent core after different depths (different number of loop steps). The model can use fewer loops and less compute for “easy” tokens, and more loops for “hard” ones, improving the compute–accuracy trade-off. But can we serve these dynamic forward passes efficiently?
Colors show 1 to 4 loop steps per token. In the math answer, “The” uses 4 steps, “result” uses 3, and “60” uses 2.
$ What is 12 × 5? The result of 12 multiplied by 5 is **60**.
uv run loopsgl serve --model "KristianS7/Ouro-1.4B-Thinking" --exit-threshold 0.1 --shell
In depth-adaptive decoding, tokens within a batch can have different numbers of loops and thus different forward passes: exited tokens need to run the coda, while remaining tokens need another loop step. Standard serving systems like vLLM assume a uniform forward pass and cannot batch these tokens together. Our inference method based on continuous depth batching (CDB) can do it by forming new batches between loop steps within the forward pass and refilling batch slots as tokens exit.
We propose three components to make this efficient:
Exited tokens need to run the coda, while remaining tokens need another loop step. Our scheduler decides which stage to run next and batches its waiting tokens. It interleaves prefill, prelude, core, and coda to refill freed slots and keep batch size high.
Four tokens start in the core. Two exit while two wait for another loop step. Coda and prelude prepare two next tokens, which refill the core batch to four.
Our cache manager handles the missing KV states left by early exits. Last-exited maps skipped depths to the final KV state and returns their slots to the cache pool. Shared caching updates one slot per token and recurrent layer at each loop step.
Tokens exit after 2, 1, and 3 loop steps. Last-exited keeps 6 KV slots and frees 3; Shared uses 3 slots. At skipped depths, later tokens attend to the last KV state computed before exit.
With standard early exit gates, the CPU must wait for the exit decisions after each loop step. It then prepares the next batch while the GPU is idle. Our lookahead gate predicts exits one loop step earlier, so batch preparation overlaps GPU execution.
Both examples run three GPU loop steps. Standard gating leaves gaps while the CPU prepares batches. Lookahead overlaps CPU preparation with GPU work.
To analyze CDB, we derive a prefill-corrected bound that estimates the maximum achievable speedup and a hardware-aware latency model that predicts when refill helps. We then evaluate CDB in offline throughput and online serving benchmarks on two prominent looped models with different architectures: Ouro is fully looped, while Huginn loops only its middle transformer block with non-looped blocks in prelude and coda.
We make the following main findings:
Our CDB implementation solves the batching problem posed by variable-depth forward passes. CDB with refill is near-optimal for fully looped models and can realize up to 99% of the speedup available from early exits.
Our results offer concrete guidance for designing looped models that benefit even more from depth-adaptive inference: cheap prelude and coda, lookahead exit decisions, and memory-efficient KV cache support.
CDB preserves the model’s predictions and exit decisions. However, early-exits and KV cache sharing affect accuracy. Meaningful speedups require models like Ouro that provide enough early exits at the desired accuracy.
@misc{schwethelm2026cdb,
title={Depth-adaptive Inference of Looped Language Models via Continuous Depth Batching},
author={Kristian Schwethelm and Daniel Rueckert and Georgios Kaissis},
year={2026},
eprint={2608.09444},
archivePrefix={arXiv},
primaryClass={cs.LG},
url={https://arxiv.org/abs/2608.09444}
}