✨ NeurIPS 2026 Spotlight ✨
We measure the capacity of parameters of looped LMs with a scaling law exponent \(\varphi\) and use \(\Delta\varphi\) as a diagnostic: truncated BPTT lowers it (worse loop), hyperconnections raise it (real gain), even though both improve validation loss. Model designers should make each loop worth its compute.
We measure how much one recurrence is worth to a looped transformer, in equivalent unique parameters. From an iso-depth pretraining sweep across recurrence counts \(r \in \{1, 2, 4, 8\}\) spanning \({\sim}50\times\) in training compute, we fit a joint scaling law
\( L = E + A\,(N_\text{once} + r^{\varphi} N_\text{rec})^{-\alpha} + B\,D^{-\beta} \)
and measure a recurrence-equivalence exponent \(\varphi = 0.46\). Intuitively, \(\varphi\) tells us whether looping a block \(r\) times is equivalent in validation loss to \(r\) unique blocks of a non-looped model (full equivalence, \(\varphi{=}1\)) or to a single block run repeatedly with no capacity gain (\(\varphi{=}0\)). Our \(\varphi = 0.46\) sits in between, so replacing unique blocks with shared recurrences increases validation loss at matched training compute. For example, at \(r{=}4\) a 410M looped model performs on par with a 580M non-looped model, but incurs the training cost of a 1B non-looped one. We demonstrate the utility of \(\varphi\) as a diagnostic tool on two case studies: commonly used truncated backpropagation lowers \(\varphi\) to \(0.38\), indicating that the loop mechanism is poorly trained under truncation, even though validation loss decreases. Conversely, hyperconnections raise \(\varphi\) to \(0.65\), a genuine capacity gain. Our method separates true loop improvements from training-side gains, a distinction raw validation loss cannot make.
A looped transformer reuses the same block of layers several times before predicting the next token. Each pass, or recurrence, updates the model’s internal representation using the same parameters.
The prelude processes the input, the recurrent core runs \(r\) times, and the coda predicts the next token. This lets a small model do more computation without adding new layers.
We ask: How much of the capacity of a larger model can loop iterations recover?
To measure the value of looping, we compare models that use roughly the same amount of compute per token. Every model executes 20 layers: the non-looped model uses 20 distinct layers, while the looped models reuse a smaller core.
For example, four passes through a four-layer core give 16 layer computations. Two prelude and two coda layers bring the total to 20. More loops mean fewer unique parameters at roughly the same compute.
We call this an iso-depth comparison.
We use a scaling study to measure how much capacity shared parameters gain through looping. We train 116 models across six compute budgets, varying model width and the number of loops. Fitting a joint scaling law lets us separate the contributions of model size, training data, and recurrence.
The key idea is an effective parameter count: we give shared parameters extra weight to account for the capacity gained by using them repeatedly.
\( N_\text{eff} = N_\text{once} + r^{\varphi} N_\text{rec} \)
\(N_\text{once}\) counts parameters used once, and \(N_\text{rec}\) counts the shared parameters. Looping the shared block \(r\) times multiplies its contribution by \(r^\varphi\).
\(\varphi = 0\): no extra capacity Looping adds computation without improving loss. The loss matches that of a parameter-matched non-looped model, which requires less compute.
\(\varphi = 1\): full equivalence Looping contributes as much as \(r\) distinct blocks. The loss matches that of a compute-matched non-looped model at the same training compute.
\(\varphi < 1\): extra loops reduce performance At matched training compute, using extra loops of the shared block in place of distinct layers raises loss.
\(\varphi > 1\): extra loops improve performance At matched training compute, using extra loops of the shared block in place of distinct layers lowers loss.
The full law combines this effective size with the number of training tokens \(D\):
\( L = E + A\,(N_\text{once} + r^{\varphi} N_\text{rec})^{-\alpha} + B\,D^{-\beta} \)
We make the following main findings:
We measure \(\varphi = 0.46\): looping increases the capacity of shared parameters, but each extra loop adds less capacity than a distinct block. Four loops provide about 1.9 blocks’ worth of capacity. At matched training compute, replacing distinct layers with extra loops therefore saves parameters but reduces performance.
No, it can change with architecture and training recipe. We propose \(\Delta\varphi\) as a diagnostic for loop capacity. Truncated BPTT saves about 30% of training compute per token but produces weaker loops (\(\varphi = 0.38\)). Hyperconnections improve information flow and produce stronger loops (\(\varphi = 0.65\)). Both methods lower loss.
We fit \(\varphi\) to downstream task performance to obtain a task-specific \(\varphi_\text{axis}\). Knowledge (\(0.58\)) and reading (\(0.71\)) yield larger values than validation loss (\(0.46\)). Math and reasoning show no consistent gains beyond variation across training runs at our budgets. Future work should measure these tasks at larger compute budgets.
Make each loop worth its compute.
What would happen if each loop added more capacity? Increase \(\varphi\) to bring the looped models closer to the non-looped baseline. Change the training budget to see how the best model size shifts.
At a fixed compute budget, a larger model uses more per-token compute and can thus train for fewer tokens. The curves show this trade-off: the lowest point is the best balance between model size and data.
Each curve uses the same training budget. Diamonds mark the model sizes with the lowest predicted loss.
Lines show the best loss predicted by the law. Crosses show optima estimated from our measured results and stay fixed when the sliders change.
@inproceedings{schwethelm2026isodepthscaling,
title={How Much Is One Recurrence Worth? Iso-Depth Scaling Laws for Looped Language Models},
author={Kristian Schwethelm and Daniel Rueckert and Georgios Kaissis},
booktitle={The Fortieth Annual Conference on Neural Information Processing Systems},
year={2026},
url={https://openreview.net/forum?id=iBvm5lCnAJ}
}