NVIDIA Dory: a 4B model that thinks harder per token, not longer

NVIDIA's Nemotron Labs has released Dory, a looped Transformer also called a latent recurrent Transformer. It's a 3.97 billion parameter model with up to 256k of context, out under an open license that allows commercial use. The release is mostly a research drop, but you can download it and run it on your own hardware.

NVIDIA shipped several formats: BF16, MXFP8, NVFP4 and GGUFs. Pre-training covered 6 trillion tokens. The first 4.5 trillion ran natively in NVFP4, then training switched to 16-bit for 300 billion tokens of context extension, followed by SFT and reinforcement learning with GRPO.

A normal Transformer runs every layer once. Dory splits its layers into three blocks: an input block of 16 layers that runs once, an output block of 16 layers that also runs once, and a recurrent block of 40 layers in the middle that repeats.

That repetition is the whole trick. At one loop you get 72 layers of effective depth, at two loops 112, at four loops 192. The weights in memory never change, so the GPU still holds the same 3.9B parameters. NVIDIA calls this elastic inference compute: you turn up the loops and pay more compute per token, not more memory for weights.

The memory savings aren't total. In per-loop mode, every loop keeps its own KV cache, so more loops means a bigger cache and more memory. At a 32k context window and two loops, that's roughly 2 GB per sequence.

There's a workaround: a last-loop mode where each recurrent attention layer keeps one cache and overwrites the old ones. Going from one to four loops then barely changes the footprint. NVIDIA says accuracy and output length stay close to per-loop mode. If you're serving long contexts to many concurrent users, that's the setting worth trying.

Against Gemma 4 26B, Dory at one loop wins one of ten benchmarks. At two loops it takes four, and at four loops five. The wins cluster around agentic work: instruction following, long contexts, SWE-bench Verified.

The losses are about knowledge. On GPQA and Humanity's Last Exam it does noticeably worse. That tracks: Gemma 4 26B is a mixture-of-experts model with about 4B active parameters per token, so compute is comparable, but it has far more weights storing facts. Dory simply has fewer places to keep them.

There's a practical snag. When you start vLLM, you have to declare how many loops it should run, so comparing loop counts means running separate instances of the same model side by side.

Sam Witteveen · agents and LLM apps suggests defaulting to two loops and routing harder agentic tasks to a four-loop version. He also notes the quality gain from more loops isn't guaranteed and isn't the same as turning up reasoning. For many prompts, output quality looks similar across loop counts, while the four-loop version takes much longer in wall time.

The bigger idea is a different way to scale. Instead of writing longer chains of thought, the model thinks harder per token rather than producing more tokens. And you choose how hard at deployment time.

Witteveen floats the possibility that a larger version, two or three times the size, could compete with mixture-of-experts models of 50B or more. He also mentions rumors that GPT-6 uses looping, though nothing there is confirmed. For now, this is a research release, not a drop-in replacement for production stacks.