Dory1.0-4B, re-measured: NVIDIA's looped nGPT against its own model card

With enough repeats and the card's own metric, Dory reproduces its model card at every loop count on GPQA Diamond, IFBench and AA-LCR, the three of its benchmarks this project can run. None of the four disagreements from the first round was about the model. The 4-loop shortfall was too few repeats plus a different IFBench metric than the card's. The AA-LCR gap was a single draw of 100 questions. The card's low score for its Qwen3.5-4B baseline comes in good part from a blank line Qwen writes before its answers: graded unstripped, Qwen loses 6 of the 6.6 points, though a faithful replay of NVIDIA's harness leaves 2–4 unexplained. And the card's 20–60% token savings were measured on multi-turn agent tasks, a different kind of task from the single-question reasoning tested here.

What the loop buys is concentrated in the second pass through the core: about 5 points on GPQA, 9 on AA-LCR and 4 on IFBench, and 0.009 bits per byte on held-out text, 95% of what eight passes achieve. It costs 1.4× the compute of one pass. Passes 3 and 4 add nothing these sets can resolve: the 2→4 step is −1.0 ± 1.0 on GPQA, +0.2 ± 0.8 on IFBench and +0.7 ± 2.7 on AA-LCR. On GPQA, one 2-loop pass matches a majority vote over five or six 1-loop samples at about a quarter of the compute.

Dory's lead over other small models is not produced by the loop. It is already there at 1 loop, and it grows on 22 instruction types this project wrote and nobody has published. NVIDIA's own non-looped 4B, run through the same harness, is 15–37 points behind. Post-training is the likeliest source, though nothing here isolates it: the only available control differs in model generation, architecture and reasoning length at once. On plain text Dory is a worse language model than Qwen3.5-4B at every loop count. The loop's largest single step is on long-document questions, which is also the noisiest set here.

Experiment 1: Mon 19 Oct 2026 (week 43), Modal workspace yaroslavvb, $41.64 of a $50 cap. Experiment 2 (the ablations): Mon 19 – Tue 20 Oct 2026, Misha Makarov's Modal workspace michael-76064, $301.94 of a $500 cap, about 63 GPU-hours. Claims: Boris Ginsburg's LinkedIn post of Sat 17 Oct 2026 and the model card it links (revision b366b87f). Code, every number's script, and this page: the private repo yaroslavvb/ngpt.

The four disagreements, settled

Single pass with a 65,536-token output budget, complete repeats only. ± is the sampling error on these same questions (the right error for comparing with another measurement of the same set); a single draw gets the binomial bound. Repeat means also cluster by job (the two low 4-loop GPQA repeats in experiment 1 came from one resumed job), so some between-run variance sits outside these bars. IFBench is NVIDIA's metric: the mean over prompts of the fraction of instructions followed, strictly. Qwen3.5-4B's GPQA comes from a 32K-budget run whose budget did not bind (0.0–0.3% of traces reached it).

1. Four loops reproduce; the old shortfall was two repeats and a metric

At 4 loops, GPQA Diamond reads 73.1 ± 0.7 over 6 repeats against the card's 72.16. Experiment 1's 70.7 was two repeats from one run that had been stopped and resumed on a smaller batch. Its two repeats scored 69.7 and 71.7, against 72.2–75.8 in the six fresh ones. Every loop count is now at or above the card: 69.2 at 1 loop (the card's digits exactly), 74.1 at 2 against 72.41, 73.4 at 3 (the card has no 3-loop column).

IFBench at 4 loops reads 78.8 ± 0.6 over 7 repeats against the card's 79.0. Experiment 1 quoted prompt-level loose accuracy, the IFBench paper's headline, which reads 77.9 ± 0.6 here. The card's number is NVIDIA's own metric, the mean fraction of instructions followed, scored strictly. NeMo Gym's IFBench verifier defines it as mean/reward × 100 with grading_mode: fraction. The all-or-nothing strict rate reads 75.9 ± 0.6 and cannot be the card's metric: it is about 5 standard errors below. On NVIDIA's metric every loop count lands on the card: 74.2 against 74.29, 78.6 against 77.94, 78.8 against 79.

"4 loops is no better than 2" holds, as a statement about power. Paired on the same questions, GPQA's 2→4 step is −1.0 ± 1.0, and the card's own GPQA columns also step down (−0.25). IFBench gains at the second loop and then flattens, 74.2 → 78.6 → 78.2 → 78.8: the 1→2 step is +4.3 ± 1.1 (card +3.65) and 2→4 is +0.2 ± 0.8 (card +1.06). AA-LCR's 2→4 step is +0.7 ± 2.7. Our 2→4 steps sit within about 1 SE of the card's own 2→4 steps on all three benchmarks, so this is not a disagreement with the card either: a 4-loop gain under about 2 points cannot be resolved on these sets.

The card's cheaper last-loop KV cache keeps one cache per recurrent layer instead of one per loop. Each loop then attends to earlier tokens' final-loop states. It is not what separates us from the card. At 2 loops it matches per-loop caching (GPQA −1.0 ± 1.2, IFBench −0.5 ± 1.2). At 4 loops it reads +0.3 ± 0.9 on GPQA and +0.8 ± 0.9 on IFBench, both null. Its value is cost: at 4 loops it serves 1.34× the tokens per second on an H200, because the smaller cache fits twice the batch.

On NVIDIA's own GPQA prompt (the Artificial Analysis multiple-choice template with one fixed answer order per question), 2 and 4 loops match the card (73.2, 72.5) and 1 loop reads 66.4, 2.8 below it. This rests on 2 repeats with the answer position held fixed, so it bundles the prompt wording with position bias. The two answer extractors, simple-evals' regex and NVIDIA's lenient one, score all 8,514 Dory GPQA records identically; they pick a different letter on one, which both mark wrong.

2. The Qwen3.5-4B baseline: a blank line costs it six points

Qwen3.5-4B closes its reasoning with </think> followed by a blank line before the answer. It does this on 99.4–99.8% of the replies where it finishes; Dory starts its answer immediately on 99.3–99.5% of them. Graded on the text after </think> with the whitespace stripped, which is this project's convention, Qwen reads 60.4 on NVIDIA's metric. That sits on Qwen's own card (59.2). IFBench's strict scorer grades the response as given, and NeMo Gym passes it the reply as the server returns it. Graded that way, Qwen reads 54.4 under NeMo Gym's sampling, against the card's 53.83. The difference is all leading whitespace: stripping only the left side reproduces the stripped score, and stripping only the right side reproduces the raw one. The loss falls on whitespace- and layout-sensitive checkers (format:no_bullets_bullets, line_indent, options, no_whitespace, custom:csv_*). 37–42 instructions flip from pass to fail and none flip back. Dory's 3,731 rollouts flip none. NVIDIA's numbers show the same pattern across the whole Qwen family: the card reads Qwen3.5-4B 5.4 points below Qwen's own report on IFBench and Qwen3.5-9B 5.5 below. The same card reads the same model 1.5 points above Qwen's GPQA and 7.7 above its AA-LCR.

It is not a complete reconciliation. NeMo Gym grades the last output item. When a reply never closes its reasoning, which happens on 21% of Qwen's IFBench replies under NVIDIA's sampling, that item is the reasoning itself rather than an empty answer. A replay of that rule reads 55.8–58.1, which leaves 2–4 points unexplained. NVIDIA also sets no output cap, and more budget would raise Qwen further. The reading also assumes a reasoning parser that splits on </think> alone; vLLM's qwen3 parser would have graded whole replies, giving about 29. The unstripped reading is the closest of 52 hypotheses checked, so its near-exact match is partly selection. What does not depend on selection is the mechanism and its size: about 6 points for Qwen and none for Dory. The other hypotheses ruled out include NVIDIA's sampling, output caps, metric definitions, reasoning switched off, reasoning graded with the answer, and IFBench and checkpoint revisions.

An output cap of about 12.5K tokens is the other way to put Qwen on 53.83, which is why we tested it. The same cap puts Dory at 60–65 against its own card's 74–78, so NVIDIA's harness did not cut answers short.

So the card overstates Dory's IFBench lead over Qwen3.5-4B. Under our convention the gap is 13.8 ± 2.3 at 1 loop and 18.2 ± 2.3 at 2 loops, paired over the same 300 prompts. The card implies 20.5 and 24.1.

3. AA-LCR: one draw of 100 questions is not a measurement

Three independent single-pass 1-loop draws of the same 100 long-document questions read 57, 68 and 63, mean 62.7 against the card's 63 (the draw-to-draw error is ±3.2); they disagree on 32 of the 100 questions. Experiment 1's 68 was a fourth draw, cut at 32K and continued. The first analysis read the 11-point gap between it and the 57 draw as an effect of the budget convention (McNemar p = 0.04). Against the two later draws the same comparison gives p = 1.00 and 0.41: it was noise. At 2 loops: 69, 72 and 73, mean 71.3 against 71. At 4 loops: 72 and 72, mean 72.0 against 73. The second loop is worth +8.7 ± 3.1 points paired on the same questions, against the card's +8; the third and fourth add nothing measurable (+0.7 ± 2.7).

The judge is not the variable. Re-running gpt-oss-120b on the same answers changed 0–2 verdicts in 100, and so did running it at temperature 1 or with the AA-LCR card's older v1.0 prompt: every pass lands within 2 points. NVIDIA's own judge, Qwen3-235B-A22B-Instruct-2507-FP8 under NeMo Gym's strict one-word protocol, re-graded 12 complete answer sets: it reads −1 to +2 points against gpt-oss-120b (mean +0.4), and puts the single-pass draws at 63.7 at 1 loop, 70.3 at 2 loops, 72.0 at 4 loops (card 63, 71, 73). Qwen3.5-4B reads 64.0 under gpt-oss-120b and 65.0 under NVIDIA's judge (card 64.7); Nemotron-3-Nano-4B reads 18.0 and 19.0: it answers 100K-token documents in a median 770 output tokens.

A note on method, because it cost money. vLLM was seeded once per engine (seed 0) and requests carried no seed of their own. So a "repeat" launched as a separate job, with the same prompts in the same order at the same batch size, reproduced the first run token for token. Two AA-LCR "repeats" were byte-identical copies (100/100 answers), and three more were copies in progress. Every run is now seeded from its tag, every pool is checked for identical text before it is used, and the copies are excluded. GPQA repeats were never affected (each repeat reorders the answer choices, so the prompts differ), and neither were the IFBench repeats, which were checked.

4. Shorter answers, but not 20–60% shorter

Total output tokens at k loops divided by total at 1 loop, paired over the same questions, single pass at 64K. The card's chart is four multi-turn agent benchmarks at 2 and 4 loops.

Looping shortens Dory's answers by 9–18%. The whole saving is in the reasoning; the final answer is the same length at every loop count. The saving keeps growing past 2 loops even though accuracy does not: on GPQA, 4 loops write 369 ± 158 fewer tokens than 2 for no change in score. On GPQA, looping also shortens what Dory gets right more than what it gets wrong: controlling for the length of the 1-loop answer, questions 1 loop never gets right shrink 10% less at 4 loops than questions it does (on IFBench and AA-LCR the same effect is within noise).

The card's 0.40–0.80 ratios are Terminal-Bench, GDPval, SWE-bench and τ³-Banking. Those are multi-turn agent tasks, where a shorter or better first step can remove whole tool-call turns. None of that exists in a single-question benchmark, and this project did not run an agent harness. Single-question traces don't bridge the gap either. Across questions, the ratio falls only weakly with how long the 1-loop answer was. After correcting for the fact that the same noisy length appears in both the ratio and the predictor, the trend is about 4% per doubling, with a confidence interval that includes zero. So the card's agent numbers stay NVIDIA's measurement, unconfirmed here.

What a loop is worth

Dory has 36 blocks, each one attention layer plus one MLP (72 layers in the usual count): 8 input blocks, a 20-block core and 8 output blocks. The core runs k times per token, with the input blocks' output re-injected on every pass. 9 of the 36 attention layers use full attention (the rest a 1,024-token sliding window), and 5 of those 9 sit inside the core, so 2 loops apply 14/9 = 1.56× the full attention of one. The loop count is set at inference time; the checkpoint ships with 2.

Per thinking token

Budget forcing cuts each reasoning trace at B tokens and asks for the answer. At every budget from 1K to 16K, 2 loops beat 1 by 3–6 points on GPQA and 2–6 on IFBench. On GPQA, 2 loops at 8K come within 2 points of 1 loop at 16K, and 2 loops reach 1 loop's full-budget accuracy with 0.74× its tokens. 4 loops track 2 loops within noise at every budget.

Against more samples

This panel pools experiment 1's 32K-continued repeats with the 64K ones, because a five- or six-sample vote needs more than four repeats: 8 repeats at 1 and 2 loops (four continued), 8 at 4 loops (two continued), 4 at 3 loops (64K only). That is why 2 loops reads 73.2 here against 74.1 in the score table, and 4 loops 72.5 against 73.1. Compute is matrix-multiply and attention FLOPs at each arm's measured prompt and output length, relative to one 1-loop sample. The pooled 1-loop arm's prompts include experiment 1's re-fed traces, which is why a 2-loop pass reads 1.37× here and 1.39× in "What a loop costs" (3 loops 1.68× against 1.79×).

Spending the compute on more samples instead of more loops, with majority vote: one 2-loop pass (73.2% at 1.37× the compute of a 1-loop sample) matches a five-sample 1-loop vote (73.1% at 5×) and a six-sample one (73.3% at 6×, 4.4× its cost). Above about twice the 1-loop cost, more samples at 2 loops own the frontier. The only deeper point on it is one 3-loop pass (73.4% at 1.68×); no 4-loop point is ever on it.

The gain also has a shape. Bin questions by how often 1 loop solves them, using held-out repeats. The second loop adds 4–9 points of solve rate to questions 1 loop sometimes gets and 0.7 to questions it never gets: it sharpens, it does not unlock. It also does not re-rank difficulty. Corrected for measurement noise, per-question solve rates correlate 0.99–1.01 across every pair of loop counts (values above 1 are an artifact of the correction and mean the orderings are indistinguishable). So there is nothing to route on: a per-question choice of loop count picked on held-out repeats scores below simply using the best single loop count.

Inside the model

Next-token loss (nats per token) of reasoning traces from 60 GPQA questions, written by the model in the row and scored teacher-forced at each loop count; the minimum of each row is in bold.

Each loop count is a different distribution, not a sharper copy of one. Text written at 1 loop is predicted best at 1 loop, text written at 2 loops at 2, and so on. The step from 1 to 2 loops moves the distribution about 6× as far as the next two loops together (Jeffreys divergence 0.088 nats per token for 1→2 against 0.015 for 2→4). On reasoning traces nobody in the Dory family wrote (Qwen3.5-4B's), the loss is lowest at exactly 2 loops and every further loop gives a little back.

One 16-loop pass, read out after each loop, shows why. The first loop rewrites the state almost completely (its cosine with the input is −0.02). The second moves it by 2.6% of its length, 1.8% as far as the first. After that the steps keep shrinking, but ever more slowly, settling near 0.3% per pass by loop 10. The loss gains shrink faster, by about 0.27 per loop, toward a limit of 0.656, which is worse than the 0.647 at 2 loops.

Bits per byte, teacher-forced, on 433 documents (1.5 MB) in five pools, including 2026 arXiv papers newer than either model; the best loop count per row is in bold. Tokenizer-free, so Qwen3.5-4B's column is directly comparable.

In aggregate, ordinary text shows a fixed point rather than an optimum. The second loop takes 0.0093 bits per byte off, which is 95% of the whole move from 1 to 8 loops. Loops 3–8 add up to another 0.0005, and the last two steps are inside their own error. The pools disagree past 2 loops: arXiv keeps improving to 8 loops, while human-written reasoning is best at 2 and slightly worse at every depth beyond. Qwen3.5-4B is the better language model on every pool. Dory needs 0.059 more bits per byte at 1 loop, and looping closes only a sixth of that gap.

What a loop costs

A 4-loop token does not cost 4× a 1-loop token: only the 20-block core repeats, and the 131,072-row output layer runs once. In matrix-multiply and attention FLOPs, an answer costs 1.00× / 1.39× / 1.79× / 2.24× at 1–4 loops, counting each loop count's shorter answers. Measured wall-clock time runs higher: at batch 256, a 4-loop token takes 2.4–3.1× the time of a 1-loop token depending on the GPU. Decoding at that batch is limited by memory traffic, and per-loop KV caching multiplies the cache that has to be read.

Same benchmark on each GPU: 198 GPQA prompts × 4,096 output tokens, 256 at a time, vLLM from the Dory PR. Dollars at Modal's list prices; a cold start (77–164 s on these runs) is extra.

On cost per token, the B200 is the best card for this model, but only if the image has a CUDA compiler. On Blackwell, this vLLM picks FlashInfer's TRT-LLM attention kernels, which compile on first use; on a slim image with no nvcc the engine dies. Forcing FlashAttention 2 instead runs at 3,849 tokens/s, which costs more per token than an A100, and Triton attention reaches 13,158. With the kernels compiled, a B200 does 17,717 tokens/s at 1 loop, 1.6× an H200, and is 15–23% cheaper per token. The A100 costs 2.5× the H200 per token: it has no FlashAttention 3.

Where Dory's advantage over other small models comes from

Same harness, prompts and grading for every model, at a 64K budget, with three exceptions. Qwen3.5-4B's GPQA is the 32K run, whose budget did not bind. gpt-oss-20b's GPQA uses NVIDIA's lenient extractor, because it bolds its answer label and simple-evals' regex misses it (54.3 with that regex); 15% of its GPQA traces also hit the budget at reasoning effort high, so treat that cell as a lower bound. IFBench and IFNovel use NVIDIA's metric, on the text after `` (after `assistantfinal` for gpt-oss-20b, whose output is Harmony). IFNovel is 300 prompts built for this project from 22 constraint types that exist in no public set, graded by this project's verifiers.

• The lead is there before the loop. 1 loop already has 74.2 of 4 loops' 78.8 on IFBench and 87.3 of 89.5 on IFNovel. Looping adds 2–5 points.
• It is not benchmark coverage, as far as IFNovel can tell. On IFNovel's unpublished constraint types the margin grows against all three baselines in 9 of 9 comparisons: over gpt-oss-20b from +5.7 to +9.9, over Nemotron-3-Nano-4B from +14.8 to +37.4, over Qwen3.5-4B from +13.8 to +39.9 (IFBench to IFNovel, at 1 loop). In absolute terms, Dory and gpt-oss-20b score higher on IFNovel than on IFBench (+13 and +9), while Qwen and Nano score lower (−13 and −10). The two that improve are the two gpt-oss-family models, so a match between IFNovel's verifiers and that family's style cannot be ruled out; a third-party held-out set would settle it. Dory at 4B beats gpt-oss-20b on both sets.
• NVIDIA's own recipe family is not enough. Nemotron-3-Nano-4B is NVIDIA's non-looped hybrid-Mamba 4B, run with the same sampling and harness. It finishes 99% of its traces and still trails Dory at 1 loop by 14.8 on IFBench and 37.4 on IFNovel. But Nano reasons 6–11× less on these two sets (counting reasoning tokens; 2× less on GPQA) and is a different model generation (Nemotron 3, where Dory's card points at Nemotron 3.5), so this control separates Dory from NVIDIA's recipes in general, not post-training from reasoning effort.
• Against Qwen, most of the gap is Qwen not finishing. 13% of Qwen's IFBench traces hit the budget under its own sampling and 22% under NVIDIA's, mostly in repetition loops; on IFNovel, 38%. On prompts where both models finished, 50–90% of the IFBench gap and 62–66% of the IFNovel gap disappear. Against the two baselines that finish their traces, none of it does.
• Probably not memorized test items, but the probes disagree. On GPQA, Dory shows no excess familiarity with the 198 Diamond questions relative to the 250 GPQA-main questions outside Diamond (bounded below 0.013 nats per byte; one of four statistics is inconclusive). Both sets are public, so this bounds benchmark-specific scraping, not exposure. On IFBench, relative to Qwen, Dory finds the test prompts easier than matched control prompts (+0.064 nats per byte); one of four statistics (second-half loss) crosses the leakage threshold, and a second (Min-K%) is positive but short of it. But the effect stays at full size after IFBench's constraint sentences are removed and only the prose is scored (101% of it), so most of it tracks which text pool the prompts came from. The part that would be item memory, the constraint assembly within a prompt, is 13× smaller and not significant at 1 loop, but 7.5× smaller and significant at 2 loops (z = 2.2). A small item-level component cannot be ruled out at the shipped loop count. Dory's absolute loss on the test items is higher than Qwen's.
• It reads like gpt-oss, and that predicts nothing. Dory's reasoning uses gpt-oss's analysis-channel vocabulary ("we need", "the user wants") at every loop count. Nemotron-3-Nano-4B is closer still to gpt-oss on that measure and is 27 points below gpt-oss-20b, and 37 below Dory, on IFNovel.
• Long context is where the loop's step is largest. With 5 of the 9 full-attention layers in the core, 2 loops apply 1.56× the global attention. On AA-LCR the second loop is worth +8.7 ± 3.1 points, the largest step on any benchmark here, though AA-LCR is also the noisiest set. Dory at 2 loops leads Qwen3.5-4B by 7.3 ± 4.9 points, not separable from noise. "More compute per token" predicts the same ordering, so the attention layout cannot be credited specifically.
• GPQA is not an advantage. Qwen3.5-4B leads Dory by 4.7–9.6 points at every loop count.

The verdict: Dory's edge over small models of its size is instruction following, and it is not produced by the loop. It is whole at 1 loop and holds up on constraint types nobody has published. NVIDIA's own non-looped 4B lacks it, and on ordinary text Dory is the weaker language model. That points at post-training, but nothing here isolates post-training: the only control differs in generation, architecture and reasoning length at once. Looping adds a consistent, modest second-pass gain on top, largest on long documents. Its practical value is "elastic compute": one checkpoint, with 2 loops buying everything looping offers that these sets can resolve.

A second implementation

Everything above runs through NVIDIA's own vLLM PR, so it measures the model and that code together. To separate them, Dory's forward pass was written a second time from scratch in pure PyTorch, from the vLLM PR and the llama.cpp fork brianchmielNV/llama.cpp at 98a74c97 read side by side (ref_dory.py). Scoring the same 70,192 teacher-forced tokens (24 documents written by Dory at 1, 2 and 4 loops and by Qwen3.5-4B), the two agree within 0.0002 nats of mean loss at every loop count (0.6649 / 0.6474 / 0.6531 against 0.6650 / 0.6474 / 0.6531). The signed per-token bias straddles zero, and the argmax matches 99.1–99.2% of the time. The remaining per-token difference, 0.015 nats, is the size of any bf16 reordering of the same mathematics; an fp32 RoPE cache alone moves it by 0.015. The smallest semantic error constructed was 35× larger: dropping the loop's re-injection moves per-token log-probabilities by 0.52 nats on average, and raises mean loss by 0.36 nats. Skipping the core entirely, which the independent implementation allows and vLLM does not, gives a loss of 12.57, worse than a uniform guess over the vocabulary (11.78): at 1 loop, the core is 40 of the 72 layers.

Method

• Models. nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-BF16 at b366b87f, BF16, loop count and KV mode set with hf_overrides (per-loop caching, the card's example, unless stated). Baselines: Qwen3.5-4B at 851bf6e8 (Qwen's thinking sampling, and NeMo Gym's), Nemotron-3-Nano-4B-BF16 at dfaf35de (reasoning on, fp32 Mamba state), gpt-oss-20b at 6cee5e81 (reasoning effort high). Every run checked that vLLM loaded the requested loop count.
• Engine. vLLM from the unmerged Dory PR vllm-project/vllm#60149 at 55076950 on the precompiled CUDA 13 wheel of 153a1716, torch 2.14.0, FlashAttention 3 on Hopper. Modal H200s for everything except the GPU comparison and the reference-implementation scoring (H100s). The fast B200 numbers and NVIDIA's 235B judge ran on NVIDIA's CUDA 13 devel image, which has the compiler their kernels need.
• Budget. Experiment 2 is one pass with a 65,536-token output budget. Experiment 1 cut at 32,768 and continued cut traces to 64K; those runs are kept apart and quoted only for history. Sampling is the card's (temperature 1.0, top-p 0.95) with thinking on; a reply that never closes its reasoning scores zero.
• GPQA Diamond. 198 questions with simple-evals' prompt, a fresh answer order per repeat, and its Answer: X regex; NVIDIA's Artificial Analysis prompt as a second arm. 4–6 repeats per loop count at 64K.
• IFBench. The 300 prompts of allenai/IFBench at 1c40f0c, AllenAI's verifiers on the stripped text after </think>, scored with NVIDIA's metric; 2–7 repeats. IFNovel: 300 prompts over 22 unpublished constraint types, graded by this project's verifiers, 2 repeats.
• AA-LCR. The 100 questions of v1.1 with documents in the dataset card's template (108K tokens on average), judged by gpt-oss-120b with the v1.1 equality-checker prompt at temperature 0; 2–3 independent draws per loop count.
• Statistics. Per-question means over complete repeats. ± is the sampling error (within-question variance over repeats). Comparisons are paired over questions (sd(d)/√n), never two independent binomials. Every pooled set of runs is checked for byte-identical text. Six analysis scripts produced the numbers. Each was re-derived by an independent verifier with its own code, a completeness pass reconciled them, and a final fact-check compared this page with the data (analysis/*.py, analysis/verify_*.py, analysis/headline.py).

What differs from NVIDIA's setup

• Harness. NVIDIA used NeMo Gym and the NeMo Evaluator SDK. This project used simple-evals' GPQA template, AllenAI's IFBench verifiers and the AA-LCR card's prompts. NVIDIA's own GPQA template was run as a check.
• Repeats. The card does not say how many it used for GPQA and IFBench. Its footnote says 16 for AA-LCR; this project has 2–3 draws per loop count.
• AA-LCR judge. NVIDIA used Qwen3-235B-A22B-Instruct-2507-FP8 under NeMo Gym's strict one-word protocol. This project used gpt-oss-120b, and re-graded with NVIDIA's judge as a check (section 3).
• The agent benchmarks (Terminal-Bench, SWE-bench, GDPval, τ³-Banking), SciCode, HLE and AA-Omniscience were not run. Neither was the card's internal "AA v4.3.2" index, which needs Artificial Analysis's private sets.

Spend

Experiment 1 cost $41.64 of its $50 cap on the yaroslavvb workspace; nothing more was spent there. Experiment 2 ran on Misha Makarov's workspace michael-76064, 63 GPU-hours, almost all on H200s at about $4.8 an hour all-in:

Per-app lines are in LEDGER.md; ledger.py --profile misha recomputes them from Modal's billing report.

What would settle the rest

1. The agent-task token savings. Terminal-Bench 2.1 or SWE-bench Verified at 1, 2 and 4 loops would show whether a loop saves whole turns. That needs a new agent harness and sandboxes, well past this budget.
2. The last 2–4 points of Qwen's IFBench. Serving Qwen3.5-4B through NeMo Gym's Responses API, uncapped, for 8 repeats would settle it (about 4 H200-hours at a 64K cap, more uncapped).
3. An equal-effort recipe control. Nemotron-3-Nano-4B reasons 6–11× less than Dory on the instruction sets. The nearest public checkpoint from the right data generation, Nemotron-3.5-Lightning-30B-A3B, is non-looped but 30B (3B active) rather than 4B, and its trace lengths are unmeasured here.
4. More AA-LCR draws. Draw-to-draw spread is about 5.5 points: 3–4 draws per loop count pin each level to about ±3, and about 7 each would pin the loop gain to ±3.