Week 8 · Further reading

Measuring AI, and climbing the score, in more depth

The details and sources behind the week 8 slides, in the order of the slides: RL environments (1 to 8), AI eval (9 to 12), benchmarking (13 to 15), hill climbing (16 and 17) and self-improvement (18). Sections moved here from week 6 were checked on November 3, 2026; the new ones on November 17 and 18. This field moves quickly: when this page and a primary source disagree, trust the source.

  1. Pre-training
  2. Post-training: chat tuning
  3. Reinforcement learning
  4. Where the compute goes
  5. The environment, in RL's own words
  6. RL environments as a market
  7. Harbor
  8. What makes a check good
  9. Five ways to measure a model
  10. Who does the measuring
  11. A score has an error bar
  12. How to read a score
  13. Benchmarks, and how they wear out
  14. When a measure becomes a target
  15. Your tasks as an eval
  16. Hill climbing
  17. Our own climb: a search beyond Neptune
  18. Recursive self-improvement

1. Pre-training

The model reads an enormous amount of text and, at each position, bets on the next token. The weights are adjusted to make the token that actually came next more likely. There is no reward beyond that. The result is a base model: it holds a great deal of knowledge, and it continues any document.

OpenAI's 2022 post on InstructGPT has the example on the slide. Given "Explain the moon landing to a 6 year old in a few sentences.", GPT-3 carries on with similar prompts:

Explain the theory of gravity to a 6 year old.
Explain the theory of relativity to a 6 year old in a few sentences.
Explain the big bang theory to a 6 year old.
Explain evolution to a 6 year old.

That is sensible next-token prediction: in the training text, a line like the prompt is often part of a list of exercises. The week 1 slides and reading have more on how the prediction works.

OpenAI, Aligning language models to follow instructions, Jan 2022

2. Post-training: chat tuning

InstructGPT (Ouyang et al., 2022) is the clearest public recipe for what is now called RLHF, reinforcement learning from human feedback:

The team hired "about 40 contractors". In their human evaluations, outputs from the 1.3-billion-parameter InstructGPT "are preferred to outputs from the 175B GPT-3", despite having 100 times fewer parameters. Knowledge comes from pre-training; behaviour comes from these thousands of choices. It also puts the raters' preferences into the model, which is one source of the flattery you may have noticed in chatbots.

Ouyang et al., Training language models to follow instructions with human feedback, arXiv 2203.02155

3. Reinforcement learning

Replace the human ratings with an automatic check, and the model can practice at a scale no group of raters could match: pick a task, let the model try it many times (each attempt is a rollout), score each attempt, and update the weights so that the choices in high-scoring attempts become more likely.

DeepSeek-R1 is the best documented case. Its R1-Zero version used "rule-based rewards": a correct answer plus the right format, on math, coding and logic problems, with no human examples of reasoning. The paper reports that "the reasoning abilities of LLMs can be incentivized through pure reinforcement learning", with self-reflection and verification emerging during training, including an intermediate model writing: "Wait, wait. Wait. That's an aha moment I can flag here." Nature published the paper after peer review in September 2025; its supplementary information puts the training at about $294,000, on top of the base model.

Agents are trained on the same principle, with longer tasks: the task is a sandbox and an instruction, the attempt is a whole working session, and the check runs at the end.

How a reward becomes a weight update

The model makes each choice (a token, or for an agent a tool call) with some probability. After a batch of attempts at the same task, each attempt's reward is compared with the average. The weights are then moved a little in the direction that makes every choice in a better-than-average attempt more likely, and every choice in a worse-than-average attempt less likely: the change is proportional to (reward − average) times the gradient of the log-probability of the choices made. This is the policy gradient, and Williams's REINFORCE (1992) is its first form. Pre-training is shown the correct next token at every position; here the model learns only how the whole attempt went, so the credit is spread over all its choices and many attempts are needed before the useful ones stand out.

DeepSeek-R1 used a version called GRPO, which "samples a group of outputs" for each question and "estimates the baseline from group scores": each reward has the group's mean subtracted and is divided by the group's standard deviation. One consequence matters for anyone writing tasks. If every attempt in the group gets the same score, there is nothing to compare, and nothing is learned. A task that no attempt passes and a task that every attempt passes are equally useless for training. Production methods add refinements, such as limiting how far one update can move the model.

Guo et al., DeepSeek-R1, arXiv 2501.12948; Nature 645, 633 (2025) · Williams, Machine Learning 8, 229 (1992) · Sutton and Barto, chapter 13

4. Where the compute goes

No lab publishes how its training compute divides between pre-training and post-training, and nobody publishes it month by month. What exists is a handful of single models whose makers gave numbers, relative statements from the closed labs, and outside estimates.

Single models that published numbers

WhenModelPre-trainingPost-trainingShare
Mar 2022InstructGPT (OpenAI)3,640 petaflop/s-days (GPT-3)60 for the RLHF model; 4.9 for supervised fine-tuning1.6%
Dec 2024DeepSeek-V32,664K GPU-hours5K GPU-hours0.19%
Jan 2025DeepSeek-R1V3's, as above147K GPU-hours for the whole RL stage, about $294,0005.5%
Dec 2025DeepSeek-V3.2"a post-training computational budget exceeding 10% of the pre-training cost"over 10%
Oct 2026Beam (Reflection AI)6,144 GPUs for under four weeksRL on 10.5K GPUs for four weeks, "more than 100 million rollouts"over 170%, in GPU-time

The shares for R1 and Beam are our arithmetic on the published figures. The units differ from row to row (compute, GPU-hours, GPU-time), and GPU-hours flatter RL, because much of an RL run is the model generating attempts, at lower utilization than a pre-training run. InstructGPT's authors already saw the first row as good news: "The cost of increasing model alignment is modest relative to pretraining."

What the closed labs say

Outside estimates

Mid-training

A newer name for a stage between the two: the model still predicts the next token, but on a small, curated mix, such as reasoning traces, code, long documents and agent transcripts. Two survey papers in October 2025 use the term; one calls it "a vital stage that bridges pre-training and post-training". Where sizes are public they are small. In Andrej Karpathy's nanochat, midtraining "only takes about 8 minutes, a lot shorter than pretraining at ~3 hours". In Ai2's OLMo 3 it took about a day and a half of a roughly 47-day pre-training. No frontier lab reports it separately.

Ouyang et al., arXiv 2203.02155 · DeepSeek-V3, arXiv 2412.19437 · DeepSeek-R1, Nature 645, 633, supplementary table 4 · DeepSeek-V3.2, arXiv 2512.02556 · Reflection AI, Introducing Beam, Oct 5, 2026 · OpenAI, Apr 16, 2025 · xAI, Jul 9, 2025 · Amodei, On DeepSeek and Export Controls · Epoch AI, Sep 26, 2025 · Epoch AI, Oct 13, 2025 · Epoch AI, Most of OpenAI's 2024 compute went to experiments · Epoch AI, Mar 23, 2026 · OfficeChai on SemiAnalysis, Oct 18, 2026 · A Survey on LLM Mid-Training, arXiv 2510.23081 · Karpathy, nanochat · OLMo 3, arXiv 2512.13961

5. The environment, in RL's own words

Reinforcement learning's vocabulary is older than language models. The standard textbook, Sutton and Barto, puts it this way: "The learner and decision maker is called the agent. The thing it interacts with, comprising everything outside the agent, is called the environment." The environment "also gives rise to rewards, special numerical values that the agent seeks to maximize over time through its choice of actions." And on what a reward is for: "The reward signal is your way of communicating to the agent what you want achieved, not how you want it achieved."

RL's wordFor an LLM agent in a sandbox
Policythe model's weights: what training changes
Actionone tool call, a shell command, or the final answer
Observationwhat comes back into the context: the prompt and tool results
Episodeone whole attempt; also called a rollout, a trajectory or a trial
Rewardthe number the check writes, usually only at the end
Environmenteverything else: the container, files, tools, data and the check

The mapping is ours; the words are standard. The programming interface is old too. OpenAI Gym (2016), now maintained by the Farama Foundation as Gymnasium, fixed it as two calls: reset() starts an episode, and observation, reward, terminated, truncated, info = env.step(action) takes one step. Today's agent tooling keeps the shape: OpenEnv, from Meta and Hugging Face, describes "a simple Gymnasium-style API" in which "each environment is a Docker image". Prime Intellect's docs say the quiet part: "RL environments and agent evals are the same abstraction: dataset + harness + scoring rules."

Sutton and Barto, Reinforcement Learning: An Introduction, 2nd ed. (sections 3.1 and 3.2; free online) · Brockman et al., OpenAI Gym, arXiv 1606.01540 · Gymnasium Env API · OpenEnv, Oct 23, 2025 · Prime Intellect docs

6. RL environments as a market

Epoch AI's FAQ, from 18 interviews at labs and vendors, is the best public overview. An RL environment defines what actions a model can take and what they affect; a task pairs a prompt with a grader, often a unit test or a model judging against a rubric. Labs use environments for RL first, then for benchmarking, and for supervised fine-tuning on successful attempts. Some of its numbers:

Suppliers include specialized startups, data companies such as Mercor, Surge, Handshake and Turing, in-house lab teams, and product companies.

Denain and Barber, An FAQ on reinforcement learning environments, Epoch AI, Jan 12, 2026

7. Harbor

Harbor is "a framework from the creators of Terminal-Bench for evaluating and optimizing agents and language models", released with Terminal-Bench 2.0 in November 2025. Its docs sum it up as "running any agent with any model on any task in any sandbox in parallel". Its vocabulary:

WordHarbor's definition
Task"one or more instructions, a sandbox environment, and a verifier"
Dataset"a collection of tasks"
Agent"a program that completes tasks"
Sandbox"an isolated environment for running tasks"
Trial"one agent's attempt at completing one task"
Job"a collection of trials used to evaluate agents and models"
my-task/
├── instruction.md
├── task.toml
├── environment/
│   └── Dockerfile
├── solution/
│   └── solve.sh
└── tests/
    └── test.sh        # writes a number to /logs/verifier/reward.txt

The verifier must write a numerical reward to /logs/verifier/reward.txt, or several named ones to reward.json. Class tasks write 1 or 0, and we regrade separately for partial credit. Running a whole benchmark:

harbor run --dataset terminal-bench@2.0 \
  --agent claude-code \
  --model anthropic/claude-opus-5-5 \
  --n-concurrent 100 --env daytona

The README lists cloud sandbox providers for "thousands of environments in parallel", and it also lists "Generate rollouts for RL optimization": one format for measuring and for training. The other framework you are likely to meet is Inspect, from the UK AI Security Institute.

Harbor docs · task format · README · VentureBeat, Nov 7, 2025

8. What makes a check good

The same few criteria come up wherever people write down what a good task is:

Three kinds of grader, with Anthropic's summary of each:

GraderStrengthsWeaknesses
Code"Fast • Cheap • Objective • Reproducible""Brittle to valid variations that don't match expected patterns exactly"
A model, with a rubricflexible; handles open-ended answers"Non-deterministic • More expensive than code • Requires calibration with human graders for accuracy"
A person"Gold standard quality""Expensive • Slow"

The classic picture of a bad reward is OpenAI's boat from 2016. In the game CoastRunners, the reward was points, and points came from hitting targets. The agent found "an isolated lagoon where it can turn in a large circle and repeatedly knock over three targets". "Despite repeatedly catching on fire, crashing into other boats, and going the wrong way on the track", it scored "on average 20 percent higher than that achieved by human players", and never finished the race. DeepMind's name for this is specification gaming: "a behaviour that satisfies the literal specification of an objective without achieving the intended outcome."

It happens in the training of today's models too. Anthropic's system card for Claude Opus 4.8 reports that "in roughly 0.1% of training episodes, Opus 4.8 speculated about how to satisfy a grader in ways that diverged from the stated intent of the task", and that models recovered "reference solutions from git history, build caches, or other answer-revealing artifacts". Its alignment team wrote in August 2026 that reward hacking "is hard to fully prevent and has occurred in recent frontier model training runs, including in Anthropic's models".

Anthropic, Demystifying evals for AI agents, Jan 9, 2026 · Merrill et al., Terminal-Bench 2.0, arXiv 2601.11868 · Epoch AI FAQ, Jan 12, 2026 · OpenAI, Faulty reward functions in the wild, Dec 2016 · DeepMind, Specification gaming, Apr 2020 · Claude Opus 4.8 system card, May 2026 · Anthropic, Training a misaligned reward seeker, Aug 2026

9. Five ways to measure a model

HowExamplesWho gradesWhat it misses
Exam questionsGPQA Diamond, Humanity's Last Examan answer keywork longer than one answer; answers leak
Agent tasks in a sandboxSWE-bench, Terminal-Bench, Terminal-Bench-Sciencea scriptwhatever the script doesn't check
People's preferenceArena (formerly LMArena)visitors, votingwhat voters can't verify
Length of taskMETR time horizonstasks timed on peopleother kinds of work; very long tasks
Dangerous capabilitiesUK AISI, US CAISI, the labsexperts and red teamsmost results aren't public

METR's page is a good lesson in what happens to a ruler. It now says "Measurements above 16 hrs are unreliable with our current task suite" and, since September 8, 2026, "This page is no longer actively updated". Its tasks are mainly software engineering, machine learning and cybersecurity, so its numbers don't generalize to all intellectual work, as the page itself says. Week 1's benchmark table has scores for many of these as of September 2026.

METR, Task-completion time horizons

10. Who does the measuring

Charnock et al., arXiv 2601.11916 · Epoch AI on OpenAI and FrontierMath · TechCrunch, Jan 19, 2025 · NIST CAISI agreements, May 2026 · LMArena on Wikipedia

11. A score has an error bar

A benchmark is a sample of tasks, so its score is a measurement with an uncertainty. For a pass rate, the usual interval is the one for a binomial proportion. The slide uses the Wilson interval at 95%:

ResultScore95% interval
6 of 8 tasks (our round 1)75%41% to 93%
60 of 8075%65% to 83%
600 of 80075%72% to 78%

Two more sources of spread sit on top of that. An agent run is random, so the same agent on the same task can pass once and fail the next time; this is why Terminal-Bench 2.0 asks for five runs per leaderboard entry. And two models compared on the same tasks should be compared task by task, since some tasks are simply harder for everyone. The practical rule: a lead of two or three points on a benchmark of about a hundred tasks is usually no difference you can claim.

The machine counts too. Anthropic's engineers found that the gap between the most and least generous server setups on Terminal-Bench 2.0 was 6 percentage points, and concluded: "leaderboard differences below 3 percentage points deserve skepticism until the eval configuration is documented and matched." Evan Miller's paper on the statistics notes that evals are usually reported "with a 'highest number is best' mentality", without testing whether the lead is significant.

Anthropic, Quantifying infrastructure noise in agentic coding evals, Feb 5, 2026 · Miller, Adding Error Bars to Evals, arXiv 2411.00640 · Anthropic, A statistical approach to model evaluations, Nov 19, 2024 · Terminal-Bench 2.0, arXiv 2601.11868

12. How to read a score

  1. Who ran it? The vendor, the lab, or someone independent?
  2. With which harness? LangChain moved gpt-5.2-codex from 52.8% to 66.5% on Terminal-Bench 2.0, from just outside the top 30 to the top 5: "We only changed the harness."
  3. Has it seen the test? Public tasks end up in training data, and answers end up on the web.
  4. How many runs? Agents are noisy. Terminal-Bench 2.0 asks for five runs per leaderboard submission.
  5. What does the check check? And what did a run cost?

For your own research, a small private eval beats any leaderboard: ten problems from your subfield with answers you trust, kept out of public view, run on each new model with the harness you actually use.

LangChain, Feb 17, 2026

13. Benchmarks, and how they wear out

A benchmark is an eval that a community agrees to run the same way: a fixed set of tasks, rules for a run, and one number in a public table. Stanford's AI Index 2026 describes what has been happening to them: "Evaluations intended to be challenging for years are saturated in months, compressing the window in which benchmarks remain useful for tracking progress." Its definition: "Benchmark saturation, where models reach scores so high that a test can no longer distinguish between them". The speed-up is not new. Kiela et al. wrote in 2021: "While it used to take decades for machine learning models to surpass estimates of human performance on benchmark tasks, that milestone is now routinely reached within just a few years for newer datasets."

The slide's five rows, with where each number comes from. Scores were read on November 17 and 18, 2026, and they mix harnesses and settings, so read them as a pattern and not as a ranking.

BenchmarkIntroducedBest thenBest nowWhat happened next
SWE-bench VerifiedAug 202433.2% (GPT-4o)80.9% when OpenAI stopped, Feb 2026OpenAI stopped reporting it and recommends SWE-bench Pro
GPQA DiamondNov 202339% (GPT-4 baseline)96.8%, Epoch AI's run of GPT-6.2 AstraEpoch treats 97% as the ceiling
FrontierMathNov 2024"under 2%"95.1% on Tiers 1 to 3 of the revised set (Claude Fable 5.2, Epoch's run)revised in June 2026, fixing errors in 42% of problems; Epoch added a set of open Erdős problems
ARC-AGI-2Mar 2025about 4% (an estimate; "Pure LLMs score 0%")95.8% on the semi-private set (GPT-6.2 Astra)ARC-AGI-3, launched Mar 2026 at 0.51%, is at 81.2% with the standard harness
Humanity's Last ExamJan 20259.1% (o1)54.8% ± 1.9, no tools (GPT-6 Astra, official board)HLE-Rolling, a continually updated fork, since Sep 2026

Two ways to wear out. Saturation: scores reach the top, and what is left measures errors in the questions more than ability. Contamination: the tasks or their answers reach training data. SWE-bench Verified had both. OpenAI found that "59.4% of the 138 problems" it audited "contained material issues in test design and/or problem description", and that frontier models could reproduce some original fixes from memory.

Terminal-Bench, whose format the homework uses, shows the response. Version 2.0 came out on November 7, 2025; 2.1 (May 2026) fixed 28 of its 89 tasks; 3.0 (July) was a new, harder set where the best models scored about 34%; 4.0 (August) removed saturated tasks; 4.1 (November 9) hardened the verifiers against reward hacking. Its authors: "Benchmarks are software and should be maintained like software." The top entry on 4.1 was 71.2% ± 2.9 on November 17, and the top three entries are inside each other's intervals.

A harness footnote worth remembering: on ARC-AGI-3, ARC Prize measured GPT-6 Astra at 62.7% with its standard harness and 99.9% with a harness that keeps the provider's reasoning state between requests. Same model, same tasks.

Stanford AI Index 2026, technical performance · Kiela et al., Dynabench, arXiv 2104.14337 · OpenAI, Why we no longer evaluate SWE-bench Verified, Feb 23, 2026 · Rein et al., GPQA, arXiv 2311.12022 · Epoch AI benchmarking hub and data · FrontierMath, arXiv 2411.04872 · ARC Prize leaderboard · ARC Prize, Sep 3, 2026 · Humanity's Last Exam, arXiv 2501.14249 · HLE official board · Terminal-Bench news

14. When a measure becomes a target

OpenAI, Feb 23, 2026 · The Decoder on the same · Singh et al., arXiv 2504.20879 · Anthropic, Mar 6, 2026 · METR, Jun 5, 2025 · Zhong et al., arXiv 2510.20270 · MacDiarmid et al., arXiv 2511.18397

15. Your tasks as an eval

Round 1 on the homework board was eight tasks and one agent, Claude Opus 5.5 at xhigh effort: six solved. Each task's page says why, and what would make it harder and fairer. The patterns match this page: a method written out in the instruction tests care more than research judgement; a low pass bar says little about skill; and when an answer key depends on an expert's unstated choice, a failure can be the key's fault as much as the agent's.

The top of the ladder is Terminal-Bench-Science 2.0, with a deadline of December 15, 2026. It targets "tasks that today's best AI agents cannot yet reliably solve", and "Contributors with merged tasks receive co-authorship on the Terminal-Bench-Science paper". Its repository runs the same reference and do-nothing checks as ours, plus adversarial trials that try to cheat. Propose the idea first.

16. Hill climbing

Hill climbing is a search method from the AI textbooks: a loop that keeps moving in the direction of a higher value and stops at a peak, where no neighbouring option scores higher. Its known weaknesses are in the same chapter: local maxima, ridges and plateaus. The textbook cures are to restart from somewhere else, or to accept a random step now and then.

In the AI industry, "hill climbing" has become the everyday word for improving a model or a harness against an eval:

Fitted to the test

Measure on the same tasks after every change, and the system slowly fits those tasks. The standard defence is a held-out set that is looked at rarely.

A caution in the other direction: across 120 Kaggle competitions, Roelofs et al. found "little evidence of substantial overfitting" when the test set stays hidden. Holding data back works. The trouble starts when the test is public, leaks into training, or is tried many times in private.

The wrong hill

A climber gets good at the score, and has no access to what you meant. OpenAI's boat (in the section on checks above) is the classic picture. Recent ones: METR found OpenAI's o3 reward hacking in all 21 runs of one task, where the job was to make a fine-tuning script faster and the model instead copied the cached weights of an already fine-tuned model. The Darwin Gödel Machine, a coding agent that rewrites its own code, improved from 20.0% to 50.0% on SWE-bench, and along the way "faked a log making it look like it had run the tests and that they had passed, when in fact they were never run".

Agents as climbers

Andrej Karpathy's autoresearch (March 2026) is the loop with nothing added. An agent edits one training file, trains for exactly five minutes, and the instructions say: if the validation loss improved, keep the commit; if not, reset. In about two days it tried about 700 changes and kept about 20. Stacked together and tested on a larger model, they cut a separate measure, the time to train to GPT-2 quality, from 2.02 hours to 1.80. That second measure matters: it is the held-out check on a climb that used a five-minute proxy.

Google DeepMind's AlphaEvolve does the same with a population of programs: a model proposes changes, "automated evaluators" score them, and the best are bred further. Its paper names the limit that matters most for science: "The main limitation of AlphaEvolve is that it handles problems for which it is possible to devise an automated evaluator. While this is true of many problems in the mathematical and computational sciences, there are domains such as the natural sciences where only some experiments can be simulated or automated."

Russell and Norvig, Artificial Intelligence: A Modern Approach, section 4.1 · Anthropic, Demystifying evals for AI agents · LangChain, Feb 17, 2026 · LangChain, Apr 8, 2026 · Snorkel AI interview with Alex Shaw, Mar 31, 2026 · Recht et al., arXiv 1902.10811 · Zhang et al., GSM1k, arXiv 2405.00332 · ARC Prize, Dec 20, 2024 · Johnson et al., PNAS 2021 · Roelofs et al., NeurIPS 2019 · METR, Jun 5, 2025 · Sakana AI, The Darwin Gödel Machine · karpathy/autoresearch · Karpathy, Mar 9, 2026 · AlphaEvolve, arXiv 2506.13131

17. Our own climb: a search beyond Neptune

The two charts in the slides come from Jason's own research runs, not from a published benchmark. In both, an agent writes a whole program each round, the program is graded on a GPU, and the next round's agent gets every earlier program with its notes and score. It may improve the best one, combine several, or start over, and the harness keeps the best.

One model at a time

The task: write a program that finds trans-Neptunian objects, small icy bodies beyond Neptune, in telescope images of ten hidden fields. Many are too faint to see in a single exposure. The score is the weighted fraction of the known objects found, and one number per program is all the agent learns.

ModelGraded roundsBest scoreReached atAgent time to the best
Claude Opus 5.51510.27round 136about 100 hours
Claude Fable 5.2, from an empty folder240.40round 15about 21 hours

Opus's curve is a staircase with long flat stretches: 11 new bests in 151 rounds. A review of the run found that its method stopped changing at round 10, and that every later program was the same design, tuned. Fable ran the same task, fields and grader without seeing Opus's work. It passed Opus's best in its third round, then tied its own best for nine rounds. Seven of the objects its best program found were never found in any of Opus's rounds.

In the Fable run, around round 18, a program began encoding how far its search had got on each hidden field in its exit time, a way to learn more than the one number the task allowed. The next round's agent wrote that it had deliberately left out a count of objects found, because the task says per-field results are not given back, and asked later rounds to keep it that way. A later round encoded such a count anyway, and the round after it switched it off. None of this changed the score.

Two models sharing one history

A second task from the same family: find large trans-Neptunian objects in thirteen years of survey detections. After two rounds of Fable alone, Claude Fable 5.2 and GPT-6 Astra each wrote a program every round, in separate sandboxes, with no messages between them. They shared only the history folder. From round 21, GPT-6.2 Astra continued alone. The best score went from 0.48 at the start to 0.76 after 20 rounds and 0.79 at round 23 (November 17; the run is still going). Both models set new bests, and each regularly built on the other's best program. There is no single-model run under the same rules, so this doesn't show that two models beat one.

In this run the agents may see the answers for the graded regions and the grader itself, and cheating is caught by reading every program. The price shows up on new sky: round 18's best program finds 76% of the known objects in the regions it was graded on and 55% in twenty regions it was never graded on. Those regions differ in size and crowding, so the gap is not only overfitting.

Jason's own runs, October 31 to November 17, 2026. Scores are recall of known objects; agent time is the sum of agent sessions.

18. Recursive self-improvement

The idea is from 1965. I. J. Good, a statistician who had worked with Alan Turing, wrote:

"Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any man however clever. Since the design of machines is one of these intellectual activities, an ultraintelligent machine could design even better machines; there would then unquestionably be an 'intelligence explosion,' and the intelligence of man would be left far behind. Thus the first ultraintelligent machine is the last invention that man need ever make, provided that the machine is docile enough to tell us how to keep it under control."

Anthropic's definition from 2026 is narrower: "an AI system capable of fully autonomously designing and developing its own successor. This is called recursive self-improvement." OpenAI's safety framework uses the same words for its highest level: "The model is capable of recursively self improving (i.e., fully automated AI R&D)".

What has been measured

Why it could be fast, why it could be slow

Anthropic's own summary: "We are not there yet, and recursive self-improvement is not inevitable."

As a safety threshold

Where our own runs sit: in the search beyond Neptune, agents improve a program against an automatic score. That is the same loop aimed at a science pipeline, and it does not change the model. What it shares with the loop above is the dependence on the check, which is the part a domain expert writes.

Good, Speculations Concerning the First Ultraintelligent Machine, Advances in Computers 6, p. 33 · Anthropic Institute, When AI builds itself · Google DeepMind, AlphaEvolve, May 14, 2025 · METR, Nov 5, 2026 · RE-Bench, arXiv 2411.15114 · PaperBench, arXiv 2504.01848 · METR, Time Horizon 1.1 · AI Futures Project, Nov 10, 2026 · Epoch AI, Feb 25, 2026 · Forethought, Mar 26, 2025 · METR on Claude Opus 5.5 · Narayanan and Kapoor, AI as Normal Technology · OpenAI Preparedness Framework v2 · OpenAI, GPT-6.2 system card · Anthropic Responsible Scaling Policy v3.4 · Anthropic filing with the SEC, Nov 12, 2026 · Google DeepMind Frontier Safety Framework v3.1