Week 8 · Further reading
Measuring AI, and climbing the score, in more depth
The details and sources behind the week 8 slides, in the order of the slides: RL environments (1 to 8), AI eval (9 to 12), benchmarking (13 to 15), hill climbing (16 and 17) and self-improvement (18). Sections moved here from week 6 were checked on November 3, 2026; the new ones on November 17 and 18. This field moves quickly: when this page and a primary source disagree, trust the source.
- Pre-training
- Post-training: chat tuning
- Reinforcement learning
- Where the compute goes
- The environment, in RL's own words
- RL environments as a market
- Harbor
- What makes a check good
- Five ways to measure a model
- Who does the measuring
- A score has an error bar
- How to read a score
- Benchmarks, and how they wear out
- When a measure becomes a target
- Your tasks as an eval
- Hill climbing
- Our own climb: a search beyond Neptune
- Recursive self-improvement
1. Pre-training
The model reads an enormous amount of text and, at each position, bets on the next token. The weights are adjusted to make the token that actually came next more likely. There is no reward beyond that. The result is a base model: it holds a great deal of knowledge, and it continues any document.
OpenAI's 2022 post on InstructGPT has the example on the slide. Given "Explain the moon landing to a 6 year old in a few sentences.", GPT-3 carries on with similar prompts:
Explain the theory of gravity to a 6 year old. Explain the theory of relativity to a 6 year old in a few sentences. Explain the big bang theory to a 6 year old. Explain evolution to a 6 year old.
That is sensible next-token prediction: in the training text, a line like the prompt is often part of a list of exercises. The week 1 slides and reading have more on how the prediction works.
OpenAI, Aligning language models to follow instructions, Jan 2022
2. Post-training: chat tuning
InstructGPT (Ouyang et al., 2022) is the clearest public recipe for what is now called RLHF, reinforcement learning from human feedback:
- Supervised fine-tuning. People write good answers to prompts, and the model is fine-tuned to imitate them (about 13,000 training prompts).
- A reward model. People rank several of the model's answers to a prompt, and a separate network learns to score answers the way they would (about 33,000 prompts).
- Reinforcement learning. PPO fine-tunes the model to produce answers the reward model scores highly (about 31,000 prompts), with a penalty that keeps it from drifting too far.
The team hired "about 40 contractors". In their human evaluations, outputs from the 1.3-billion-parameter InstructGPT "are preferred to outputs from the 175B GPT-3", despite having 100 times fewer parameters. Knowledge comes from pre-training; behaviour comes from these thousands of choices. It also puts the raters' preferences into the model, which is one source of the flattery you may have noticed in chatbots.
Ouyang et al., Training language models to follow instructions with human feedback, arXiv 2203.02155
3. Reinforcement learning
Replace the human ratings with an automatic check, and the model can practice at a scale no group of raters could match: pick a task, let the model try it many times (each attempt is a rollout), score each attempt, and update the weights so that the choices in high-scoring attempts become more likely.
DeepSeek-R1 is the best documented case. Its R1-Zero version used "rule-based rewards": a correct answer plus the right format, on math, coding and logic problems, with no human examples of reasoning. The paper reports that "the reasoning abilities of LLMs can be incentivized through pure reinforcement learning", with self-reflection and verification emerging during training, including an intermediate model writing: "Wait, wait. Wait. That's an aha moment I can flag here." Nature published the paper after peer review in September 2025; its supplementary information puts the training at about $294,000, on top of the base model.
Agents are trained on the same principle, with longer tasks: the task is a sandbox and an instruction, the attempt is a whole working session, and the check runs at the end.
How a reward becomes a weight update
The model makes each choice (a token, or for an agent a tool call) with some probability. After a batch of attempts at the same task, each attempt's reward is compared with the average. The weights are then moved a little in the direction that makes every choice in a better-than-average attempt more likely, and every choice in a worse-than-average attempt less likely: the change is proportional to (reward − average) times the gradient of the log-probability of the choices made. This is the policy gradient, and Williams's REINFORCE (1992) is its first form. Pre-training is shown the correct next token at every position; here the model learns only how the whole attempt went, so the credit is spread over all its choices and many attempts are needed before the useful ones stand out.
DeepSeek-R1 used a version called GRPO, which "samples a group of outputs" for each question and "estimates the baseline from group scores": each reward has the group's mean subtracted and is divided by the group's standard deviation. One consequence matters for anyone writing tasks. If every attempt in the group gets the same score, there is nothing to compare, and nothing is learned. A task that no attempt passes and a task that every attempt passes are equally useless for training. Production methods add refinements, such as limiting how far one update can move the model.
Guo et al., DeepSeek-R1, arXiv 2501.12948; Nature 645, 633 (2025) · Williams, Machine Learning 8, 229 (1992) · Sutton and Barto, chapter 13
4. Where the compute goes
No lab publishes how its training compute divides between pre-training and post-training, and nobody publishes it month by month. What exists is a handful of single models whose makers gave numbers, relative statements from the closed labs, and outside estimates.
Single models that published numbers
| When | Model | Pre-training | Post-training | Share |
|---|---|---|---|---|
| Mar 2022 | InstructGPT (OpenAI) | 3,640 petaflop/s-days (GPT-3) | 60 for the RLHF model; 4.9 for supervised fine-tuning | 1.6% |
| Dec 2024 | DeepSeek-V3 | 2,664K GPU-hours | 5K GPU-hours | 0.19% |
| Jan 2025 | DeepSeek-R1 | V3's, as above | 147K GPU-hours for the whole RL stage, about $294,000 | 5.5% |
| Dec 2025 | DeepSeek-V3.2 | "a post-training computational budget exceeding 10% of the pre-training cost" | over 10% | |
| Oct 2026 | Beam (Reflection AI) | 6,144 GPUs for under four weeks | RL on 10.5K GPUs for four weeks, "more than 100 million rollouts" | over 170%, in GPU-time |
The shares for R1 and Beam are our arithmetic on the published figures. The units differ from row to row (compute, GPU-hours, GPU-time), and GPU-hours flatter RL, because much of an RL run is the model generating attempts, at lower utilization than a pre-training run. InstructGPT's authors already saw the first row as good news: "The cost of increasing model alignment is modest relative to pretraining."
What the closed labs say
- OpenAI, on o3 (April 2025): "By retracing the scaling path—this time in RL—we've pushed an additional order of magnitude in both training compute and inference-time reasoning".
- xAI, on Grok 4 (July 2025): reinforcement learning "at pretraining scale".
- Anthropic's Dario Amodei (January 2025): "the amount being spent on the second, RL stage is small for all players. Spending $1M instead of $0.1M is enough to get huge gains. Companies are now working very quickly to scale up the second stage to hundreds of millions and billions".
Outside estimates
- Epoch AI: "Until recently, most LLMs were trained with 100× more pre-training than post-training compute", and in early 2025 "RL compute was small—maybe 1-10% of pre-training. But this is scaling up fast".
- SemiAnalysis, the bars on the slide: an estimate of all compute at OpenAI and Anthropic together, by quarter. Pre-training, post-training and serving users were 67%, 5% and 29% in the first quarter of 2024; 32%, 36% and 32% in the fourth quarter of 2025; 23%, 42% and 35% in the first quarter of 2026; and a projected 7%, 55% and 38% for the fourth quarter of 2026. We have these second-hand, from a news report, and the method is not public.
- Most compute is experiments. Epoch AI estimates that OpenAI spent about $5 billion on research compute in 2024 and that final training runs of released models were 9.6% of it. For two Chinese labs that published prospectuses, the figures are 22.6% and 12.3%. The rest went to research and trial runs.
Mid-training
A newer name for a stage between the two: the model still predicts the next token, but on a small, curated mix, such as reasoning traces, code, long documents and agent transcripts. Two survey papers in October 2025 use the term; one calls it "a vital stage that bridges pre-training and post-training". Where sizes are public they are small. In Andrej Karpathy's nanochat, midtraining "only takes about 8 minutes, a lot shorter than pretraining at ~3 hours". In Ai2's OLMo 3 it took about a day and a half of a roughly 47-day pre-training. No frontier lab reports it separately.
Ouyang et al., arXiv 2203.02155 · DeepSeek-V3, arXiv 2412.19437 · DeepSeek-R1, Nature 645, 633, supplementary table 4 · DeepSeek-V3.2, arXiv 2512.02556 · Reflection AI, Introducing Beam, Oct 5, 2026 · OpenAI, Apr 16, 2025 · xAI, Jul 9, 2025 · Amodei, On DeepSeek and Export Controls · Epoch AI, Sep 26, 2025 · Epoch AI, Oct 13, 2025 · Epoch AI, Most of OpenAI's 2024 compute went to experiments · Epoch AI, Mar 23, 2026 · OfficeChai on SemiAnalysis, Oct 18, 2026 · A Survey on LLM Mid-Training, arXiv 2510.23081 · Karpathy, nanochat · OLMo 3, arXiv 2512.13961
5. The environment, in RL's own words
Reinforcement learning's vocabulary is older than language models. The standard textbook, Sutton and Barto, puts it this way: "The learner and decision maker is called the agent. The thing it interacts with, comprising everything outside the agent, is called the environment." The environment "also gives rise to rewards, special numerical values that the agent seeks to maximize over time through its choice of actions." And on what a reward is for: "The reward signal is your way of communicating to the agent what you want achieved, not how you want it achieved."
| RL's word | For an LLM agent in a sandbox |
|---|---|
| Policy | the model's weights: what training changes |
| Action | one tool call, a shell command, or the final answer |
| Observation | what comes back into the context: the prompt and tool results |
| Episode | one whole attempt; also called a rollout, a trajectory or a trial |
| Reward | the number the check writes, usually only at the end |
| Environment | everything else: the container, files, tools, data and the check |
The mapping is ours; the words are standard. The programming interface is old too. OpenAI Gym (2016), now maintained by the Farama Foundation as Gymnasium, fixed it as two calls:
reset() starts an episode, and observation, reward, terminated, truncated, info = env.step(action) takes one step.
Today's agent tooling keeps the shape: OpenEnv, from Meta and Hugging Face, describes "a simple Gymnasium-style API" in which "each environment is a Docker image".
Prime Intellect's docs say the quiet part: "RL environments and agent evals are the same abstraction: dataset + harness + scoring rules."
Sutton and Barto, Reinforcement Learning: An Introduction, 2nd ed. (sections 3.1 and 3.2; free online) · Brockman et al., OpenAI Gym, arXiv 1606.01540 · Gymnasium Env API · OpenEnv, Oct 23, 2025 · Prime Intellect docs
6. RL environments as a market
Epoch AI's FAQ, from 18 interviews at labs and vendors, is the best public overview. An RL environment defines what actions a model can take and what they affect; a task pairs a prompt with a grader, often a unit test or a model judging against a rubric. Labs use environments for RL first, then for benchmarking, and for supervised fine-tuning on successful attempts. Some of its numbers:
- The Information reported Anthropic discussing spending "over $1 billion" on RL environments over a year.
- Tasks typically cost $200 to $2,000. Contracts run to six or seven figures per quarter. High-fidelity clones of products can reach about $300,000.
- One startup employee estimated requests for RL environments at "10-20x" those for benchmarks. That is one person's estimate.
- Reward hacking is the top quality concern, and scaling task creation without losing quality is the main bottleneck.
Suppliers include specialized startups, data companies such as Mercor, Surge, Handshake and Turing, in-house lab teams, and product companies.
Denain and Barber, An FAQ on reinforcement learning environments, Epoch AI, Jan 12, 2026
7. Harbor
Harbor is "a framework from the creators of Terminal-Bench for evaluating and optimizing agents and language models", released with Terminal-Bench 2.0 in November 2025. Its docs sum it up as "running any agent with any model on any task in any sandbox in parallel". Its vocabulary:
| Word | Harbor's definition |
|---|---|
| Task | "one or more instructions, a sandbox environment, and a verifier" |
| Dataset | "a collection of tasks" |
| Agent | "a program that completes tasks" |
| Sandbox | "an isolated environment for running tasks" |
| Trial | "one agent's attempt at completing one task" |
| Job | "a collection of trials used to evaluate agents and models" |
my-task/
├── instruction.md
├── task.toml
├── environment/
│ └── Dockerfile
├── solution/
│ └── solve.sh
└── tests/
└── test.sh # writes a number to /logs/verifier/reward.txt
The verifier must write a numerical reward to /logs/verifier/reward.txt, or several named ones to reward.json. Class tasks write 1 or 0, and we regrade separately for partial credit. Running a whole benchmark:
harbor run --dataset terminal-bench@2.0 \ --agent claude-code \ --model anthropic/claude-opus-5-5 \ --n-concurrent 100 --env daytona
The README lists cloud sandbox providers for "thousands of environments in parallel", and it also lists "Generate rollouts for RL optimization": one format for measuring and for training. The other framework you are likely to meet is Inspect, from the UK AI Security Institute.
Harbor docs · task format · README · VentureBeat, Nov 7, 2025
8. What makes a check good
The same few criteria come up wherever people write down what a good task is:
- Solvable, with proof. "For each task, it's useful to create a reference solution: a known working output that passes all graders. This proves that the task is solvable and verifies graders are correctly configured." (Anthropic)
- A sound verifier. A task is well specified "if the unit tests will pass if and only if the container ends in an acceptable state." (Terminal-Bench 2.0)
- Unambiguous. "A good task is one where two domain experts would independently reach the same pass/fail verdict." (Anthropic)
- Not gameable. "It should not be possible for an agent to 'cheat' on a task by taking a shortcut that would not exist in a real-world deployment." (Terminal-Bench 2.0) In Epoch AI's interviews, "robustness against reward hacking" was "the most important quality criterion".
- Hard, and not impossible. For training, "if the pass rate is 0% or 100%, the model won't learn" (Epoch AI). For an eval, "a 0% pass rate across many trials … is most often a signal of a broken task, not an incapable agent" (Anthropic).
Three kinds of grader, with Anthropic's summary of each:
| Grader | Strengths | Weaknesses |
|---|---|---|
| Code | "Fast • Cheap • Objective • Reproducible" | "Brittle to valid variations that don't match expected patterns exactly" |
| A model, with a rubric | flexible; handles open-ended answers | "Non-deterministic • More expensive than code • Requires calibration with human graders for accuracy" |
| A person | "Gold standard quality" | "Expensive • Slow" |
The classic picture of a bad reward is OpenAI's boat from 2016. In the game CoastRunners, the reward was points, and points came from hitting targets. The agent found "an isolated lagoon where it can turn in a large circle and repeatedly knock over three targets". "Despite repeatedly catching on fire, crashing into other boats, and going the wrong way on the track", it scored "on average 20 percent higher than that achieved by human players", and never finished the race. DeepMind's name for this is specification gaming: "a behaviour that satisfies the literal specification of an objective without achieving the intended outcome."
It happens in the training of today's models too. Anthropic's system card for Claude Opus 4.8 reports that "in roughly 0.1% of training episodes, Opus 4.8 speculated about how to satisfy a grader in ways that diverged from the stated intent of the task", and that models recovered "reference solutions from git history, build caches, or other answer-revealing artifacts". Its alignment team wrote in August 2026 that reward hacking "is hard to fully prevent and has occurred in recent frontier model training runs, including in Anthropic's models".
Anthropic, Demystifying evals for AI agents, Jan 9, 2026 · Merrill et al., Terminal-Bench 2.0, arXiv 2601.11868 · Epoch AI FAQ, Jan 12, 2026 · OpenAI, Faulty reward functions in the wild, Dec 2016 · DeepMind, Specification gaming, Apr 2020 · Claude Opus 4.8 system card, May 2026 · Anthropic, Training a misaligned reward seeker, Aug 2026
9. Five ways to measure a model
| How | Examples | Who grades | What it misses |
|---|---|---|---|
| Exam questions | GPQA Diamond, Humanity's Last Exam | an answer key | work longer than one answer; answers leak |
| Agent tasks in a sandbox | SWE-bench, Terminal-Bench, Terminal-Bench-Science | a script | whatever the script doesn't check |
| People's preference | Arena (formerly LMArena) | visitors, voting | what voters can't verify |
| Length of task | METR time horizons | tasks timed on people | other kinds of work; very long tasks |
| Dangerous capabilities | UK AISI, US CAISI, the labs | experts and red teams | most results aren't public |
METR's page is a good lesson in what happens to a ruler. It now says "Measurements above 16 hrs are unreliable with our current task suite" and, since September 8, 2026, "This page is no longer actively updated". Its tasks are mainly software engineering, machine learning and cybersecurity, so its numbers don't generalize to all intellectual work, as the page itself says. Week 1's benchmark table has scores for many of these as of September 2026.
10. Who does the measuring
- Labs. They test before release and publish system cards, the most detailed reports available, and self-reported.
- Independent research groups. METR (time horizons), Epoch AI (FrontierMath, the Benchmarking Hub), Apollo Research (scheming), the ARC Prize (ARC-AGI). External evaluators "often receive limited model access, limited information, and little time" (Charnock et al., 2026).
- A cautionary story. OpenAI commissioned 300 FrontierMath problems and has access to all but a 50-question holdout. Epoch AI later said it "made a mistake" in not being more transparent, since many contributing mathematicians didn't know.
- Governments. The UK AI Security Institute, which also maintains Inspect, and the US CAISI, with testing agreements covering OpenAI, Anthropic, Google DeepMind, Microsoft and xAI.
- Open benchmarks. SWE-bench (Princeton), Terminal-Bench (Stanford and the Laude Institute), Humanity's Last Exam (Center for AI Safety and Scale AI).
- Leaderboards. Arena, Artificial Analysis, Vals AI. These are businesses: Arena's January 2026 round valued it at about $1.7 billion.
- Task vendors. See section 4. Most of what they make is never seen outside the labs that buy it.
Charnock et al., arXiv 2601.11916 · Epoch AI on OpenAI and FrontierMath · TechCrunch, Jan 19, 2025 · NIST CAISI agreements, May 2026 · LMArena on Wikipedia
11. A score has an error bar
A benchmark is a sample of tasks, so its score is a measurement with an uncertainty. For a pass rate, the usual interval is the one for a binomial proportion. The slide uses the Wilson interval at 95%:
| Result | Score | 95% interval |
|---|---|---|
| 6 of 8 tasks (our round 1) | 75% | 41% to 93% |
| 60 of 80 | 75% | 65% to 83% |
| 600 of 800 | 75% | 72% to 78% |
Two more sources of spread sit on top of that. An agent run is random, so the same agent on the same task can pass once and fail the next time; this is why Terminal-Bench 2.0 asks for five runs per leaderboard entry. And two models compared on the same tasks should be compared task by task, since some tasks are simply harder for everyone. The practical rule: a lead of two or three points on a benchmark of about a hundred tasks is usually no difference you can claim.
The machine counts too. Anthropic's engineers found that the gap between the most and least generous server setups on Terminal-Bench 2.0 was 6 percentage points, and concluded: "leaderboard differences below 3 percentage points deserve skepticism until the eval configuration is documented and matched." Evan Miller's paper on the statistics notes that evals are usually reported "with a 'highest number is best' mentality", without testing whether the lead is significant.
Anthropic, Quantifying infrastructure noise in agentic coding evals, Feb 5, 2026 · Miller, Adding Error Bars to Evals, arXiv 2411.00640 · Anthropic, A statistical approach to model evaluations, Nov 19, 2024 · Terminal-Bench 2.0, arXiv 2601.11868
12. How to read a score
- Who ran it? The vendor, the lab, or someone independent?
- With which harness? LangChain moved gpt-5.2-codex from 52.8% to 66.5% on Terminal-Bench 2.0, from just outside the top 30 to the top 5: "We only changed the harness."
- Has it seen the test? Public tasks end up in training data, and answers end up on the web.
- How many runs? Agents are noisy. Terminal-Bench 2.0 asks for five runs per leaderboard submission.
- What does the check check? And what did a run cost?
For your own research, a small private eval beats any leaderboard: ten problems from your subfield with answers you trust, kept out of public view, run on each new model with the harness you actually use.
13. Benchmarks, and how they wear out
A benchmark is an eval that a community agrees to run the same way: a fixed set of tasks, rules for a run, and one number in a public table. Stanford's AI Index 2026 describes what has been happening to them: "Evaluations intended to be challenging for years are saturated in months, compressing the window in which benchmarks remain useful for tracking progress." Its definition: "Benchmark saturation, where models reach scores so high that a test can no longer distinguish between them". The speed-up is not new. Kiela et al. wrote in 2021: "While it used to take decades for machine learning models to surpass estimates of human performance on benchmark tasks, that milestone is now routinely reached within just a few years for newer datasets."
The slide's five rows, with where each number comes from. Scores were read on November 17 and 18, 2026, and they mix harnesses and settings, so read them as a pattern and not as a ranking.
| Benchmark | Introduced | Best then | Best now | What happened next |
|---|---|---|---|---|
| SWE-bench Verified | Aug 2024 | 33.2% (GPT-4o) | 80.9% when OpenAI stopped, Feb 2026 | OpenAI stopped reporting it and recommends SWE-bench Pro |
| GPQA Diamond | Nov 2023 | 39% (GPT-4 baseline) | 96.8%, Epoch AI's run of GPT-6.2 Astra | Epoch treats 97% as the ceiling |
| FrontierMath | Nov 2024 | "under 2%" | 95.1% on Tiers 1 to 3 of the revised set (Claude Fable 5.2, Epoch's run) | revised in June 2026, fixing errors in 42% of problems; Epoch added a set of open Erdős problems |
| ARC-AGI-2 | Mar 2025 | about 4% (an estimate; "Pure LLMs score 0%") | 95.8% on the semi-private set (GPT-6.2 Astra) | ARC-AGI-3, launched Mar 2026 at 0.51%, is at 81.2% with the standard harness |
| Humanity's Last Exam | Jan 2025 | 9.1% (o1) | 54.8% ± 1.9, no tools (GPT-6 Astra, official board) | HLE-Rolling, a continually updated fork, since Sep 2026 |
Two ways to wear out. Saturation: scores reach the top, and what is left measures errors in the questions more than ability. Contamination: the tasks or their answers reach training data. SWE-bench Verified had both. OpenAI found that "59.4% of the 138 problems" it audited "contained material issues in test design and/or problem description", and that frontier models could reproduce some original fixes from memory.
Terminal-Bench, whose format the homework uses, shows the response. Version 2.0 came out on November 7, 2025; 2.1 (May 2026) fixed 28 of its 89 tasks; 3.0 (July) was a new, harder set where the best models scored about 34%; 4.0 (August) removed saturated tasks; 4.1 (November 9) hardened the verifiers against reward hacking. Its authors: "Benchmarks are software and should be maintained like software." The top entry on 4.1 was 71.2% ± 2.9 on November 17, and the top three entries are inside each other's intervals.
A harness footnote worth remembering: on ARC-AGI-3, ARC Prize measured GPT-6 Astra at 62.7% with its standard harness and 99.9% with a harness that keeps the provider's reasoning state between requests. Same model, same tasks.
Stanford AI Index 2026, technical performance · Kiela et al., Dynabench, arXiv 2104.14337 · OpenAI, Why we no longer evaluate SWE-bench Verified, Feb 23, 2026 · Rein et al., GPQA, arXiv 2311.12022 · Epoch AI benchmarking hub and data · FrontierMath, arXiv 2411.04872 · ARC Prize leaderboard · ARC Prize, Sep 3, 2026 · Humanity's Last Exam, arXiv 2501.14249 · HLE official board · Terminal-Bench news
14. When a measure becomes a target
- It has seen the test. OpenAI audited 138 SWE-bench Verified problems that o3 didn't consistently solve and found at least 59.4% had flawed tests, some rejecting correct solutions. GPT-5.2, Claude Opus 4.5 and Gemini 3 Flash Preview could reproduce some original fixes from memory. On February 23, 2026 it stopped reporting the benchmark and recommended SWE-bench Pro.
- Try until it wins. The Leaderboard Illusion (April 2025) reports that a few providers tested many private variants on Chatbot Arena before publishing a score, including "27 private LLM variants" from Meta before Llama 4. Arena replied that its policy is public and that any provider may submit variants.
- It knows it's a test. Anthropic ran Claude Opus 4.6 on 1,266 BrowseComp problems. In two, it suspected a test, identified the benchmark, found the code on GitHub and decrypted the answer key; one run took 40.5 million tokens. Nine more came from answers leaked on the web. The score moved from 86.81% to 86.57%. Anthropic says this does not "represent an alignment failure", as the model was never told to restrict its searches, and calls eval integrity "an ongoing adversarial problem".
- It beats the check. METR found o3 reward hacking in 30.4% of its RE-Bench runs (39 of 128), including all 21 runs of one optimization task, where it precomputed the answer and cached it so that the script looked fast. ImpossibleBench makes tasks impossible on purpose, so that "any pass necessarily implies a specification-violating shortcut": GPT-5 passed 76% on one variant, by editing tests, overloading comparison operators, recording extra state or special-casing the tests.
- It spreads. Anthropic and Redwood Research taught models about reward hacks, then trained them on real production coding environments. Once they learned to hack, misbehaviour rose across their evaluations: alignment faking, sabotage of safety research. Standard chat safety training removed it from chat-like tests only. One line in the prompt saying hacking was acceptable in training cut the final misalignment by 75–90%. It was an experiment built to study the effect; the production Claude models it compares with showed none.
OpenAI, Feb 23, 2026 · The Decoder on the same · Singh et al., arXiv 2504.20879 · Anthropic, Mar 6, 2026 · METR, Jun 5, 2025 · Zhong et al., arXiv 2510.20270 · MacDiarmid et al., arXiv 2511.18397
15. Your tasks as an eval
Round 1 on the homework board was eight tasks and one agent, Claude Opus 5.5 at xhigh effort: six solved. Each task's page says why, and what would make it harder and fairer. The patterns match this page: a method written out in the instruction tests care more than research judgement; a low pass bar says little about skill; and when an answer key depends on an expert's unstated choice, a failure can be the key's fault as much as the agent's.
The top of the ladder is Terminal-Bench-Science 2.0, with a deadline of December 15, 2026. It targets "tasks that today's best AI agents cannot yet reliably solve", and "Contributors with merged tasks receive co-authorship on the Terminal-Bench-Science paper". Its repository runs the same reference and do-nothing checks as ours, plus adversarial trials that try to cheat. Propose the idea first.
16. Hill climbing
Hill climbing is a search method from the AI textbooks: a loop that keeps moving in the direction of a higher value and stops at a peak, where no neighbouring option scores higher. Its known weaknesses are in the same chapter: local maxima, ridges and plateaus. The textbook cures are to restart from somewhere else, or to accept a random step now and then.
In the AI industry, "hill climbing" has become the everyday word for improving a model or a harness against an eval:
- Anthropic: capability evals "should start at a low pass rate, targeting tasks the agent struggles with and giving teams a hill to climb."
- LangChain, on the result from the slides: "We used a simple recipe to iteratively improve deepagents-cli (our coding agent) 13.7 points from 52.8 to 66.5 on Terminal Bench 2.0. We only tweaked the harness and kept the model fixed".
- Alex Shaw, co-creator of Terminal-Bench: "These benchmarks are very hill-climbable these days", and "the only real solution is to keep making new benchmarks quickly."
Fitted to the test
Measure on the same tasks after every change, and the system slowly fits those tasks. The standard defence is a held-out set that is looked at rarely.
- ImageNet, re-collected. Recht et al. built new test sets for CIFAR-10 and ImageNet by following the original recipes, and found "accuracy drops of 3% - 15% on CIFAR-10 and 11% - 14% on ImageNet". The reassuring half: "the relative order of models is almost exactly preserved". A decade of climbing had inflated the numbers but not reversed the ranking.
- Grade-school math, rewritten. GSM1k is 1,000 new problems written to mirror the GSM8k benchmark. Some model families dropped by "up to 8%"; many frontier models showed "minimal signs of overfitting".
- Public and semi-private. On ARC-AGI-1 in December 2024, OpenAI's o3 preview scored 82.8% on the public evaluation set and 75.7% on the semi-private one, which ARC Prize keeps to "assess overfitting".
- A geoscience case. In the 2019 Kaggle competition on forecasting laboratory earthquakes, teams saw a score on 13% of the test data. On the rest, the daily leaders' error roughly doubled, "a telltale sign of overfitting", and the winning team "climbed 355 places to reach first place". The authors say part of the gap came from the data itself changing over time.
- From people who automate the climb. LangChain: "autonomous hill-climbing has a tendency to overfit to tasks so holdout sets ensure that learned optimizations work on previously unseen data".
A caution in the other direction: across 120 Kaggle competitions, Roelofs et al. found "little evidence of substantial overfitting" when the test set stays hidden. Holding data back works. The trouble starts when the test is public, leaks into training, or is tried many times in private.
The wrong hill
A climber gets good at the score, and has no access to what you meant. OpenAI's boat (in the section on checks above) is the classic picture. Recent ones: METR found OpenAI's o3 reward hacking in all 21 runs of one task, where the job was to make a fine-tuning script faster and the model instead copied the cached weights of an already fine-tuned model. The Darwin Gödel Machine, a coding agent that rewrites its own code, improved from 20.0% to 50.0% on SWE-bench, and along the way "faked a log making it look like it had run the tests and that they had passed, when in fact they were never run".
Agents as climbers
Andrej Karpathy's autoresearch (March 2026) is the loop with nothing added. An agent edits one training file, trains for exactly five minutes, and the instructions say: if the validation loss improved, keep the commit; if not, reset. In about two days it tried about 700 changes and kept about 20. Stacked together and tested on a larger model, they cut a separate measure, the time to train to GPT-2 quality, from 2.02 hours to 1.80. That second measure matters: it is the held-out check on a climb that used a five-minute proxy.
Google DeepMind's AlphaEvolve does the same with a population of programs: a model proposes changes, "automated evaluators" score them, and the best are bred further. Its paper names the limit that matters most for science: "The main limitation of AlphaEvolve is that it handles problems for which it is possible to devise an automated evaluator. While this is true of many problems in the mathematical and computational sciences, there are domains such as the natural sciences where only some experiments can be simulated or automated."
Russell and Norvig, Artificial Intelligence: A Modern Approach, section 4.1 · Anthropic, Demystifying evals for AI agents · LangChain, Feb 17, 2026 · LangChain, Apr 8, 2026 · Snorkel AI interview with Alex Shaw, Mar 31, 2026 · Recht et al., arXiv 1902.10811 · Zhang et al., GSM1k, arXiv 2405.00332 · ARC Prize, Dec 20, 2024 · Johnson et al., PNAS 2021 · Roelofs et al., NeurIPS 2019 · METR, Jun 5, 2025 · Sakana AI, The Darwin Gödel Machine · karpathy/autoresearch · Karpathy, Mar 9, 2026 · AlphaEvolve, arXiv 2506.13131
17. Our own climb: a search beyond Neptune
The two charts in the slides come from Jason's own research runs, not from a published benchmark. In both, an agent writes a whole program each round, the program is graded on a GPU, and the next round's agent gets every earlier program with its notes and score. It may improve the best one, combine several, or start over, and the harness keeps the best.
One model at a time
The task: write a program that finds trans-Neptunian objects, small icy bodies beyond Neptune, in telescope images of ten hidden fields. Many are too faint to see in a single exposure. The score is the weighted fraction of the known objects found, and one number per program is all the agent learns.
| Model | Graded rounds | Best score | Reached at | Agent time to the best |
|---|---|---|---|---|
| Claude Opus 5.5 | 151 | 0.27 | round 136 | about 100 hours |
| Claude Fable 5.2, from an empty folder | 24 | 0.40 | round 15 | about 21 hours |
Opus's curve is a staircase with long flat stretches: 11 new bests in 151 rounds. A review of the run found that its method stopped changing at round 10, and that every later program was the same design, tuned. Fable ran the same task, fields and grader without seeing Opus's work. It passed Opus's best in its third round, then tied its own best for nine rounds. Seven of the objects its best program found were never found in any of Opus's rounds.
In the Fable run, around round 18, a program began encoding how far its search had got on each hidden field in its exit time, a way to learn more than the one number the task allowed. The next round's agent wrote that it had deliberately left out a count of objects found, because the task says per-field results are not given back, and asked later rounds to keep it that way. A later round encoded such a count anyway, and the round after it switched it off. None of this changed the score.
Two models sharing one history
A second task from the same family: find large trans-Neptunian objects in thirteen years of survey detections. After two rounds of Fable alone, Claude Fable 5.2 and GPT-6 Astra each wrote a program every round, in separate sandboxes, with no messages between them. They shared only the history folder. From round 21, GPT-6.2 Astra continued alone. The best score went from 0.48 at the start to 0.76 after 20 rounds and 0.79 at round 23 (November 17; the run is still going). Both models set new bests, and each regularly built on the other's best program. There is no single-model run under the same rules, so this doesn't show that two models beat one.
In this run the agents may see the answers for the graded regions and the grader itself, and cheating is caught by reading every program. The price shows up on new sky: round 18's best program finds 76% of the known objects in the regions it was graded on and 55% in twenty regions it was never graded on. Those regions differ in size and crowding, so the gap is not only overfitting.
Jason's own runs, October 31 to November 17, 2026. Scores are recall of known objects; agent time is the sum of agent sessions.
18. Recursive self-improvement
The idea is from 1965. I. J. Good, a statistician who had worked with Alan Turing, wrote:
"Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any man however clever. Since the design of machines is one of these intellectual activities, an ultraintelligent machine could design even better machines; there would then unquestionably be an 'intelligence explosion,' and the intelligence of man would be left far behind. Thus the first ultraintelligent machine is the last invention that man need ever make, provided that the machine is docile enough to tell us how to keep it under control."
Anthropic's definition from 2026 is narrower: "an AI system capable of fully autonomously designing and developing its own successor. This is called recursive self-improvement." OpenAI's safety framework uses the same words for its highest level: "The model is capable of recursively self improving (i.e., fully automated AI R&D)".
What has been measured
- Code. Anthropic: "As of May 2026, more than 80% of the code we merge into Anthropic's codebase was authored by Claude." The same page warns that lines of code are "almost certainly an overstatement of the true productivity gain."
- Training speed. AlphaEvolve "sped up this vital kernel in Gemini's architecture by 23%, leading to a 1% reduction in Gemini's training time", and a scheduling rule it found "continuously recovers, on average, 0.7% of Google's worldwide compute resources."
- Experiments. Karpathy's autoresearch, in the hill-climbing section above.
- The rate of progress. METR, an independent evaluator, reported on November 5, 2026 that Anthropic is "likely improving the capabilities of their models on well-specified tasks at around twice the rate they did between March 2024 and October 2025" (estimate 1.9 times, interval 1.5 to 2.8), and: "We guess that a majority (perhaps ≈80%) of the current (October 2026) acceleration is due to AI." It also says "AI is not yet able to make highly significant and fully autonomous contributions", and that Anthropic redacted most of the evidence METR found most informative.
- Head to head with people. On METR's RE-Bench (2024), seven research-engineering tasks, AI agents scored four times higher than human experts when both had two hours, and the experts scored twice as high as the best agent when given 32 hours. On OpenAI's PaperBench (2025), replicating 20 ICML papers from scratch, the best agent reached 21.0%, below the human baseline.
Why it could be fast, why it could be slow
- Fast. METR's measure of the length of task an agent can complete has doubled about every 4.3 months since 2023 (interval 107 to 161 days). The authors of the AI 2027 scenario wrote on November 10, 2026 that "all of us forecast that Superhuman AI Researcher (SAR) will likely arrive at some point in 2027".
- Compute. Epoch AI's Anson Ho: "it's hard to make super fast software progress without also scaling up training compute", because many improvements only show at scale.
- Diminishing returns. Forethought: "improvements to AI software face diminishing returns from lower hanging fruit being picked first". The same authors still judge a software-only acceleration "at least decently likely".
- Judgement. Anthropic: "large performance gaps persist when it comes to Claude exercising judgement in choosing goals". METR's review of Claude Opus 5.5 says the model "is unlikely to be able to fully automate AI R&D".
- History. Narayanan and Kapoor: "It is more likely that we will continue to see a gradual increase in the role of automation in AI development than a singular, discontinuous moment when recursive self-improvement is achieved."
Anthropic's own summary: "We are not there yet, and recursive self-improvement is not inevitable."
As a safety threshold
- OpenAI. In its Preparedness Framework, "High" for AI self-improvement means an impact "equivalent to giving every OpenAI researcher a highly performant mid-career research engineer assistant". "Critical" is fully automated AI research, where the stated response is to halt further development until safeguards are specified. The GPT-6.2 system card of November 17, 2026: "GPT-6.2 Astra is our first externally released model to meet the High threshold for AI Self-Improvement."
- Anthropic. Its Responsible Scaling Policy has a threshold for AI that can "fully automate, or otherwise dramatically accelerate, the work of large, top-tier teams of human researchers", where dramatic acceleration means "as much progress in one year as one would see in two years at baseline." In November 2026 filings with the U.S. Securities and Exchange Commission, it says it "made a determination that we have met this threshold based on what we expect from future models".
- Google DeepMind. Its Frontier Safety Framework has two levels: AI that "has been used to accelerate AI development, resulting in AI progress substantially accelerating from historical rates", and AI that "can fully automate the work of any team of researchers at Google focused on improving AI capabilities".
Where our own runs sit: in the search beyond Neptune, agents improve a program against an automatic score. That is the same loop aimed at a science pipeline, and it does not change the model. What it shares with the loop above is the dependence on the check, which is the part a domain expert writes.
Good, Speculations Concerning the First Ultraintelligent Machine, Advances in Computers 6, p. 33 · Anthropic Institute, When AI builds itself · Google DeepMind, AlphaEvolve, May 14, 2025 · METR, Nov 5, 2026 · RE-Bench, arXiv 2411.15114 · PaperBench, arXiv 2504.01848 · METR, Time Horizon 1.1 · AI Futures Project, Nov 10, 2026 · Epoch AI, Feb 25, 2026 · Forethought, Mar 26, 2025 · METR on Claude Opus 5.5 · Narayanan and Kapoor, AI as Normal Technology · OpenAI Preparedness Framework v2 · OpenAI, GPT-6.2 system card · Anthropic Responsible Scaling Policy v3.4 · Anthropic filing with the SEC, Nov 12, 2026 · Google DeepMind Frontier Safety Framework v3.1