Week 8 slides: measure, then climb

← → to move, F for full screen, N for speaker notes

AI in Research · Week 8 · November 18, 2026

Measure, then climb

How an AI's ability is trained and measured, what a benchmark is, and what happens when people, and then the AI itself, start climbing the score.

From week 6

AI eval, the part we didn't reach on Nov 4: how models are trained, how they are measured, and by whom

New

RL environments up close, benchmarks and how they wear out, hill climbing, and recursive self-improvement

class.xhelio.ai/week-8

Week 8 · Recap of week 602

Last time: an agent that stays on

PartThe one thing to keep
Waking upIt's mostly asleep. Code watches; the model is called only when something happens.
RememberingCompaction trades detail for room. What must survive goes in files.
ControlBudgets, timeouts and the kill switch are code, not a prompt.
SurvivingOnly the log survives a crash: append, resume, do each action once.

Your overnight failures were each found by checking, and one memory write passed its own check while destroying the notes. So: how do you check an AI at all?

Week 8 · Today03

Today: five ideas, each built on the one before

1 RL environment2 AI eval3 Benchmark4 Hill climbing5 Self-improvement A task, a sandbox andan automatic check.The model practicesagainst it. The same task withthe weights frozen.Out comes a score. A fixed set of evalsthat everyone runsthe same way, soscores can be compared. Change the system,run the benchmark,keep what raisesthe score. Repeat. The system does thechanging itself: AIthat improves howAI is made.

Parts 1 to 3 include week 6's AI eval slides, which we didn't reach on Nov 4. Concepts only today.

Week 8 · 1 RL environments04

Where a model's ability comes from

Post-training Pre-trainingChat tuningReinforcement learningEvaluation trillions of wordsexample chatstasks in a sandboxthe same kind of task reward: the next tokenreward: human ratingsreward: an automatic checkweights frozen a base modelan assistantreasoning, agentsa score what gets measured gets trained

Week 1 named these stages. Today: what each one does, how it gets measured, and by whom.

Week 8 · 1 RL environments05

Pre-training: learn to continue any text

The solar wind near Earth blows at about 400 ___ kmkilometersmileseverything else 0.860.090.020.03 Illustrative numbers. The model bets on every next token.

Hide the next token, predict it, nudge the weights. Repeat over trillions of words.

A real base model: GPT-3

# the prompt Explain the moon landing to a 6 year old in a few sentences. # what it writes next Explain the theory of gravity to a 6 year old. Explain the theory of relativity to a 6 year old in a few sentences. Explain the big bang theory to a 6 year old. Explain evolution to a 6 year old.

It sees a list of exercises, so it writes more exercises. It isn't broken: nobody has told it what an answer is.

Out comes a base model. What it knows comes from here. How it behaves doesn't exist yet.

OpenAI, Aligning language models to follow instructions, Jan 2022 (the moon-landing example) · Ouyang et al., arXiv 2203.02155

Week 8 · 1 RL environments06

Post-training: learn how to behave

The same prompt, after post-training: InstructGPT

Explain the moon landing to a 6 year old in a few sentences. People went to the moon, and they took pictures of what they saw, and sent them back to the earth so we could all see them.

100× smaller

People still preferred it: 1.3B parameters beat the 175B GPT-3.

1

People write good answers; the model imitates them.

2

People rank pairs of the model's answers.

3

A reward model learns what they prefer.

4

RL pushes the model toward high scores.

About 40 people and tens of thousands of prompts changed how it behaves. This is the step that led to ChatGPT.

Ouyang et al., arXiv 2203.02155 (2022): about 40 labelers; 13k, 33k and 31k training prompts

Week 8 · 1 RL environments07

Reinforcement learning: practice against a check

1 Pick a task2 Try it, many times3 The check scores it4 Update the weights an instruction and a sandboxone attempt is a rollout; for anagent, hundreds of steps1 or 0, or anything betweenmore of what scored,less of what didn't again, thousands of times

DeepSeek-R1, 2025

Reward: is the answer right? Plus a format check. No examples of how to reason.

Wait, wait. Wait. That's an aha moment I can flag here.

Reflecting and checking its own work emerged in training.

About $294,000 of compute, on top of an existing base model.

Agents learn agent work the same way, in sandboxes. The check is what the model learns to want.

Guo et al., DeepSeek-R1, Nature 645, 633 (2025), arXiv 2501.12948 · cost from its supplementary information, reported Sep 2025

Week 8 · 1 RL environments08

Step 4, up close: how a reward becomes a weight update

1 Four attempts at one task2 The check scores each3 Compare with the average, then nudge each square is one choice: a token, a tool callaverage of the four: 0.5every choice in the attempt moves together passed: reward 1 +0.5 its choices become a little more likely failed: reward 0 −0.5 its choices become a little less likely failed: reward 0 −0.5 its choices become a little less likely passed: reward 1 +0.5 its choices become a little more likely

weight change ∝ (reward − average) × (what makes those choices more likely)

Nobody says which step was right.

The whole attempt shares the credit. Over many attempts, the useful choices stand out.

Pre-training is shown the right token.

Here the model learns only how the whole attempt went.

All attempts score the same?

Then reward − average is zero, and nothing is learned. Too hard and too easy both teach nothing.

Policy gradient: Williams, Machine Learning 8, 229 (1992); Sutton and Barto, ch. 13 · DeepSeek-R1 (arXiv 2501.12948) compares each attempt with its group's average

Week 8 · 1 RL environments09

The compute is moving to post-training

One model: post-training compute as a share of its pre-training log scale · five models that published numbers · units differ, so read the order of magnitude 0.1% 1% 10% 100% as much as pre-training 2022 2023 2024 2025 2026 InstructGPT · 1.6% DeepSeek-V3 · 0.19% DeepSeek-R1 · 5.5% DeepSeek-V3.2 · over 10% Beam · over 170% All compute at OpenAI + Anthropic an outside estimate (SemiAnalysis); method not public pre-trainingpost-training, RLserving users 67% 5% 29% 1Q 2024 32% 36% 32% 4Q 2025 23% 42% 35% 1Q 2026 7% 55% 38% 4Q 2026 projected

Mid-training is a newer name for a stage in between: still predicting the next token, but on a small, curated mix. Where its size is public, it is a few percent of pre-training.

No lab publishes this split. Every public number points the same way: the compute is moving into environments.

arXiv 2203.02155 · arXiv 2412.19437 · DeepSeek-R1, Nature 645, 633, suppl. table 4 · arXiv 2512.02556 · Reflection AI, Oct 5, 2026 · SemiAnalysis as reported by OfficeChai, Oct 18, 2026

Week 8 · 1 RL environments10

An environment is whatever answers the agent

AgentEnvironment the model,in its harnessa sandbox,and a check action: a tool callobservation: its result hundreds of times reward: one number, when the attempt ends episode = rollout = trial = one attempt
RL's wordFor your agent
policythe model's weights: what gets trained
actionone tool call, or the final answer
observationwhat comes back into the context
episodeone whole attempt at the task
rewardthe number the check writes at the end
environmenteverything else: files, tools, data, the check

It's week 2's loop, seen from the other side. You have been building environments since homework 1.

Sutton and Barto, Reinforcement Learning: An Introduction, 2nd ed. (2018), section 3.1: the environment is "everything outside the agent"

Week 8 · 1 RL environments11

An RL environment and an eval are the same object

In training

The reward goes back into the weights. The model changes.

a task instruction sandbox automatic check → a reward

In an eval

The weights stay frozen. The reward goes into a table.

over $1 billion

Anthropic discussed spending this on RL environments in one year, as reported in 2025

$200–$2,000

the typical price of one task

10–20×

requests for RL environments compared with benchmarks, by one vendor's estimate

Your homework 1 task could do either job. Once a task has been trained on, it stops measuring.

Epoch AI, An FAQ on reinforcement learning environments, Jan 12, 2026 (18 interviews; the $1 billion figure is The Information's)

Week 8 · 1 RL environments12

Harbor: any agent, any model, any task, any sandbox

a task: the folder from your homework 1

my-task/ ├── instruction.md # what the agent is told ├── task.toml # time limit, CPUs, memory, network ├── environment/ # the sandbox: a Dockerfile and data ├── solution/solve.sh # proof that it can be done └── tests/test.sh # writes /logs/verifier/reward.txt

a job: every trial, in parallel

harbor run --dataset terminal-bench@2.0 \ --agent claude-code \ --model anthropic/claude-opus-5-5 \ --n-concurrent 100 --env daytona

Trial: one agent's attempt at one task.

Job: a collection of trials.

agents and models →task 1task 2task 3task 4 ← one trial

From the makers of Terminal-Bench. One format to report a score, or to "generate rollouts for RL optimization."

docs.harborframework.com · github.com/harbor-framework/harbor (README, 6.8k stars) · VentureBeat, Nov 7, 2025

Week 8 · 1 RL environments13

The hard part of an environment is the check

1

Can it be done?

A reference solution passes.

2

Does doing nothing fail?

An empty answer scores zero.

3

Does every right answer pass?

A second expert, working another way, still gets credit.

4

Can it pass without the work?

No shortcut reaches the reward.

Who gradesGood atWatch for
A scriptcheap, fast, the same every timeit checks only what you thought of
Another model, with a rubricopen-ended answers, at scalenot the same every time; must be checked against people
A personjudgementslow and costly, so it can't sit inside a training loop

These four questions are homework 6. A model in training will find the answer to number 4 before you do.

Anthropic, Demystifying evals for AI agents, Jan 9, 2026 · Terminal-Bench 2.0, arXiv 2601.11868 · Epoch AI FAQ, Jan 2026

Week 8 · 2 AI eval14

Five ways to measure a model

HowExamplesWho gradesWhat it misses
Exam questionsGPQA Diamond, Humanity's Last Examan answer keywork longer than one answer; answers leak
Agent tasks in a sandboxSWE-bench, Terminal-Bench, Terminal-Bench-Sciencea scriptwhatever the script doesn't check
People's preferenceArena (formerly LMArena)visitors, votingwhat voters can't verify
Length of taskMETR time horizonstasks timed on peoplemost other work; "above 16 hrs … unreliable"
Dangerous capabilitiesUK AISI, US CAISI, the labs themselvesexperts and red teamsmost results aren't public

Before you ask what the score is, ask what kind of test it is.

metr.org/time-horizons: "Measurements above 16 hrs are unreliable with our current task suite"; no longer updated (Sep 2026)

Week 8 · 2 AI eval15

Who does the measuring

WhoExamplesWhat they doKeep in mind
LabsOpenAI, Anthropic, Google DeepMindpre-release tests, system cardsthey grade their own work
Independent researchMETR, Epoch AI, Apollo Research, ARC Prizetime horizons, FrontierMath, scheming tests, ARC-AGIlimited access; OpenAI funded FrontierMath
GovernmentsUK AI Security Institute, US CAISIpre-release tests; Inspect, open sourcefive US developers test with CAISI
Open benchmarksSWE-bench, Terminal-Bench, Humanity's Last Examtasks and tools anyone can runpublic tasks leak into training
LeaderboardsArena, Artificial Analysis, Vals AIrun models, rank them, publisha business too: Arena, valued at $1.7B
Task vendorsMercor, Surge, Turing, Handshake, startupssell environments to labsmostly private
Youyour homework 1 task, Terminal-Bench-Sciencetasks from your own fieldfew others can check your science

epoch.ai/blog/openai-and-frontiermath · NIST CAISI, May 2026 · aisi.gov.uk · Arena, Jan 6, 2026 · Epoch AI FAQ, Jan 2026

Week 8 · 2 AI eval16

A score is a measurement, so it has an error bar

0%25%50%75%100% 6 of 8 tasks60 of 80600 of 800 41%93%65%83%72%78%

Few tasks, wide bar. Our round 1, 6 of 8, is the top row: the true rate could be anywhere from about 40% to 90%.

Agents are noisy. The same agent on the same task doesn't always land the same way, so run it more than once.

Two models three points apart on a hundred tasks: usually no difference you can claim.

Intervals: Wilson, 95% · Anthropic, Quantifying infrastructure noise in agentic coding evals, Feb 5, 2026 · Miller, Adding error bars to evals, arXiv 2411.00640

Week 8 · 2 AI eval17

How to read a score

  1. Who ran it? The vendor, the lab, or someone independent?
  2. With which harness? A score belongs to a model and a harness.
  3. Has it seen the test? Public tasks end up in training data.
  4. How many runs? Terminal-Bench 2.0 entries need five.
  5. What does the check check? And what did a run cost?

Same model, new harness

52.8% → 66.5%

gpt-5.2-codex on Terminal-Bench 2.0, from just outside the top 30 to the top 5. "We only changed the harness."

In your own research: keep ten problems with known answers, and run every new model on them.

LangChain, Improving Deep Agents with harness engineering, Feb 17, 2026 · VentureBeat, Nov 7, 2025 (five runs per leaderboard submission)

Week 8 · 3 Benchmarking18

A benchmark: one eval that everyone runs the same way

A fixed set of tasks. The same ones for everybody, this year and next.

Rules for a run. What the agent may use, how long it gets, how many tries.

One number, and a public table of who got what.

Leaderboardillustration, not real scores model A + harness Xmodel A + harness Ymodel B + harness Xmodel C + harness Z

An eval tells you how a model does. A benchmark lets strangers compare: across labs, across models, across years.

Week 8 · 3 Benchmarking19

A benchmark wears out

0%25%50%75%100% SWE-bench Verified 33% · Aug 202481% · Feb 2026 retired GPQA Diamond 39% · Nov 202397% · Nov 2026 FrontierMath under 2% · Nov 202495% · Nov 2026 ARC-AGI-2 about 4% · Mar 202596% · Nov 2026 Humanity's Last Exam 9% · Jan 202555% · Nov 2026
1 Born hard→2 Climbed→3 Worn out: near the top, or leaked into training→4 Replaced by a harder one

"Evaluations intended to be challenging for years are saturated in months." Terminal-Bench went from 2.0 to 4.1 in one year.

Open circle: best score when introduced; filled: best now · OpenAI; Epoch AI data; ARC Prize; Scale and CAIS boards (read Nov 17, 2026) · quote: Stanford AI Index 2026

Week 8 · 3 Benchmarking20

When a measure becomes a target

It has seen the test

SWE-bench Verified. Models reproduced fixes from memory, and at least 59% of 138 audited problems had flawed tests. OpenAI stopped reporting it (Feb 2026).

Try until it wins

Arena. Meta tested 27 private variants before Llama 4; a provider could publish just its best (The Leaderboard Illusion, Apr 2025). Arena replied that any provider may do this.

It knows it's a test

BrowseComp. In 2 of 1,266 problems, Claude Opus 4.6 suspected a test, named the benchmark, found its code on GitHub and decrypted the answer key (Anthropic, Mar 2026).

It beats the check

o3: cached a precomputed answer so a script looked fast, 21 of 21 runs on one METR task. GPT-5: passed 76% of one set of impossible tasks, where every pass is a cheat.

And it spreads: in an Anthropic experiment, models that learned to hack coding checks went on to fake alignment and sabotage safety research.

openai.com/index/why-we-no-longer-evaluate-swe-bench-verified · arXiv 2504.20879 · anthropic.com/engineering/eval-awareness-browsecomp · metr.org, Jun 5, 2025 · arXiv 2510.20270 · arXiv 2511.18397

Week 8 · 3 Benchmarking21

Your tasks are an eval

6 of 8

solved by Claude Opus 5.5, xhigh effort, round 1

TaskWhat happenedWhat it says about the eval
sector-momentum-backtestSolved in 2 minutes, for $0.30Every step was in the instruction: it tested care
magnetosphere-density-transformerSolved: R² 0.104 against a bar of 0.02; it tuned its loss to the verifier's unitsA low bar, and a check the agent could aim at
raw-images-vortex-spacingFound 39 of the expert's 46 vortices, and failed for reporting 23 moreThe answer key had no stated criterion
insight-ifg-interference-cleaning629 of 647 windows within 1 nT, and failedAll or nothing, and part of the key is a convention

Harder and fairer is the rest of the quarter. Top of the ladder: Terminal-Bench-Science 2.0: pull requests until Dec 15, and merged tasks earn co-authorship.

class.xhelio.ai/homework · terminal-bench-science.ai/contribute, accessed Nov 3, 2026

Week 8 · 4 Hill climbing22

Hill climbing: change, measure, keep

each stepone small change, keptonly if the score rose stuckevery small change now scores lower a higher peakinvisible from where you stand to get there, the score must first go down every version of the system you could build →score ↑

You never see the landscape. You only learn whether the last step went up, so you need a score you can trust and afford to run.

Week 8 · 4 Hill climbing23

The same loop, at four speeds

Who climbsWhat they changeThe scoreOne step takes
Pre-trainingthe weights, a littlehow well it predicts the next tokenunder a second
Reinforcement learningthe weights, a littlethe reward from an environmentone rollout: minutes to hours
You, or a companythe prompt, the tools, the harnessa set of tasks with known answersone eval run: hours
The whole fieldthe next modelpublic benchmarksmonths

The highlighted row is week 6's example: the same model went from 52.8% to 66.5% on Terminal-Bench 2.0 by changing only the harness. That was a hill climb.

In every row the climber gets good at the score. Whether that is what you wanted depends on the score.

Week 8 · 4 Hill climbing24

An agent can do the climbing: a search beyond Neptune

0.00.10.20.30.4150100150 best score so far (1.0 = every known object found)round Claude Fable 5.2: 0.40 in 24 rounds started from an empty folder Claude Opus 5.5: 0.27 in 151 rounds round 10: the method stops changing

The task. Write a program that finds faint objects beyond Neptune in telescope images of ten hidden fields.

One round. An agent writes a whole program. A GPU grades it against the known objects. One number comes back.

The next round gets every earlier program and its score, and tries to do better.

Opus spent 116 agent-hours polishing one design. Fable, starting over, passed it in round 3. A different start reached a higher hill.

Jason's own runs, Oct 31 to Nov 12, 2026: the same task, fields and grader for both models; scores are weighted recall of known objects

Week 8 · 4 Hill climbing25

Two models, one shared history

0.50.60.70.80510152023 best score so far (fraction of known objects found)round Fable 5.2 and GPT-6 Astra, side by sideGPT-6.2 Astra round 18's best program, on new sky: 0.55 start: 0.48 0.76 0.79

Fable 5.2 GPT-6 Astra GPT-6.2 Astra
A dot is a new best, coloured by who wrote it.

They never talk. Each round both write a program; all they share is the folder of past programs and scores.

They build on each other. One model's best program becomes the other's starting point, round after round.

A harder cousin of the same task. Here the agents may see the answers, so the red line matters: 0.76 on the graded sky, 0.55 on new sky.

Jason's own run, Nov 14 to 17, 2026, still going · new sky: 20 regions the programs were never graded on

Week 8 · 4 Hill climbing26

Three ways a climb goes wrong

Stuck on a small hill

Every small change scores lower, so the climb stops.

In our runs: Opus's method froze at round 10. 140 more rounds polished it.

Do: start again from somewhere else.

Fitted to the test

The score rises on the tasks you measure, and nowhere else.

In our runs: 0.76 on the graded sky, 0.55 on new sky.

Do: keep a held-out set, and look at it rarely.

The wrong hill

The score goes up without the thing you wanted.

In our runs: a program began signalling hidden results through how long it ran.

Do: read what was built, not only the number.

None of these shows up in the score. The number keeps looking fine.

Week 8 · 5 Self-improvement27

Recursive self-improvement: the climber rebuilds itself

an AI systemdoes AI researcha better AI system today's best modelcode, experiments,environments, evalswhich is betterat AI research each lap makesthe next lap easier

"…an ultraintelligent machine could design even better machines; there would then unquestionably be an 'intelligence explosion'…"

I. J. Good, 1965

Our agents improved a search program. Aim the same loop at how AI is made, and each lap leaves a stronger climber for the next one.

I. J. Good, Speculations Concerning the First Ultraintelligent Machine, Advances in Computers 6 (1965)

Week 8 · 5 Self-improvement28

How much of the loop is closed today

What AI already doesWhat has been measured
Writes the codeMore than 80% of the code merged at Anthropic was written by Claude (May 2026).
Speeds up its own trainingAlphaEvolve made a kernel inside Gemini 23% faster: 1% off Gemini's training time (2025).
Runs the experimentsKarpathy's autoresearch: about 700 changes tried in two days, about 20 kept, training 11% faster (Mar 2026).
All of it, seen from outsideMETR, Nov 2026: Anthropic is improving its models at about twice its earlier rate, most of that likely due to AI.

Still people: choosing what to work on, and judging whether it worked.

The loop is partly closed, with people in it. Our search beyond Neptune is the same loop, aimed at a telescope program.

anthropic.com/institute/recursive-self-improvement · deepmind.google, AlphaEvolve, May 14, 2025 · github.com/karpathy/autoresearch · metr.org, Nov 5, 2026

Week 8 · 5 Self-improvement29

Will it run away? Nobody knows yet

Why it could be fast

The length of task an agent can finish has doubled about every four months since 2023.

Most of the code is already written by AI, and each model helps build the next.

Some forecasters expect an AI that out-researches people in 2027.

Why it could be slow

Experiments need compute. Better code doesn't make more of it.

Judgement lags. Models are weakest at choosing which goals are worth pursuing.

History favours "a gradual increase" in automation over one sudden moment.

Either way, the loop climbs what can be checked. Whoever writes the checks steers it.

Labs now treat it as a safety threshold. Yesterday OpenAI rated GPT-6.2 Astra "High" for AI self-improvement, the first model it has released at that level.

metr.org, Time Horizon 1.1 · AI Futures, Nov 10, 2026 · Epoch AI, Feb 25, 2026 · Narayanan and Kapoor, AI as Normal Technology, 2025 · OpenAI GPT-6.2 system card, Nov 17, 2026

Week 8 · Homework30

Homework 6 is due today: your task as a benchmark

  1. Open your pull request with the hardened task in week-1/submissions/.
  2. The checks run: the reference solution scores 1, doing nothing scores 0.
  3. We run GPT-5.6 Sol on it, once, at xhigh effort. You don't need Docker or a key.
  4. The result and the full trace are posted on your pull request. Read the attempt.

Today's words, on your task

Your task is an environment.

Sol's run on it is an eval.

The class's tasks together are a small benchmark.

Solved? Harden it and ask for another run: you are hill climbing, downhill for the agent.

Still open, and encouraged: propose your task to Terminal-Bench-Science by Dec 15. One merged task qualifies you for co-authorship.

class.xhelio.ai/week-6/homework

Week 8 · Wrap-up31

Takeaways

1 RL environmentA task, a sandbox and a check. The model practices against it, so the check is what it learns to want.
2 AI evalThe same object with the weights frozen. A score has an error bar, and belongs to a model and a harness together.
3 BenchmarkA shared, fixed eval. It wears out: once it is a target, it stops being a good measure.
4 Hill climbingChange, measure, keep. It makes a system good at the score, which is only as good as the score.
5 Self-improvementHill climbing where the AI chooses the changes. Whoever writes the check steers it.

In your field, the people who can write the check are you.

Reading, with sources: class.xhelio.ai/week-8/reading · Homework 6: class.xhelio.ai/week-6/homework · Still to come: how AI fails; keep your failures.md.