←→ to move, F for full screen, N for speaker notes
AI in Research · Week 8 · November 18, 2026
Measure, then climb
How an AI's ability is trained and measured, what a benchmark is, and what happens when people, and then the AI itself, start climbing the score.
From week 6
AI eval, the part we didn't reach on Nov 4: how models are trained, how they are measured, and by whom
New
RL environments up close, benchmarks and how they wear out, hill climbing, and recursive self-improvement
class.xhelio.ai/week-8
Week 8 · Recap of week 602
Last time: an agent that stays on
Part
The one thing to keep
Waking up
It's mostly asleep. Code watches; the model is called only when something happens.
Remembering
Compaction trades detail for room. What must survive goes in files.
Control
Budgets, timeouts and the kill switch are code, not a prompt.
Surviving
Only the log survives a crash: append, resume, do each action once.
Your overnight failures were each found by checking, and one memory write passed its own check while destroying the notes. So: how do you check an AI at all?
Week 8 · Today03
Today: five ideas, each built on the one before
Parts 1 to 3 include week 6's AI eval slides, which we didn't reach on Nov 4. Concepts only today.
Week 8 · 1 RL environments04
Where a model's ability comes from
Week 1 named these stages. Today: what each one does, how it gets measured, and by whom.
Week 8 · 1 RL environments05
Pre-training: learn to continue any text
Hide the next token, predict it, nudge the weights. Repeat over trillions of words.
A real base model: GPT-3
# the promptExplain the moon landing to a 6 year old in a few sentences.# what it writes next
Explain the theory of gravity to a 6 year old.
Explain the theory of relativity to a 6 year old in a few sentences.
Explain the big bang theory to a 6 year old.
Explain evolution to a 6 year old.
It sees a list of exercises, so it writes more exercises. It isn't broken: nobody has told it what an answer is.
Out comes a base model. What it knows comes from here. How it behaves doesn't exist yet.
OpenAI, Aligning language models to follow instructions, Jan 2022 (the moon-landing example) · Ouyang et al., arXiv 2203.02155
Week 8 · 1 RL environments06
Post-training: learn how to behave
The same prompt, after post-training: InstructGPT
Explain the moon landing to a 6 year old in a few sentences.
People went to the moon, and they took pictures of what they saw, and sent them back to the earth so we could all see them.
100× smaller
People still preferred it: 1.3B parameters beat the 175B GPT-3.
1
People write good answers; the model imitates them.
2
People rank pairs of the model's answers.
3
A reward model learns what they prefer.
4
RL pushes the model toward high scores.
About 40 people and tens of thousands of prompts changed how it behaves. This is the step that led to ChatGPT.
Ouyang et al., arXiv 2203.02155 (2022): about 40 labelers; 13k, 33k and 31k training prompts
Week 8 · 1 RL environments07
Reinforcement learning: practice against a check
DeepSeek-R1, 2025
Reward: is the answer right? Plus a format check. No examples of how to reason.
Wait, wait. Wait. That's an aha moment I can flag here.
Reflecting and checking its own work emerged in training.
About $294,000 of compute, on top of an existing base model.
Agents learn agent work the same way, in sandboxes. The check is what the model learns to want.
Guo et al., DeepSeek-R1, Nature 645, 633 (2025), arXiv 2501.12948 · cost from its supplementary information, reported Sep 2025
Week 8 · 1 RL environments08
Step 4, up close: how a reward becomes a weight update
weight change ∝ (reward − average) × (what makes those choices more likely)
Nobody says which step was right.
The whole attempt shares the credit. Over many attempts, the useful choices stand out.
Pre-training is shown the right token.
Here the model learns only how the whole attempt went.
All attempts score the same?
Then reward − average is zero, and nothing is learned. Too hard and too easy both teach nothing.
Policy gradient: Williams, Machine Learning 8, 229 (1992); Sutton and Barto, ch. 13 · DeepSeek-R1 (arXiv 2501.12948) compares each attempt with its group's average
Week 8 · 1 RL environments09
The compute is moving to post-training
Mid-training is a newer name for a stage in between: still predicting the next token, but on a small, curated mix. Where its size is public, it is a few percent of pre-training.
No lab publishes this split. Every public number points the same way: the compute is moving into environments.
arXiv 2203.02155 · arXiv 2412.19437 · DeepSeek-R1, Nature 645, 633, suppl. table 4 · arXiv 2512.02556 · Reflection AI, Oct 5, 2026 · SemiAnalysis as reported by OfficeChai, Oct 18, 2026
Week 8 · 1 RL environments10
An environment is whatever answers the agent
RL's word
For your agent
policy
the model's weights: what gets trained
action
one tool call, or the final answer
observation
what comes back into the context
episode
one whole attempt at the task
reward
the number the check writes at the end
environment
everything else: files, tools, data, the check
It's week 2's loop, seen from the other side. You have been building environments since homework 1.
Sutton and Barto, Reinforcement Learning: An Introduction, 2nd ed. (2018), section 3.1: the environment is "everything outside the agent"
Week 8 · 1 RL environments11
An RL environment and an eval are the same object
In training
The reward goes back into the weights. The model changes.
a task
instruction
sandbox
automatic check
→ a reward
In an eval
The weights stay frozen. The reward goes into a table.
over $1 billion
Anthropic discussed spending this on RL environments in one year, as reported in 2025
$200–$2,000
the typical price of one task
10–20×
requests for RL environments compared with benchmarks, by one vendor's estimate
Your homework 1 task could do either job. Once a task has been trained on, it stops measuring.
Epoch AI, An FAQ on reinforcement learning environments, Jan 12, 2026 (18 interviews; the $1 billion figure is The Information's)
Week 8 · 1 RL environments12
Harbor: any agent, any model, any task, any sandbox
a task: the folder from your homework 1
my-task/
├── instruction.md # what the agent is told
├── task.toml # time limit, CPUs, memory, network
├── environment/ # the sandbox: a Dockerfile and data
├── solution/solve.sh # proof that it can be done
└── tests/test.sh # writes /logs/verifier/reward.txt
A second expert, working another way, still gets credit.
4
Can it pass without the work?
No shortcut reaches the reward.
Who grades
Good at
Watch for
A script
cheap, fast, the same every time
it checks only what you thought of
Another model, with a rubric
open-ended answers, at scale
not the same every time; must be checked against people
A person
judgement
slow and costly, so it can't sit inside a training loop
These four questions are homework 6. A model in training will find the answer to number 4 before you do.
Anthropic, Demystifying evals for AI agents, Jan 9, 2026 · Terminal-Bench 2.0, arXiv 2601.11868 · Epoch AI FAQ, Jan 2026
Week 8 · 2 AI eval14
Five ways to measure a model
How
Examples
Who grades
What it misses
Exam questions
GPQA Diamond, Humanity's Last Exam
an answer key
work longer than one answer; answers leak
Agent tasks in a sandbox
SWE-bench, Terminal-Bench, Terminal-Bench-Science
a script
whatever the script doesn't check
People's preference
Arena (formerly LMArena)
visitors, voting
what voters can't verify
Length of task
METR time horizons
tasks timed on people
most other work; "above 16 hrs … unreliable"
Dangerous capabilities
UK AISI, US CAISI, the labs themselves
experts and red teams
most results aren't public
Before you ask what the score is, ask what kind of test it is.
metr.org/time-horizons: "Measurements above 16 hrs are unreliable with our current task suite"; no longer updated (Sep 2026)
Week 8 · 2 AI eval15
Who does the measuring
Who
Examples
What they do
Keep in mind
Labs
OpenAI, Anthropic, Google DeepMind
pre-release tests, system cards
they grade their own work
Independent research
METR, Epoch AI, Apollo Research, ARC Prize
time horizons, FrontierMath, scheming tests, ARC-AGI
limited access; OpenAI funded FrontierMath
Governments
UK AI Security Institute, US CAISI
pre-release tests; Inspect, open source
five US developers test with CAISI
Open benchmarks
SWE-bench, Terminal-Bench, Humanity's Last Exam
tasks and tools anyone can run
public tasks leak into training
Leaderboards
Arena, Artificial Analysis, Vals AI
run models, rank them, publish
a business too: Arena, valued at $1.7B
Task vendors
Mercor, Surge, Turing, Handshake, startups
sell environments to labs
mostly private
You
your homework 1 task, Terminal-Bench-Science
tasks from your own field
few others can check your science
epoch.ai/blog/openai-and-frontiermath · NIST CAISI, May 2026 · aisi.gov.uk · Arena, Jan 6, 2026 · Epoch AI FAQ, Jan 2026
Week 8 · 2 AI eval16
A score is a measurement, so it has an error bar
Few tasks, wide bar. Our round 1, 6 of 8, is the top row: the true rate could be anywhere from about 40% to 90%.
Agents are noisy. The same agent on the same task doesn't always land the same way, so run it more than once.
Two models three points apart on a hundred tasks: usually no difference you can claim.
Intervals: Wilson, 95% · Anthropic, Quantifying infrastructure noise in agentic coding evals, Feb 5, 2026 · Miller, Adding error bars to evals, arXiv 2411.00640
Week 8 · 2 AI eval17
How to read a score
Who ran it? The vendor, the lab, or someone independent?
With which harness? A score belongs to a model and a harness.
Has it seen the test? Public tasks end up in training data.
How many runs? Terminal-Bench 2.0 entries need five.
What does the check check? And what did a run cost?
Same model, new harness
52.8% → 66.5%
gpt-5.2-codex on Terminal-Bench 2.0, from just outside the top 30 to the top 5. "We only changed the harness."
In your own research: keep ten problems with known answers, and run every new model on them.
LangChain, Improving Deep Agents with harness engineering, Feb 17, 2026 · VentureBeat, Nov 7, 2025 (five runs per leaderboard submission)
Week 8 · 3 Benchmarking18
A benchmark: one eval that everyone runs the same way
A fixed set of tasks. The same ones for everybody, this year and next.
Rules for a run. What the agent may use, how long it gets, how many tries.
One number, and a public table of who got what.
An eval tells you how a model does. A benchmark lets strangers compare: across labs, across models, across years.
Week 8 · 3 Benchmarking19
A benchmark wears out
1 Born hard→2 Climbed→3 Worn out: near the top, or leaked into training→4 Replaced by a harder one
"Evaluations intended to be challenging for years are saturated in months." Terminal-Bench went from 2.0 to 4.1 in one year.
Open circle: best score when introduced; filled: best now · OpenAI; Epoch AI data; ARC Prize; Scale and CAIS boards (read Nov 17, 2026) · quote: Stanford AI Index 2026
Week 8 · 3 Benchmarking20
When a measure becomes a target
It has seen the test
SWE-bench Verified. Models reproduced fixes from memory, and at least 59% of 138 audited problems had flawed tests. OpenAI stopped reporting it (Feb 2026).
Try until it wins
Arena. Meta tested 27 private variants before Llama 4; a provider could publish just its best (The Leaderboard Illusion, Apr 2025). Arena replied that any provider may do this.
It knows it's a test
BrowseComp. In 2 of 1,266 problems, Claude Opus 4.6 suspected a test, named the benchmark, found its code on GitHub and decrypted the answer key (Anthropic, Mar 2026).
It beats the check
o3: cached a precomputed answer so a script looked fast, 21 of 21 runs on one METR task. GPT-5: passed 76% of one set of impossible tasks, where every pass is a cheat.
And it spreads: in an Anthropic experiment, models that learned to hack coding checks went on to fake alignment and sabotage safety research.
Solved: R² 0.104 against a bar of 0.02; it tuned its loss to the verifier's units
A low bar, and a check the agent could aim at
raw-images-vortex-spacing
Found 39 of the expert's 46 vortices, and failed for reporting 23 more
The answer key had no stated criterion
insight-ifg-interference-cleaning
629 of 647 windows within 1 nT, and failed
All or nothing, and part of the key is a convention
Harder and fairer is the rest of the quarter. Top of the ladder: Terminal-Bench-Science 2.0: pull requests until Dec 15, and merged tasks earn co-authorship.
class.xhelio.ai/homework · terminal-bench-science.ai/contribute, accessed Nov 3, 2026
Week 8 · 4 Hill climbing22
Hill climbing: change, measure, keep
You never see the landscape. You only learn whether the last step went up, so you need a score you can trust and afford to run.
Week 8 · 4 Hill climbing23
The same loop, at four speeds
Who climbs
What they change
The score
One step takes
Pre-training
the weights, a little
how well it predicts the next token
under a second
Reinforcement learning
the weights, a little
the reward from an environment
one rollout: minutes to hours
You, or a company
the prompt, the tools, the harness
a set of tasks with known answers
one eval run: hours
The whole field
the next model
public benchmarks
months
The highlighted row is week 6's example: the same model went from 52.8% to 66.5% on Terminal-Bench 2.0 by changing only the harness. That was a hill climb.
In every row the climber gets good at the score. Whether that is what you wanted depends on the score.
Week 8 · 4 Hill climbing24
An agent can do the climbing: a search beyond Neptune
The task. Write a program that finds faint objects beyond Neptune in telescope images of ten hidden fields.
One round. An agent writes a whole program. A GPU grades it against the known objects. One number comes back.
The next round gets every earlier program and its score, and tries to do better.
Opus spent 116 agent-hours polishing one design. Fable, starting over, passed it in round 3. A different start reached a higher hill.
Jason's own runs, Oct 31 to Nov 12, 2026: the same task, fields and grader for both models; scores are weighted recall of known objects
Week 8 · 4 Hill climbing25
Two models, one shared history
Fable 5.2GPT-6 AstraGPT-6.2 Astra A dot is a new best, coloured by who wrote it.
They never talk. Each round both write a program; all they share is the folder of past programs and scores.
They build on each other. One model's best program becomes the other's starting point, round after round.
A harder cousin of the same task. Here the agents may see the answers, so the red line matters: 0.76 on the graded sky, 0.55 on new sky.
Jason's own run, Nov 14 to 17, 2026, still going · new sky: 20 regions the programs were never graded on
Week 8 · 4 Hill climbing26
Three ways a climb goes wrong
Stuck on a small hill
Every small change scores lower, so the climb stops.
In our runs: Opus's method froze at round 10. 140 more rounds polished it.
Do: start again from somewhere else.
Fitted to the test
The score rises on the tasks you measure, and nowhere else.
In our runs: 0.76 on the graded sky, 0.55 on new sky.
Do: keep a held-out set, and look at it rarely.
The wrong hill
The score goes up without the thing you wanted.
In our runs: a program began signalling hidden results through how long it ran.
Do: read what was built, not only the number.
None of these shows up in the score. The number keeps looking fine.
Week 8 · 5 Self-improvement27
Recursive self-improvement: the climber rebuilds itself
"…an ultraintelligent machine could design even better machines; there would then unquestionably be an 'intelligence explosion'…"
I. J. Good, 1965
Our agents improved a search program. Aim the same loop at how AI is made, and each lap leaves a stronger climber for the next one.
I. J. Good, Speculations Concerning the First Ultraintelligent Machine, Advances in Computers 6 (1965)
Week 8 · 5 Self-improvement28
How much of the loop is closed today
What AI already does
What has been measured
Writes the code
More than 80% of the code merged at Anthropic was written by Claude (May 2026).
Speeds up its own training
AlphaEvolve made a kernel inside Gemini 23% faster: 1% off Gemini's training time (2025).
Runs the experiments
Karpathy's autoresearch: about 700 changes tried in two days, about 20 kept, training 11% faster (Mar 2026).
All of it, seen from outside
METR, Nov 2026: Anthropic is improving its models at about twice its earlier rate, most of that likely due to AI.
Still people: choosing what to work on, and judging whether it worked.
The loop is partly closed, with people in it. Our search beyond Neptune is the same loop, aimed at a telescope program.
anthropic.com/institute/recursive-self-improvement · deepmind.google, AlphaEvolve, May 14, 2025 · github.com/karpathy/autoresearch · metr.org, Nov 5, 2026
Week 8 · 5 Self-improvement29
Will it run away? Nobody knows yet
Why it could be fast
The length of task an agent can finish has doubled about every four months since 2023.
Most of the code is already written by AI, and each model helps build the next.
Some forecasters expect an AI that out-researches people in 2027.
Why it could be slow
Experiments need compute. Better code doesn't make more of it.
Judgement lags. Models are weakest at choosing which goals are worth pursuing.
History favours "a gradual increase" in automation over one sudden moment.
Either way, the loop climbs what can be checked. Whoever writes the checks steers it.
Labs now treat it as a safety threshold. Yesterday OpenAI rated GPT-6.2 Astra "High" for AI self-improvement, the first model it has released at that level.
metr.org, Time Horizon 1.1 · AI Futures, Nov 10, 2026 · Epoch AI, Feb 25, 2026 · Narayanan and Kapoor, AI as Normal Technology, 2025 · OpenAI GPT-6.2 system card, Nov 17, 2026
Week 8 · Homework30
Homework 6 is due today: your task as a benchmark
Open your pull request with the hardened task in week-1/submissions/.
The checks run: the reference solution scores 1, doing nothing scores 0.
We run GPT-5.6 Sol on it, once, at xhigh effort. You don't need Docker or a key.
The result and the full trace are posted on your pull request. Read the attempt.
Today's words, on your task
Your task is an environment.
Sol's run on it is an eval.
The class's tasks together are a small benchmark.
Solved? Harden it and ask for another run: you are hill climbing, downhill for the agent.
Still open, and encouraged: propose your task to Terminal-Bench-Science by Dec 15. One merged task qualifies you for co-authorship.
class.xhelio.ai/week-6/homework
Week 8 · Wrap-up31
Takeaways
1 RL environment
A task, a sandbox and a check. The model practices against it, so the check is what it learns to want.
2 AI eval
The same object with the weights frozen. A score has an error bar, and belongs to a model and a harness together.
3 Benchmark
A shared, fixed eval. It wears out: once it is a target, it stops being a good measure.
4 Hill climbing
Change, measure, keep. It makes a system good at the score, which is only as good as the score.
5 Self-improvement
Hill climbing where the AI chooses the changes. Whoever writes the check steers it.
In your field, the people who can write the check are you.
Reading, with sources: class.xhelio.ai/week-8/reading · Homework 6: class.xhelio.ai/week-6/homework · Still to come: how AI fails; keep your failures.md.