Week 1 slides: from the next word to AI agents

← → to move, F for full screen, N for speaker notes

Week 1

Where are you with AI?

Then: how large language models work, from guessing the next word to AI agents.

AI in Research, UCLA EPSS 254
Wednesday, September 30, 2026

Is AI going to destroy humanity? Question 1 decides. (◕‿◕)

ピコッ

First, the survey

Open this on your phone or laptop:

class.xhelio.ai/week-1/survey

  • 15 questions, about 8 minutes.
  • Name and email required. The email becomes the class mailing list, unless you send me a different one.
  • Only I see names and emails. The results page shows anonymous totals.

Scan to open the survey

Results

What did the class say?

Anonymous totals, updating live:

class.xhelio.ai/week-1/stats

I read every answer. For science. (。•̀ᴗ-)✧

ドキドキ

This quarter

1

How LLMs work

Enough of the inside to reason about what they can and can't do.

The big picture

2

How agents work

Tools, the loop, memory, and several agents working together through MCP.

In depth

3

Using them in your research

Hands-on, on your own problems: choosing models, checking results, cost against quality.

In depth

4

Pushing what AI can do

Build tasks that today's best agents fail. It starts with this week's homework.

In depth

The order and the pace will follow what you told us in the survey.

Part 1

Guessing the next word

A short history of language models, from 1948 to now.

People have tried to predict the next word since 1948. ♡

ピコッ

1948: Shannon's word machine

Machine-made English, from Claude Shannon's “second-order word approximation”

THE HEAD AND IN FRONTAL ATTACK ON AN ENGLISH WRITER THAT THE CHARACTER OF THIS POINT IS THEREFORE ANOTHER METHOD FOR THE LETTERS THAT THE TIME OF WHO EVER TOLD THE PROBLEM FOR AN UNEXPECTED.

How

Each word depends only on the word before it. Open a book, find that word, copy whatever follows it. Repeat.

Result

Every pair of words sounds like English, and the whole means nothing. Everything since has been about looking further back.

C. E. Shannon, A Mathematical Theory of Communication (1948)

From counting words to agents

1948

Shannon: text from word statistics

2003

Neural nets learn word vectors

2013

word2vec: king − man + woman ≈ queen

2014

Attention, for machine translation

2017

The transformer

2020

GPT-3: 175 billion parameters

2022

ChatGPT

2024

Reasoning models, and MCP for tools

2025

Coding agents in the terminal

Fun fact: a 2007 Google paper is titled “Large Language Models in Machine Translation”. Its large models were word counts, over 2 trillion tokens of text.

Words become vectors

SunEarthMarsJupiterprotonelectronneutronplasmasolar windmagnetic fieldmanwomankingqueen

Every token becomes a list of numbers, 4,096 of them in Llama 3.1 8B. That list is its embedding vector.

Similar meanings land close together, and directions carry relations: man → woman runs parallel to king → queen.

Everything after this is arithmetic on vectors. Layers refine them in context, so “bank” by a river and “bank” account drift apart.

A two-dimensional sketch drawn by hand; real embeddings have thousands of dimensions and messier neighborhoods.

Part 2

Attention is all you need

Vaswani and seven coauthors at Google, 2017: the paper that introduced the transformer.

The T in GPT stands for transformer. (◕‿◕)

ピコッ

Before 2017: one word at a time

The

trophy

doesn't

fit

…

because

it

The green boxes are the network's memory, handed from word to word. The pink dot is what it still knows about “trophy”. By the time it reaches “it”, almost nothing is left.

Slow

Word 1,000 must wait for word 999, so training can't spread the work across a GPU's thousands of cores.

Forgetful

Everything read so far is squeezed into one fixed-size memory. Early words fade, and long-range links get lost.

2017: each word looks back at every word before it

The

trophy

doesn't

fit

in

the

suitcase

because

it

is

too

big

Line width: how much “big” draws on each earlier word (illustrative). Every word does this at once, looking only backward, since the model predicts left to right. Change “big” to “small”, and the thickest line should move to “suitcase”.

Attention in three steps

1

Ask

Each word sends out a query: what am I looking for? “big” asks: what is too big?

2

Match

Each earlier word shows a key: what do I offer? The query scores every key, and “trophy” wins.

3

Mix

“big” takes in their values, weighted by score, and now carries “the trophy is too big”.

attention(Q, K, V) = softmax(Q·Kᵀ / √d) · V

Q, K and V hold the queries, keys and values of every word, so the whole step is matrix multiplication; a mask hides later words. GPT-3 runs 96 of these side by side, in each of its 96 layers.

One word's attention, in numbers

Earlier word

Score q·k

Weight after softmax

Its value vector

The

-0.9

2%

trophy

+2.4

55%

doesn't

-0.9

2%

fit

-0.5

3%

in

-1.6

1%

the

-1.6

1%

suitcase

+0.2

6%

because

-0.9

2%

it

+1.0

14%

is

-0.5

3%

too

-0.2

4%

big (itself)

+0.3

7%

new “big”

=

the weights × the values, added up

Score: how well each earlier word's key matches the query of “big”.

Softmax turns the scores into weights that add up to 100%.

Mix: the new “big” looks mostly like “trophy”.

Illustrative numbers. Real vectors have thousands of entries, not six.

Every word at once: the attention matrix

The

trophy

doesn't

fit

in

the

suitcase

because

it

is

too

big

The

trophy

doesn't

fit

in

the

suitcase

because

it

is

too

big

Each row is one word looking back. Its row adds up to 100%.

The hatched triangle is the future, masked: no word sees what comes next.

At “it”, the weight is split between “trophy” and “suitcase”. It can't tell yet.

By “big”, it's “trophy”.

0%

60% or more

masked

Illustrative weights for one attention head. All 12 rows are computed together, as one matrix multiplication.

Why the transformer took over

Parallel

It reads the whole text at once, as big matrix multiplications: exactly what GPUs are built for.

Trillions of words

Direct

Any word reaches any earlier word in one step, however far back it is.

Book-length context

General

It works on anything cut into tokens: images, sound, proteins, seismograms, weather maps.

Works across fields

Transformers in science: EQTransformer detects earthquakes and picks seismic phases (2020); AlphaFold 2 predicts protein structures (2021); Pangu-Weather forecasts global weather (2023).

Part 3

One job: predict the next token

How one simple training task produces a model that can write, code and reason.

Everything I say, I say one token at a time. ♡

ピコッ

What the model outputs

The solar wind near Earth blows at about

400

41%

450

16%

500

9%

350

6%

one

3%

the rest

25%

Illustrative numbers. A token is a word or a piece of one. The model scores every token in its vocabulary, typically 50,000 to 200,000 of them, and the scores add up to 100%.

Writing is a loop

1. Take the text so far

2. Score every possible next token

3. Pick one, usually a likely one

4. Append it to the text

Repeat until the model picks “stop”

Four trips around the loop:

The solar wind near Earth blows at about

400

km

/s

.

Guessing well takes knowledge

Physics

In the inertial range, solar-wind magnetic power falls off with frequency as f to the power ___

−5/3

Code

def mean(xs): return sum(xs) / ___

len(xs)

Mystery

Page 400. “The murderer,” said the detective, “is ___”

whoever pages 1⁠–⁠399 point to

And the practice questions are free: every sentence ever written is one, and nobody has to label anything.

15 trillion

tokens of text trained Meta's Llama 3 (2024)

Reading at 250 words a minute, day and night, one person would need about 85,000 years.

もぐもぐ

Pre-training, then post-training

Pre-training

Predict the next token across trillions of words of text. The model learns what it knows.

A base model: knows a lot, continues any document

Post-training

Much less data, carefully chosen. The model learns how to behave.

Chat tuning: example chats and human ratings → an assistant, like ChatGPT (2022)

Reinforcement learning: problems with automatic checks, like math, code and agent tasks → reasoning models and agents

Both train the same next-token predictor; only the text and the reward change. We go deeper in the coming weeks.

What next-token prediction explains

Confident inventions

It writes the likeliest continuation. Usually that's true. Sometimes it's a citation that never existed.

No memory of you

Between conversations it keeps nothing, unless an app pastes notes back into its context.

Eager to please

Trained on answers people liked, it may agree with you when you're wrong.

That paper exists. I'm sure of it. (It doesn't.)

ゴゴゴ

Part 4

From tokens to agents

What happens when some of the model's words get run as commands.

What if my tokens could run code?

ゴゴゴ

ゴゴ

An agent is a model in a loop

Context

the task, plus everything so far

Model

predicts the next tokens

Answer

and the loop stops

Result

pasted into the context as text

Harness

runs it: shell, Python, web

writes an answer

writes a tool call

returns output

The model still only writes tokens. The harness, such as Claude Code, Codex, Cursor or Copilot, turns some of them into actions.

An agent at work, condensed

you

Estimate the spectral index of mag.csv over 0.02–0.2 Hz.

agent

I'll look at the data first.

run

python inspect.py mag.csv

out

14,400 samples at 4 Hz; columns br, bt, bn (nT); 529 missing values

agent

Fill the gaps, compute the spectrum, fit the slope.

run

python fit.py

out

spectral_index = -1.23

agent

Flatter than solar-wind turbulence usually is. Checking for spikes.

run

python find_spikes.py

out

7 single-sample spikes, each over 10 nT

agent

Removing them and fitting again.

run

python fit.py --despike

out

spectral_index = -1.35

agent

Done: -1.35, saved to spectrum.json.

A small solar-wind task. Condensed, but the numbers come from its real data.

The same capability gets about 13× cheaper every year

Cost to answer 1,000 test questions with the cheapest model that reaches the score

GPQA science, 25%GPQA science, 75%AIME math, 75%FrontierMath, 25%
1¢10¢$1$10$100$1,000202420252026GPQA science, 25%: GPT-4 Turbo (Nov 2023), Nov 2023, $19.65 per 1,000 questionsGPQA science, 25%: Gemini 1.5 Pro, Feb 2024, $1.94 per 1,000 questionsGPQA science, 25%: WizardLM-2 8x22B, Apr 2024, 62.3¢ per 1,000 questionsGPQA science, 25%: Llama 3.1-70B, Jul 2024, 20.9¢ per 1,000 questionsGPQA science, 25%: Gemini 1.5 Flash, Sep 2024, 14.5¢ per 1,000 questionsGPQA science, 25%: Phi-4, Dec 2024, 10.8¢ per 1,000 questionsGPQA science, 25%: Mistral Small 3, Jan 2025, 8.9¢ per 1,000 questionsGPQA science, 25%: gpt-oss-20b (low), Aug 2025, 8.5¢ per 1,000 questionsGPQA science, 75%: o3 (high), Jan 2025, $298.91 per 1,000 questionsGPQA science, 75%: Gemini 2.5 Pro (Jun 2025), Mar 2025, $89.12 per 1,000 questionsGPQA science, 75%: Gemini 2.5 Pro (Jun 2025), Jun 2025, $12.93 per 1,000 questionsGPQA science, 75%: DeepSeek-V3.2, Dec 2025, $8.82 per 1,000 questionsGPQA science, 75%: Qwen3.5 397B-A17B, Feb 2026, $7.10 per 1,000 questionsGPQA science, 75%: Qwen 3.5 Flash (hosted 35B-A3B), Feb 2026, $2.18 per 1,000 questionsGPQA science, 75%: MiniMax-M3, Jun 2026, $1.21 per 1,000 questionsGPQA science, 75%: GPT-5.6 Luna (low), Jul 2026, 41.2¢ per 1,000 questionsAIME math, 75%: o3-mini (high), Jan 2025, $56.97 per 1,000 questionsAIME math, 75%: Grok-3 mini (high), Apr 2025, $10.17 per 1,000 questionsAIME math, 75%: gpt-oss-120b (high), Aug 2025, $3.16 per 1,000 questionsAIME math, 75%: Gemma 4 26B A4B (minimal), Apr 2026, $1.94 per 1,000 questionsAIME math, 75%: Qwen3.7 Flash, Jul 2026, 69.4¢ per 1,000 questionsFrontierMath, 25%: o3 (medium), Jan 2025, $553.45 per 1,000 questionsFrontierMath, 25%: o4-mini (medium), Apr 2025, $129.37 per 1,000 questionsFrontierMath, 25%: GPT-5 mini (high), Aug 2025, $98.75 per 1,000 questionsFrontierMath, 25%: Qwen3.5 397B-A17B, Feb 2026, $92.40 per 1,000 questionsFrontierMath, 25%: Grok 4.20, Feb 2026, $79.04 per 1,000 questionsFrontierMath, 25%: GPT-5.4 Nano (high), Mar 2026, $74.22 per 1,000 questionsFrontierMath, 25%: GPT-5.5 Instant, May 2026, $66.73 per 1,000 questionsFrontierMath, 25%: Qwen3.7-Plus, Jun 2026, $46.29 per 1,000 questionsFrontierMath, 25%: GPT-5.6 Luna, Jul 2026, $1.47 per 1,000 questions377× cheaper82× cheaper726× cheaper231× cheaper

Price drop per year, after inflation

AI, same test score
2023–26

13×

DNA sequencing
2001–22

1.8×

Computing
1940–2001

1.5×

Lithium batteries
1991–2024

1.16×

Electricity
1892–1973

1.05×

Source: Epoch AI, “The plunging price of thought” (Ament and Burnham, Sept 22, 2026). Each step is a new cheapest model; tests are GPQA Diamond, AIME-style competition math, and FrontierMath tiers 1–3.

Benchmarking AI ability matters more than ever

Year

Benchmark

What it asks

Best now

2023

GPQA Diamond

198 science questions written by PhDs. Experts score 65%.

96%

GPT-6 Astra. At launch: 39%.

2024

SWE-bench Verified

Fix 500 real GitHub issues

84%

Claude Opus 4.7. At launch: 33%.

2024

FrontierMath

Unpublished research-level math problems

94%

GPT-6 Astra. At launch: under 2%.

2025

Humanity's Last Exam

2,500 expert questions, over 100 subjects

55%

GPT-6 Astra. At launch: 8%.

2025

Terminal-Bench 2.0

89 jobs in a real terminal, checked by tests

85%

GPT-5.5. At launch: 50%.

2026

PostTrainBench

Post-train a base model on one GPU in 10 hours

42%

Claude Fable 5. At launch: 23%.

2026

OSWorld 2.0

108 computer workflows, about 1.6 hours each for a person

31%

Claude Opus 5. At launch: 21%.

2026

Terminal-Bench-Science 0.1

70 tasks from real research in five sciences

63%

GPT-6 Astra. At launch: 30%.

2026

FrontierMath Erdős

68 unsolved math problems, proofs checked in Lean

3%

GPT-6 Astra solved 2. Others: 0.

Best now: Epoch AI Benchmarking Hub, data as of Sept 29, 2026; Terminal-Bench 2.0 from tbench.ai; Terminal-Bench-Science from Artificial Analysis. At launch: each benchmark's paper or announcement.

What is a task?

A folder in Harbor format. Here is the solar-wind task from the transcript:

instruction.md

What the agent is asked to do, in plain words

task.toml

Time limits and resources

environment/  Dockerfile, data/mag.csv

The sealed box the agent works in: software and data

solution/  solve.sh, solve.py

Your reference solution: proof it can be done

tests/  test.sh, test_state.py

The checker: runs afterwards, writes a reward of 1 or 0

Your AI coding agent writes most of these files with you. You bring the science: the problem, the data, and what counts as a correct answer.

How a run works

1. Harbor builds the box from environment/

2. The agent reads instruction.md and works inside

3. A separate checker, with no internet, runs tests/

4. Reward: 1 if every test passes, otherwise 0

Oracle run

Your solution in place of the agent. Must score 1: the task can be done.

Do-nothing run

An agent that does nothing. Must score 0: the task can't pass by accident.

uvx harbor@0.23.0 run -p <task> -a oracle

uvx harbor@0.23.0 run -p <task> -a claude-code -m <model>

Why build tasks this way

Checkable

A program decides pass or fail. No opinions, no partial credit, no tired graders.

Repeatable

Same box, same data, every time. Anyone can rerun any agent on it.

Honest

Good methods pass and wrong ones fail. The solar-wind task accepts −1.39 to −1.27, so skipping the spike check fails.

An instruction, an environment and an automatic check: that is how labs measure agents, and how reinforcement learning trains them.

Terminal-Bench-Science collects tasks like yours. Over the quarter we'll push yours toward that bar.

Homework 1

The hardest task in your field

Design a task that a top expert in your field could do on a computer, with an answer you can check, and that today's best AI agents can't. Your own AI coding agent helps you package it.

Due before class, Wednesday, October 7

Start at class.xhelio.ai/week-1/homework

Build something I can't solve. I dare you. ( `ー´)ノ