Week 2 slides: how an AI agent works
← → to move, F for full screen, N for speaker notes
AI in Research · Week 2 · October 7, 2026
How an AI agent works
From one API call to an agent loop, and what every token costs along the way.
class.xhelio.ai/week-2
October 6, 2026: mathematician as an occupation is officially dead
722
math manuscripts in 372 result families, from one unreleased OpenAI model
- About 4,000 open problems posed
- About 3 hours of ChatGPT Pro thinking per result, on average
- Includes the Mahler conjectures and the isomorphism of free group factors
- Many, not all, checked by computer in Lean
openai.com/index/sharing-ai-progress-in-mathematics · github.com/openai/math (README: "Some of the unformalized results could have issues.")
Last week: an agent is a model in a loop
Today we open each box in that loop, using one real API, the OpenAI Responses API. Other providers' APIs are dialects of the same thing.
Part 1
One call
What an API, HTTP and JSON are, then one real request and its reply.
Part 2
Inside the call
Tool calls as tokens, strict JSON, reasoning, the KV cache, and token prices.
Part 3
The loop
Chaining calls into an agent, and keeping its context small.
Part 4
Using it
Subscriptions against the API, practical basics, research hygiene.
Two ways to use a language model
Most of you have used the first. Agents are built on the second.
A chat app
You type, you read
Good for asking, drafting, exploring.
Paid by a monthly subscription.
The API
A program asks for you
Good for automation: a thousand papers, a nightly job, an agent.
Paid per token. Codex and Claude Code are programs that use the API.
What is an API?
An API (application programming interface) is a fixed list of requests one program can send another, and the reply it promises back.
You already use one
Fetching spacecraft data
A script asks a data archive such as CDAWeb for this dataset, this time range. The server sends the numbers back.
You never see the archive's database. You only see the request you may send and the reply you get.
Today
Asking a language model
Your script sends this text, these settings to a model provider. The server runs the model and sends text back.
You never see the GPUs or the weights. Same idea, different server.
Programs talk over HTTP, like your browser does
Your browser sends a request to a URL and gets a response. A program does exactly the same.
Request: your program to the server
Response: the server back to you
The URL says which service. POST means "here is data, do something with it".
The key proves who you are and who pays. Treat it like a password.
The status code: 200 worked, 401 bad key, 429 slow down.
What is JSON?
JSON is plain text that holds structured data. Humans can read it, every programming language can parse it, and nearly every API speaks it.
| You see | It means |
|---|---|
{ } | An object: named fields |
"key": value | One field: a name and its value |
"text" | A string, always in double quotes |
0.17 | A number, no quotes |
true false null | Yes, no, nothing |
[ ] | A list, in order |
"plasma": { } | Objects nest inside objects |
The example's values are made up; the four instrument suites are real.
Reading JSON, and describing it
Reading it in Python
Every API reply is read this way: resp["output"][0]["type"].
A JSON Schema
JSON that describes what valid JSON looks like: which keys, which types. Tools are defined with one.
Everything is counted in tokens
From week 1: a token is a word or a piece of one. About 1,000 tokens make 750 English words.
What you send
Input tokens. Your instructions, the conversation, tool definitions, files.
What comes back
Output tokens. The answer, tool calls, and any hidden reasoning.
The limit
The context window: the most tokens one call can hold, in and out together.
The bill
Prices are quoted in dollars per million tokens, and input and output are priced differently.
Rough scale: a 7,000-word paper is about 9,000 tokens. A million tokens is roughly 750,000 words.
Putting it together: your first API call
One command sends JSON to OpenAI and gets JSON back. The model remembers nothing between calls, so a conversation means resending it every time.
You send
You get back
The reply is trimmed. Every chat app and every agent is built from calls like this one.
The model never sees JSON
The server flattens your request into one stream of tokens with role markers. The model just continues it, as in week 1.
Roles are text markers. The model is trained to weight them, not forced to.
Tool schemas become text in the developer message.
Hidden reasoning is the analysis channel; the answer is final.
OpenAI's open-weight gpt-oss "harmony" format, shortened from OpenAI's own example (developers.openai.com/cookbook/articles/openai-harmony).
The request
- What to say
model, instructions, input - What it may do
tools, tool_choice, text.format - How hard, how long
reasoning.effort, max_output_tokens - Cost and memory
service_tier, prompt caching, store, previous_response_id
The response
- output is a list of typed items: reasoning, message, function_call.
- arguments is a string of JSON. Your program parses it.
- status is incomplete if the model ran out of max_output_tokens, sometimes while still thinking.
- usage is the bill. In OpenAI's own example, 832 of 1,035 output tokens were hidden reasoning.
Response to the weather request, from OpenAI's API reference (developers.openai.com/api/reference/resources/responses/methods/create).
Two kinds of output
Every turn ends one of two ways. Your program branches on which one came back.
type: "message"
Text for a person
Plain words in output_text.
Nothing to run, so the loop stops.
type: "function_call"
A request to your program
A tool name, JSON arguments, and a call_id.
You run it, send back a function_call_output with the same call_id, and call again.
tool_choice steers it: "auto" lets the model decide, "required" forces a call, "none" forbids one. A third shape, text forced into a JSON schema (text.format), uses the same machinery as tool calls.
Other APIs are dialects of the same idea
Same request, different field names. Read any provider's docs with this table.
| Dialect | System prompt | Conversation | Text comes back in | A tool call is |
|---|---|---|---|---|
| OpenAI Responses (this class) | instructions | input items | output[] message | function_call item |
| OpenAI Chat Completions | a system message | messages | choices[0].message | tool_calls[] |
| Anthropic Messages | system | messages | content[] text block | tool_use block |
| Google Gemini | systemInstruction | contents, parts | candidates[0] parts | functionCall part |
Chat Completions became a de facto standard: local servers such as Ollama and vLLM speak it, so the same code can run an open model on a lab GPU by changing one URL.
A tool call is just tokens
The model is fine-tuned to write calls in a special format. The server watches for the <|call|> token, stops generating, and hands you the call as JSON.
Without strict mode
Usually right. Sometimes invalid JSON or a missing field, because it is still just sampled text.
With strict: true
The server forces the JSON to match your schema, token by token. Next slide.
The last layer: a score for every token
After all the attention layers, the model multiplies its final vector by one big matrix. Out comes one number per vocabulary token, called a logit: higher means "more likely next".
"The solar wind near Earth blows at about ___". Illustrative scores, chosen to match week 1's slide. Scores can be any real number, negative too.
Softmax turns scores into probabilities
| Token | Score z | ez | Probability |
|---|---|---|---|
| 400 | 5.11 | 165 | 41% |
| 450 | 4.17 | 65 | 16% |
| 500 | 3.59 | 36 | 9% |
| 350 | 3.19 | 24 | 6% |
| one | 2.49 | 12 | 3% |
| every other token | … | 101 together | 25% |
| Sum | 403 | 100% |
- Exponentiate each score: every result is positive, and bigger scores grow much faster.
- Divide by the total: the shares add up to 100%.
- Only the differences between scores matter. Add 10 to every score and nothing changes.
The probabilities are week 1's slide exactly; the scores here are chosen to produce them.
Then pick one: sampling and temperature
The server draws one token at random, weighted by these probabilities, appends it, and runs the whole network again. Temperature divides every score before the softmax.
Temperature 0.5: sharper
400
81%
450
12%
500
4%
350
2%
one
0%
Temperature 1: as trained
400
55%
450
21%
500
12%
350
8%
one
4%
Temperature 2: flatter
400
36%
450
23%
500
17%
350
14%
one
10%
Shares among these five tokens only. Temperature near 0 always picks the top token. As the next slide shows, today's frontier APIs mostly no longer let you set it.
Temperature is fading from frontier APIs
Reasoning models run their own sampling during long hidden thinking, so providers stopped exposing it. The dials you turn now are how long the model thinks and how much it writes.
| Where | Can you set temperature? | What you turn instead |
|---|---|---|
| OpenAI GPT-6, reasoning on | No: remove temperature, top_p, top_logprobs | reasoning.effort, text.verbosity |
| Claude Opus 5.5 | No: the API rejects sampling parameters | effort |
| Open-weight models you run yourself (vLLM, Ollama) | Yes, plus top_p, top_k | gpt-oss recommends temperature=1.0, top_p=1.0 |
Need repeatable output? Temperature 0 never guaranteed it anyway (Part 4). Log, and run several times.
Need a fixed format? Use a JSON schema (text.format or a strict tool), not a low temperature.
Strict mode masks the softmax
The model has written {"location": and the schema says a string must come next. The server sets the score of every forbidden token to minus infinity before the softmax. Since e−∞ = 0, those tokens get probability zero; the rest are renormalized.
Before the mask
Grammar
After the mask
"
0.62
allowed
0.97
Paris
0.21
masked
0.00
null
0.09
masked
0.00
"P
0.02
allowed
0.03
other
0.06
masked
0.00
Illustrative numbers. OpenAI: the schema is converted "into a context-free grammar", and masking "effectively lowers the probability of invalid tokens to 0." The shape is guaranteed; the content is not: "Pairs" would pass.
Reasoning effort: hidden tokens before the answer
832 of 1,035
output tokens in OpenAI's reasoning example were hidden reasoning, billed at the output price.
■ reasoning, 832 ■ visible answer, 203
- Where the skill comes from: reinforcement learning rewards working problems out step by step.
- How the dial works: in gpt-oss it is literally the line Reasoning: high in the system message. Changing reasoning.effort breaks the prompt cache, so it is part of the input there too.
- Levels: none to max; each model accepts a subset.
- Higher is not always better. For extraction, low is cheaper, faster and often as accurate.
developers.openai.com/api/docs/guides/reasoning · gpt-oss harmony format. The closed models' exact mechanism is not published.
Service tier: the same model in a different queue
The weights are identical. You pay for how your request is scheduled on the provider's GPUs.
| Tier | Price | What you get |
|---|---|---|
| Batch (separate API) | 0.5× | Results within 24 hours. Grading 1,000 papers overnight. |
flex | 0.5× | Synchronous but slower; sometimes unavailable |
default | 1× | Standard |
fast (called priority until July 30, 2026) | 2× | Faster, more reliable throughput |
ultrafast | 6× | Highest speed, few models |
gpt-6-astra, relative to Standard. developers.openai.com/api/docs/pricing and the flex, batch and priority-processing guides.
The context window is per call
1,050,000
tokens fit in one gpt-6-astra call: up to 922,000 input and 128,000 output, reasoning included.
- Per call, not per conversation. Every call resends everything, so a long agent run fills it even if no single message is big.
- Bigger costs more. Each call bills the whole context again.
- Bigger can be worse. Accuracy drops as old, irrelevant text piles up ("context rot").
developers.openai.com/api/docs/models/gpt-6-astra
Recap from week 1: queries, keys and values
Each token makes three vectors in every layer. Week 1: "The trophy doesn't fit in the suitcase because it is too big."
Q · query
"What am I looking for?"
"big" asks: what is too big?
K · key
"What do I offer?"
Every earlier token shows a key. "trophy" matches the query best.
V · value
"What I pass on"
"big" takes in the values, weighted by those match scores.
Why this matters today: to write token 101, only token 101 needs a new query. The keys and values of tokens 1 to 100 have not changed, so the server keeps them instead of recomputing. That store is the KV cache.
The KV cache: week 1's K and V, kept around
Prefill
Read the whole prompt at once. All input tokens go through the network in one parallel pass, and each layer stores their keys and values: the KV cache.
Decode
Write one token at a time. Each new token makes a query and attends over all stored keys and values.
Live, OpenRouter median
t = 0.0 s
1st token
Writing
$ in / out
GPT-6.1 Sol flex
11.1 s
25 tok/s
$1 / $5
GPT-6.1 Sol standard
4.7 s
34 tok/s
$2 / $10
GPT-6.1 Sol fast
2.7 s
58 tok/s
$4 / $20
Claude Opus 5.5 standard
4.2 s
64 tok/s
$4 / $20
Claude Opus 5.5 fast
2.5 s
131 tok/s
$8 / $40
Grey: waiting for the first token (prefill, queue, thinking). Orange: decode at the measured speed. fast costs 2× and is about 2× quicker. openrouter.ai, P50 of real traffic, Oct 7 2026.
Prompt caching: the prefix must match exactly
The server keeps the KV cache of your prompt's beginning for a while. The next call that starts with exactly the same tokens skips that work.
Call 1
everything is new: written to cache
Call 2
only the new tail is full price
Call 3
a timestamp at the top changes token 1: nothing hits
■ read from cache ■ new, full price ■ cache broken
Why exactly: token i's key and value depend on every token before it. Stable things go first. On OpenAI caching is automatic from 1,024 tokens; changing tools, output format or reasoning effort also breaks the prefix.
Why tokens cost different amounts
gpt-6-astra, dollars per million tokens. Output costs 5× input and 50× a cache read.
Cache read
$1.00
keys and values already computed; just load them
Input
$10.00
prefill: one parallel pass, the GPU stays busy
Cache write
$12.50
the same prefill, plus memory held for 30+ minutes
Output
$50.00
decode: one token per full pass through the network
Reasoning
$50.00
billed as output: it is output, just hidden
developers.openai.com/api/docs/pricing, standard tier, prompts up to 272K tokens
A cached token went from 1/2 to 1/20 of the price
Price of a cached input token as a fraction of a fresh one (log scale). OpenAI in orange.
OpenAI DevDay and GPT-4.1 posts, pricing snapshots and changelog; anthropic.com/news/prompt-caching; Claude pricing; DeepSeek news 2024-08-02 and pricing.
Cached reads across providers today
Cached-read price as a fraction of the fresh input price. Orange: 1/20 or cheaper.
deepseek-flash · DeepSeek
1/50
$0.006 vs $0.30
Claude Fable 5.1 · Anthropic
1/40
$0.25 vs $10.00
deepseek-v4-pro · DeepSeek
1/30
$0.044 vs $1.32
gpt-6.1-sol · OpenAI
1/20
$0.10 vs $2.00
Claude Opus 5.5 · Anthropic
1/20
$0.20 vs $4.00
gpt-6-astra · OpenAI
1/10
$1.00 vs $10.00
Gemini 3.1 Pro · Google
1/10
$0.20 vs $2.00
Kimi K3 · Moonshot
1/10
$0.30 vs $3.00
GLM-5.3 · Zhipu
1/5
$0.26 vs $1.40
MiniMax-M3 · MiniMax
1/5
$0.06 vs $0.30
Grok 4.7 · xAI
1/4
$0.50 vs $2.00
Each provider's pricing page, 2026-10-07, cross-checked with LiteLLM's price list. $ per 1M tokens.
An agent is a loop of stateless API calls
The whole loop in 15 lines
context only grows. Every trip resends all of it.
call_id ties each result to its call; several calls can come at once.
Real harnesses add a turn limit and a budget.
Each call resends everything
Only the new tail is full price; the rest is read from the cache.
Call 1
→ call 1
Call 2
→ call 2
Call 3
→ answer
System prompt
instructions: role, rules, style. Codex adds your AGENTS.md here.
Tool schemas
One JSON Schema per tool. Fifty tools can cost tens of thousands of tokens before the task starts.
Agent context
The task, every call, every result. Mostly tool output.
■ read from cache ■ new tokens, full price. Widths not to scale.
A tool schema is a prompt
- The model reads the description as text. Write it like instructions to a new student.
- Typed parameters let strict mode enforce the JSON.
- An MCP server (later this quarter) is a standard way to hand a harness a set of these, and run the calls.
One agent run: $19 or $5
Cumulative cost of a 30-turn run on gpt-6-astra. Without caching, every turn costs more than the last.
Assumed: a 10,000-token prefix; each turn 1,000 reasoning + 300 call tokens out and a 3,000-token tool result back; every cache lookup hits. With caching, output is 40% of the bill.
Five ways to keep the context small
| Strategy | What it does | What can go wrong |
|---|---|---|
| Truncation | Drop the oldest turns (truncation: "auto") | Loses the original task; breaks the cache |
| Clear tool results | Replace old outputs with a short placeholder | The agent re-runs the tool |
| Compaction | Near a threshold, summarize the history and restart from the summary | The summary drops a detail that matters later |
| Sub-agents | Run a sub-task in a fresh context; only a short report comes back | The report hides how it got the answer |
| Memory files | Write notes and plans to disk, re-read on demand | Notes go stale or are never re-read |
OpenAI: context_management: [{type: "compaction", compact_threshold: 200000}] compacts on the server and returns an encrypted compaction item; Codex has /compact and an auto-compact threshold.
A $200 subscription is a bundle of tokens
| Plan | Price | What it promises |
|---|---|---|
| Claude Max 20x | $200/mo | 20× Pro's usage per 5-hour session, plus a weekly cap |
| ChatGPT Pro | $200/mo | 10× Plus for Codex; no 5-hour limit (the old $200 plan was 20×) |
Neither company states the limits in tokens or dollars. Both include their coding agents, Claude Code and Codex.
$200 at API prices buys about
| Model | Tokens |
|---|---|
| gpt-6.1-sol | 565 M |
| Claude Opus 5.5 | 265 M |
| gpt-6-astra | 90 M |
Our estimate for a typical agent mix: 93% cache reads, 5% cache writes, 0.5% fresh input, 1.5% output.
support.claude.com (Max plan) · claude.com/pricing · learn.chatgpt.com/docs/pricing · TechCrunch 2026-04-09 · OfficeChai (20× to 10×, not confirmed on OpenAI's site)
Live data: what heavy users pull from a $200 plan
viberank.app ranks developers by the API-equivalent cost of their agent logs (via ccusage). September 2026:
Median $6,091 a month at API prices: about 30 times a $200 plan.
97% of tokens are cache reads. Caching (Part 2) is what makes flat-rate plans possible.
Read with care: self-reported, self-selected heavy users, priced at list rates.
viberank.app/stats/monthly/2026-09 (live; checked 2026-10-07) · try it on yourself: npx ccusage
Before your first call
- The API is not ChatGPT. A separate account, billed per token, with its own key.
- Keys live in an environment variable, OPENAI_API_KEY, never in code or a commit. Our homework repos are public, and bots scan GitHub for keys.
- Set a budget first. An agent loop with a bug can spend all night.
- Stream long answers (stream: true): same price, no waiting on a blank screen.
| You see | It means |
|---|---|
401 | Bad or missing key |
400 | Bad request: read the message, it names the field |
429 | Rate limit: wait and retry; the SDK retries for you |
5xx | The provider's problem: retry later |
incomplete | Hit max_output_tokens, maybe mid-thought |
Research hygiene
Log every call
Exact model ID, every parameter, the full request and response, and usage. Model names move and old models retire.
Expect variation
One prompt, 1,000 runs at temperature 0: 80 different answers, first split at token 103. Run important calls several times.
Know where data goes
Responses are stored by default (store: true, at least 30 days). Check before sending unpublished or personal data.
Untrusted input
A web page the agent reads lands in the same token stream as your instructions, and can carry instructions of its own.
Variation: Thinking Machines, "Defeating Nondeterminism in LLM Inference", Sept 2025 (Qwen3-235B). Cause: each request is batched with a different number of others.
Takeaways
- An API call is stateless JSON in, JSON out; the model only ever sees one stream of tokens.
- A turn ends in text or a tool call; strict mode masks the softmax so the JSON is valid.
- An agent is a loop: run the call, append the result, call again.
- Output costs ~5× input; a cached read costs 1/10 to 1/50. Keep stable things first.
- Context only grows: compaction, sub-agents and memory files keep it small, at the risk of forgetting.