Week 2 · Further reading
How an AI agent works, in more depth
The slides walked from one API call to an agent loop. This page has the details and sources behind them. Prices and model names are from October 7, 2026, and change often: when this page and a provider's docs disagree, trust the docs.
1. One API call
An API call is an HTTP request: a URL (which service), headers (who you are, with your key), and a JSON body (what you want).
The reply has a status code and a JSON body, including usage, the token counts you are billed for.
Everything is counted in tokens: about 1,000 tokens per 750 English words, so a 7,000-word paper is about 9,000 tokens.
| Dialect | System prompt | Conversation | A tool call is |
|---|---|---|---|
| OpenAI Responses (the class's examples) | instructions | input items | a function_call item |
| OpenAI Chat Completions (your harness) | a system message | messages | tool_calls[] |
| Anthropic Messages | system | messages | a tool_use block |
| Google Gemini | systemInstruction | contents | a functionCall part |
Chat Completions is the de facto standard: local servers such as Ollama and vLLM speak it, so one harness can drive an open model on a lab GPU by changing a URL.
2. What the model actually sees
The model never sees JSON. The server flattens your request into one stream of tokens with role markers, and the model continues it, exactly as in week 1. OpenAI's open-weight gpt-oss models document their format, "harmony", in full:
<|start|>system<|message|>You are ChatGPT ... Reasoning: high<|end|>
<|start|>developer<|message|># Tools ... get_current_weather ...<|end|>
<|start|>user<|message|>What is the weather like in SF?<|end|>
<|start|>assistant<|channel|>commentary to=functions.get_current_weather
<|constrain|>json<|message|>{"location":"San Francisco"}<|call|>
Tool definitions, your instructions and even the reasoning level are just text in that stream. That is why a tool schema is a prompt, and why text in a web page the agent reads can carry instructions of its own (week 3: prompt injection).
3. Tool calls and strict mode
A tool call is tokens the model was fine-tuned to write. The server watches for the <|call|> token, stops generating,
and hands you the call as JSON. The model never runs anything: your program does.
Without strict mode the JSON is usually right and sometimes broken, because it is still sampled text. With strict: true,
OpenAI converts your schema "into a context-free grammar" and, before each token is sampled, sets the score of every token the grammar forbids
to minus infinity, so its probability after the softmax is exactly zero. The shape is guaranteed; the content is not:
a misspelled city still passes.
4. Sampling, temperature and reasoning
The last layer gives a score for every token in the vocabulary; softmax turns the scores into probabilities; then one token is sampled. Temperature divides the scores before the softmax: low makes the top token dominate, high flattens the choice.
| Where | Can you set temperature? | What you turn instead |
|---|---|---|
| OpenAI GPT-6, reasoning on | no | reasoning.effort, text.verbosity |
| Claude Opus 5.5 | no: the API rejects sampling parameters | effort |
| Open-weight models you run (vLLM, Ollama) | yes, plus top_p, top_k | gpt-oss recommends temperature 1.0 |
Reasoning effort sets roughly how many hidden tokens the model writes before answering. In OpenAI's own example, 832 of 1,035 output tokens
were hidden reasoning, billed at the output price. In gpt-oss the dial is literally the line Reasoning: high in the system message,
so changing it also breaks the prompt cache. Higher is not always better: for extraction, low is cheaper, faster and often as accurate.
Service tiers run the same weights in a different queue: Batch (results within 24 hours) and flex cost half, fast about twice. For research jobs that can wait, Batch halves the bill.
5. The KV cache and prompt caching
Prefill reads the whole prompt in one parallel pass, and each layer stores every token's keys and values (week 1's K and V): the KV cache. Decode then writes one token at a time, each attending over everything stored. That's why output is slow and expensive, and input is fast and cheap.
Prompt caching keeps the KV cache of a prompt's beginning for a while. The next call that starts with exactly the same tokens skips that work, because token i's keys and values depend on every token before it. So: put stable things first (tool schemas, then the system prompt, then the conversation), and never put a timestamp at the top. On OpenAI caching is automatic from 1,024 tokens; changing the tools, the output format or the reasoning effort breaks the prefix.
The context window is per call: 1,050,000 tokens for gpt-6-astra, input and output together. A long agent run fills it because every call resends everything, and accuracy drops as old, irrelevant text piles up ("context rot").
OpenAI: prompt caching · Anthropic: prompt caching (August 2024)
6. What tokens cost
| gpt-6-astra, $ per million tokens | Price | Why |
|---|---|---|
| Cache read | $1.00 | keys and values already computed; just load them |
| Fresh input | $10.00 | prefill: one parallel pass, the GPU stays busy |
| Cache write | $12.50 | the same prefill, plus memory held for 30+ minutes |
| Output, and hidden reasoning | $50.00 | decode: one full pass through the network per token |
A cached token has fallen from half the price of a fresh one (OpenAI, October 2024) to a tenth (GPT-5, August 2025) and a twentieth (gpt-6.1-sol and Claude Opus 5.5, September 2026); Claude Fable 5.1 charges a fortieth and DeepSeek Flash a fiftieth. For agents, which mostly re-read their own history, the cache price matters more than the headline price.
One agent run, worked out: 30 turns on gpt-6-astra with a 10,000-token prefix, 1,300 tokens out and a 3,000-token tool result per turn, costs $19.31 without caching and $4.90 with it. Without caching the run re-reads 1.74 million input tokens. Try it with 30,000-token tool results.
OpenAI pricing · Claude pricing · each provider's pricing page on October 7, 2026, cross-checked with LiteLLM's price list
7. The loop and its context
An agent is a loop of stateless API calls: call the model with the whole context; if it asks for a tool, run it, append the call and its result (matched by call_id), and call again; if it writes text, stop. Real harnesses add a turn limit and a budget.
| Keeping the context small | What it does | What can go wrong |
|---|---|---|
| Truncation | drop the oldest turns | loses the original task; breaks the cache |
| Clearing tool results | replace old outputs with a placeholder | the agent re-runs the tool |
| Compaction | near a threshold, summarize and restart from the summary | the summary drops a detail that matters later |
| Subagents | a sub-task in a fresh context; a short report comes back | the report hides how it got the answer |
| Memory files | notes and plans on disk, re-read on demand | notes go stale or are never re-read |
OpenAI's API can compact on the server (context_management: [{type: "compaction", compact_threshold: 200000}]); Codex has /compact. Week 3 goes deeper into memory and subagents: week 3 reading.
8. Subscriptions against the API
A $200 plan (Claude Max 20x, ChatGPT Pro) is a prepaid bundle with fuzzy limits; neither company states them in tokens. At API prices, with an agent's typical mix (about 93% cache reads), $200 buys roughly 565 million tokens of gpt-6.1-sol, 265 million of Claude Opus 5.5, or 90 million of gpt-6-astra.
On viberank, heavy users who publish their agent logs pulled a median of about $6,100 a month at API prices in September 2026,
about 30 times a $200 plan, and 97% of their tokens were cache reads. Caching is what makes flat-rate plans possible. Read it with care: self-reported, self-selected, priced at list rates.
See your own usage with npx ccusage.
Claude Help Center: Max plan · claude.com/pricing · ChatGPT and Codex pricing
9. Before your first call, and research hygiene
- The API is not ChatGPT: a separate account, billed per token, with its own key. Keep the key in an environment variable, never in code or a commit: bots scan GitHub for keys.
- Set a budget first. An agent loop with a bug can spend all night.
- Read the status code: 401 bad key, 400 bad request (the message names the field), 429 rate limit (wait and retry), 5xx the provider's problem.
- Log every call: the exact model ID, every parameter, the full request and response, and the usage. Model names move and old models retire.
- Expect variation. One prompt run 1,000 times at temperature 0 gave 80 different answers, first splitting at token 103, because each request is batched with a different number of others (Thinking Machines, September 2025). Run important calls several times.
- Know where data goes. Responses are stored by default (
store: true, at least 30 days). Check before sending unpublished or personal data. - Valid is not correct. Strict mode guarantees the JSON's shape, not its content, and an agent's confident final answer is not a checked one.