Week 2 · Further reading

How an AI agent works, in more depth

The slides walked from one API call to an agent loop. This page has the details and sources behind them. Prices and model names are from October 7, 2026, and change often: when this page and a provider's docs disagree, trust the docs.

  1. One API call
  2. What the model actually sees
  3. Tool calls and strict mode
  4. Sampling, temperature and reasoning
  5. The KV cache and prompt caching
  6. What tokens cost
  7. The loop and its context
  8. Subscriptions against the API
  9. Before your first call, and research hygiene

1. One API call

An API call is an HTTP request: a URL (which service), headers (who you are, with your key), and a JSON body (what you want). The reply has a status code and a JSON body, including usage, the token counts you are billed for. Everything is counted in tokens: about 1,000 tokens per 750 English words, so a 7,000-word paper is about 9,000 tokens.

DialectSystem promptConversationA tool call is
OpenAI Responses (the class's examples)instructionsinput itemsa function_call item
OpenAI Chat Completions (your harness)a system messagemessagestool_calls[]
Anthropic Messagessystemmessagesa tool_use block
Google GeminisystemInstructioncontentsa functionCall part

Chat Completions is the de facto standard: local servers such as Ollama and vLLM speak it, so one harness can drive an open model on a lab GPU by changing a URL.

OpenAI Responses API reference · OpenAI: function calling

2. What the model actually sees

The model never sees JSON. The server flattens your request into one stream of tokens with role markers, and the model continues it, exactly as in week 1. OpenAI's open-weight gpt-oss models document their format, "harmony", in full:

<|start|>system<|message|>You are ChatGPT ... Reasoning: high<|end|>
<|start|>developer<|message|># Tools ... get_current_weather ...<|end|>
<|start|>user<|message|>What is the weather like in SF?<|end|>
<|start|>assistant<|channel|>commentary to=functions.get_current_weather
  <|constrain|>json<|message|>{"location":"San Francisco"}<|call|>

Tool definitions, your instructions and even the reasoning level are just text in that stream. That is why a tool schema is a prompt, and why text in a web page the agent reads can carry instructions of its own (week 3: prompt injection).

OpenAI: the harmony response format

3. Tool calls and strict mode

A tool call is tokens the model was fine-tuned to write. The server watches for the <|call|> token, stops generating, and hands you the call as JSON. The model never runs anything: your program does.

Without strict mode the JSON is usually right and sometimes broken, because it is still sampled text. With strict: true, OpenAI converts your schema "into a context-free grammar" and, before each token is sampled, sets the score of every token the grammar forbids to minus infinity, so its probability after the softmax is exactly zero. The shape is guaranteed; the content is not: a misspelled city still passes.

OpenAI: structured outputs

4. Sampling, temperature and reasoning

The last layer gives a score for every token in the vocabulary; softmax turns the scores into probabilities; then one token is sampled. Temperature divides the scores before the softmax: low makes the top token dominate, high flattens the choice.

WhereCan you set temperature?What you turn instead
OpenAI GPT-6, reasoning onnoreasoning.effort, text.verbosity
Claude Opus 5.5no: the API rejects sampling parameterseffort
Open-weight models you run (vLLM, Ollama)yes, plus top_p, top_kgpt-oss recommends temperature 1.0

Reasoning effort sets roughly how many hidden tokens the model writes before answering. In OpenAI's own example, 832 of 1,035 output tokens were hidden reasoning, billed at the output price. In gpt-oss the dial is literally the line Reasoning: high in the system message, so changing it also breaks the prompt cache. Higher is not always better: for extraction, low is cheaper, faster and often as accurate.

Service tiers run the same weights in a different queue: Batch (results within 24 hours) and flex cost half, fast about twice. For research jobs that can wait, Batch halves the bill.

OpenAI: reasoning · OpenAI: pricing and tiers

5. The KV cache and prompt caching

Prefill reads the whole prompt in one parallel pass, and each layer stores every token's keys and values (week 1's K and V): the KV cache. Decode then writes one token at a time, each attending over everything stored. That's why output is slow and expensive, and input is fast and cheap.

Prompt caching keeps the KV cache of a prompt's beginning for a while. The next call that starts with exactly the same tokens skips that work, because token i's keys and values depend on every token before it. So: put stable things first (tool schemas, then the system prompt, then the conversation), and never put a timestamp at the top. On OpenAI caching is automatic from 1,024 tokens; changing the tools, the output format or the reasoning effort breaks the prefix.

The context window is per call: 1,050,000 tokens for gpt-6-astra, input and output together. A long agent run fills it because every call resends everything, and accuracy drops as old, irrelevant text piles up ("context rot").

OpenAI: prompt caching · Anthropic: prompt caching (August 2024)

6. What tokens cost

gpt-6-astra, $ per million tokensPriceWhy
Cache read$1.00keys and values already computed; just load them
Fresh input$10.00prefill: one parallel pass, the GPU stays busy
Cache write$12.50the same prefill, plus memory held for 30+ minutes
Output, and hidden reasoning$50.00decode: one full pass through the network per token

A cached token has fallen from half the price of a fresh one (OpenAI, October 2024) to a tenth (GPT-5, August 2025) and a twentieth (gpt-6.1-sol and Claude Opus 5.5, September 2026); Claude Fable 5.1 charges a fortieth and DeepSeek Flash a fiftieth. For agents, which mostly re-read their own history, the cache price matters more than the headline price.

One agent run, worked out: 30 turns on gpt-6-astra with a 10,000-token prefix, 1,300 tokens out and a 3,000-token tool result per turn, costs $19.31 without caching and $4.90 with it. Without caching the run re-reads 1.74 million input tokens. Try it with 30,000-token tool results.

OpenAI pricing · Claude pricing · each provider's pricing page on October 7, 2026, cross-checked with LiteLLM's price list

7. The loop and its context

An agent is a loop of stateless API calls: call the model with the whole context; if it asks for a tool, run it, append the call and its result (matched by call_id), and call again; if it writes text, stop. Real harnesses add a turn limit and a budget.

Keeping the context smallWhat it doesWhat can go wrong
Truncationdrop the oldest turnsloses the original task; breaks the cache
Clearing tool resultsreplace old outputs with a placeholderthe agent re-runs the tool
Compactionnear a threshold, summarize and restart from the summarythe summary drops a detail that matters later
Subagentsa sub-task in a fresh context; a short report comes backthe report hides how it got the answer
Memory filesnotes and plans on disk, re-read on demandnotes go stale or are never re-read

OpenAI's API can compact on the server (context_management: [{type: "compaction", compact_threshold: 200000}]); Codex has /compact. Week 3 goes deeper into memory and subagents: week 3 reading.

8. Subscriptions against the API

A $200 plan (Claude Max 20x, ChatGPT Pro) is a prepaid bundle with fuzzy limits; neither company states them in tokens. At API prices, with an agent's typical mix (about 93% cache reads), $200 buys roughly 565 million tokens of gpt-6.1-sol, 265 million of Claude Opus 5.5, or 90 million of gpt-6-astra.

On viberank, heavy users who publish their agent logs pulled a median of about $6,100 a month at API prices in September 2026, about 30 times a $200 plan, and 97% of their tokens were cache reads. Caching is what makes flat-rate plans possible. Read it with care: self-reported, self-selected, priced at list rates. See your own usage with npx ccusage.

Claude Help Center: Max plan · claude.com/pricing · ChatGPT and Codex pricing

9. Before your first call, and research hygiene

Also this week