Week 2 slides: how an AI agent works

← → to move, F for full screen, N for speaker notes

AI in Research · Week 2 · October 7, 2026

How an AI agent works

From one API call to an agent loop, and what every token costs along the way.

class.xhelio.ai/week-2

Week 2 · Yesterday02

October 6, 2026: mathematician as an occupation is officially dead

OpenAI post: Sharing AI progress in mathematics, October 6, 2026

722

math manuscripts in 372 result families, from one unreleased OpenAI model

  • About 4,000 open problems posed
  • About 3 hours of ChatGPT Pro thinking per result, on average
  • Includes the Mahler conjectures and the isomorphism of free group factors
  • Many, not all, checked by computer in Lean

openai.com/index/sharing-ai-progress-in-mathematics · github.com/openai/math (README: "Some of the unformalized results could have issues.")

Week 2 · Recap03

Last week: an agent is a model in a loop

Today we open each box in that loop, using one real API, the OpenAI Responses API. Other providers' APIs are dialects of the same thing.

Part 1

One call

What an API, HTTP and JSON are, then one real request and its reply.

Part 2

Inside the call

Tool calls as tokens, strict JSON, reasoning, the KV cache, and token prices.

Part 3

The loop

Chaining calls into an agent, and keeping its context small.

Part 4

Using it

Subscriptions against the API, practical basics, research hygiene.

Week 2 · Part 1 · One call04

Two ways to use a language model

Most of you have used the first. Agents are built on the second.

A chat app

You type, you read

You
→
ChatGPT, Claude.ai
→
the model

Good for asking, drafting, exploring.

Paid by a monthly subscription.

The API

A program asks for you

Your program
→
the API
→
the model

Good for automation: a thousand papers, a nightly job, an agent.

Paid per token. Codex and Claude Code are programs that use the API.

Week 2 · Part 1 · One call05

What is an API?

An API (application programming interface) is a fixed list of requests one program can send another, and the reply it promises back.

You already use one

Fetching spacecraft data

A script asks a data archive such as CDAWeb for this dataset, this time range. The server sends the numbers back.

You never see the archive's database. You only see the request you may send and the reply you get.

Today

Asking a language model

Your script sends this text, these settings to a model provider. The server runs the model and sends text back.

You never see the GPUs or the weights. Same idea, different server.

Week 2 · Part 1 · One call06

Programs talk over HTTP, like your browser does

Your browser sends a request to a URL and gets a response. A program does exactly the same.

Request: your program to the server

POST https://api.openai.com/v1/responses # headers: who you are, what you are sending Authorization: Bearer sk-...your key... Content-Type: application/json # body: what you want { "model": "gpt-6-astra", "input": "What is a CME?" }

Response: the server back to you

200 OK # the status code: it worked Content-Type: application/json # body: the answer, plus the bill { "output": [ ... "A coronal mass ejection is..." ... ], "usage": { "input_tokens": 19, "output_tokens": 41 } }

The URL says which service. POST means "here is data, do something with it".

The key proves who you are and who pays. Treat it like a password.

The status code: 200 worked, 401 bad key, 429 slow down.

Week 2 · Part 1 · One call07

What is JSON?

JSON is plain text that holds structured data. Humans can read it, every programming language can parse it, and nearly every API speaks it.

{ "spacecraft": "Parker Solar Probe", "time": "2026-10-07T11:00:00Z", "distance_au": 0.17, "in_encounter": true, "plasma": { "speed_km_s": 412.5, "density_cm3": 85.0 }, "instruments": ["FIELDS", "SWEAP", "ISOIS", "WISPR"], "notes": null }
You seeIt means
{ }An object: named fields
"key": valueOne field: a name and its value
"text"A string, always in double quotes
0.17A number, no quotes
true false nullYes, no, nothing
[ ]A list, in order
"plasma": { }Objects nest inside objects

The example's values are made up; the four instrument suites are real.

Week 2 · Part 1 · One call08

Reading JSON, and describing it

Reading it in Python

import json text = ('{"plasma": {"speed_km_s": 412.5},' ' "instruments": ["FIELDS", "SWEAP"]}') data = json.loads(text) # JSON text -> Python dict data["plasma"]["speed_km_s"] # 412.5 data["instruments"][0] # "FIELDS" json.dumps(data) # Python dict -> JSON text

Every API reply is read this way: resp["output"][0]["type"].

A JSON Schema

{ "type": "object", "properties": { "speed_km_s": { "type": "number" }, "spacecraft": { "type": "string" } }, "required": ["speed_km_s"] }

JSON that describes what valid JSON looks like: which keys, which types. Tools are defined with one.

Week 2 · Part 1 · One call09

Everything is counted in tokens

From week 1: a token is a word or a piece of one. About 1,000 tokens make 750 English words.

What you send

Input tokens. Your instructions, the conversation, tool definitions, files.

What comes back

Output tokens. The answer, tool calls, and any hidden reasoning.

The limit

The context window: the most tokens one call can hold, in and out together.

The bill

Prices are quoted in dollars per million tokens, and input and output are priced differently.

Rough scale: a 7,000-word paper is about 9,000 tokens. A million tokens is roughly 750,000 words.

Week 2 · Part 1 · One call10

Putting it together: your first API call

One command sends JSON to OpenAI and gets JSON back. The model remembers nothing between calls, so a conversation means resending it every time.

You send

curl https://api.openai.com/v1/responses \ -H "Authorization: Bearer $OPENAI_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "gpt-6-astra", "instructions": "Be brief.", "input": "What is a CME?" }'

You get back

{ "id": "resp_67ca...", "status": "completed", "output": [{ "type": "message", "content": [{ "type": "output_text", "text": "A coronal mass ejection is..." }] }], "usage": { "input_tokens": 19, "output_tokens": 41 } }

The reply is trimmed. Every chat app and every agent is built from calls like this one.

Week 2 · Part 1 · One call11

The model never sees JSON

The server flattens your request into one stream of tokens with role markers. The model just continues it, as in week 1.

<|start|>system<|message|>You are ChatGPT, a large language model trained by OpenAI. Reasoning: high <|end|><|start|>developer<|message|># Tools // Gets the current weather in the provided location. type get_current_weather = (_: { location: string }) => any;<|end|> <|start|>user<|message|>What is the weather like in SF?<|end|> <|start|>assistant<|channel|>analysis<|message|>User wants weather; call the tool.<|end|> <|start|>assistant<|channel|>commentary to=functions.get_current_weather <|constrain|>json<|message|>{"location":"San Francisco"}<|call|> <|start|>functions.get_current_weather to=assistant<|message|>{"sunny": true, "temperature": 20}<|end|> <|start|>assistant<|channel|>final<|message|>It is sunny and 20 °C in San Francisco.<|return|>

Roles are text markers. The model is trained to weight them, not forced to.

Tool schemas become text in the developer message.

Hidden reasoning is the analysis channel; the answer is final.

OpenAI's open-weight gpt-oss "harmony" format, shortened from OpenAI's own example (developers.openai.com/cookbook/articles/openai-harmony).

Week 2 · Part 1 · One call12

The request

{ "model": "gpt-6-astra", # which model "instructions": "You are a weather assistant.", # system prompt "input": "What is the weather in Boston today?", "tools": [{ # what it may call "type": "function", "name": "get_current_weather", "description": "Get the current weather in a location", "parameters": { "type": "object", "properties": { "location": { "type": "string" } }, "required": ["location"], "additionalProperties": false }, "strict": true }], "tool_choice": "auto", # let the model decide "reasoning": { "effort": "medium" }, # how long to think "max_output_tokens": 32000, "service_tier": "default" # which queue }
  • What to say
    model, instructions, input
  • What it may do
    tools, tool_choice, text.format
  • How hard, how long
    reasoning.effort, max_output_tokens
  • Cost and memory
    service_tier, prompt caching, store, previous_response_id
Week 2 · Part 1 · One call13

The response

{ "status": "completed", "output": [{ "type": "function_call", "call_id": "call_unLAR8MvFNptuiZK6K6HCy5k", "name": "get_current_weather", "arguments": "{\"location\":\"Boston, MA\"}" }], "usage": { "input_tokens": 291, "input_tokens_details": { "cached_tokens": 0 }, "output_tokens": 23, "output_tokens_details": { "reasoning_tokens": 0 } } }
  • output is a list of typed items: reasoning, message, function_call.
  • arguments is a string of JSON. Your program parses it.
  • status is incomplete if the model ran out of max_output_tokens, sometimes while still thinking.
  • usage is the bill. In OpenAI's own example, 832 of 1,035 output tokens were hidden reasoning.

Response to the weather request, from OpenAI's API reference (developers.openai.com/api/reference/resources/responses/methods/create).

Week 2 · Part 1 · One call14

Two kinds of output

Every turn ends one of two ways. Your program branches on which one came back.

type: "message"

Text for a person

Plain words in output_text.

Nothing to run, so the loop stops.

type: "function_call"

A request to your program

A tool name, JSON arguments, and a call_id.

You run it, send back a function_call_output with the same call_id, and call again.

tool_choice steers it: "auto" lets the model decide, "required" forces a call, "none" forbids one. A third shape, text forced into a JSON schema (text.format), uses the same machinery as tool calls.

Week 2 · Part 1 · One call15

Other APIs are dialects of the same idea

Same request, different field names. Read any provider's docs with this table.

DialectSystem promptConversationText comes back inA tool call is
OpenAI Responses (this class)instructionsinput itemsoutput[] messagefunction_call item
OpenAI Chat Completionsa system messagemessageschoices[0].messagetool_calls[]
Anthropic Messagessystemmessagescontent[] text blocktool_use block
Google GeminisystemInstructioncontents, partscandidates[0] partsfunctionCall part

Chat Completions became a de facto standard: local servers such as Ollama and vLLM speak it, so the same code can run an open model on a lab GPU by changing one URL.

Week 2 · Part 2 · Inside the call16

A tool call is just tokens

The model is fine-tuned to write calls in a special format. The server watches for the <|call|> token, stops generating, and hands you the call as JSON.

<|start|>assistant<|channel|>commentary to=functions.get_current_weather <|constrain|>json<|message|>{"location":"San Francisco"}<|call|>

Without strict mode

Usually right. Sometimes invalid JSON or a missing field, because it is still just sampled text.

With strict: true

The server forces the JSON to match your schema, token by token. Next slide.

Week 2 · Part 2 · Inside the call17

The last layer: a score for every token

After all the attention layers, the model multiplies its final vector by one big matrix. Out comes one number per vocabulary token, called a logit: higher means "more likely next".

All the layers attention, as in week 1, 96 deep in GPT-3 One vector for the last position: thousands of numbers × one big matrix a row of weights per vocabulary token A score per token 4005.114504.175003.593503.19one2.49 …and the rest of the vocabulary

"The solar wind near Earth blows at about ___". Illustrative scores, chosen to match week 1's slide. Scores can be any real number, negative too.

Week 2 · Part 2 · Inside the call18

Softmax turns scores into probabilities

TokenScore zezProbability
4005.1116541%
4504.176516%
5003.59369%
3503.19246%
one2.49123%
every other token…101 together25%
Sum403100%
p = ez / (sum of all ez)
  • Exponentiate each score: every result is positive, and bigger scores grow much faster.
  • Divide by the total: the shares add up to 100%.
  • Only the differences between scores matter. Add 10 to every score and nothing changes.

The probabilities are week 1's slide exactly; the scores here are chosen to produce them.

Week 2 · Part 2 · Inside the call19

Then pick one: sampling and temperature

The server draws one token at random, weighted by these probabilities, appends it, and runs the whole network again. Temperature divides every score before the softmax.

Temperature 0.5: sharper

400

81%

450

12%

500

4%

350

2%

one

0%

Temperature 1: as trained

400

55%

450

21%

500

12%

350

8%

one

4%

Temperature 2: flatter

400

36%

450

23%

500

17%

350

14%

one

10%

Shares among these five tokens only. Temperature near 0 always picks the top token. As the next slide shows, today's frontier APIs mostly no longer let you set it.

Week 2 · Part 2 · Inside the call20

Temperature is fading from frontier APIs

Reasoning models run their own sampling during long hidden thinking, so providers stopped exposing it. The dials you turn now are how long the model thinks and how much it writes.

WhereCan you set temperature?What you turn instead
OpenAI GPT-6, reasoning onNo: remove temperature, top_p, top_logprobsreasoning.effort, text.verbosity
Claude Opus 5.5No: the API rejects sampling parameterseffort
Open-weight models you run yourself (vLLM, Ollama)Yes, plus top_p, top_kgpt-oss recommends temperature=1.0, top_p=1.0

Need repeatable output? Temperature 0 never guaranteed it anyway (Part 4). Log, and run several times.

Need a fixed format? Use a JSON schema (text.format or a strict tool), not a low temperature.

Week 2 · Part 2 · Inside the call21

Strict mode masks the softmax

The model has written {"location": and the schema says a string must come next. The server sets the score of every forbidden token to minus infinity before the softmax. Since e−∞ = 0, those tokens get probability zero; the rest are renormalized.

Before the mask

Grammar

After the mask

"

0.62

allowed

0.97

Paris

0.21

masked

0.00

null

0.09

masked

0.00

"P

0.02

allowed

0.03

other

0.06

masked

0.00

Illustrative numbers. OpenAI: the schema is converted "into a context-free grammar", and masking "effectively lowers the probability of invalid tokens to 0." The shape is guaranteed; the content is not: "Pairs" would pass.

Week 2 · Part 2 · Inside the call22

Reasoning effort: hidden tokens before the answer

832 of 1,035

output tokens in OpenAI's reasoning example were hidden reasoning, billed at the output price.

■ reasoning, 832   ■ visible answer, 203

  • Where the skill comes from: reinforcement learning rewards working problems out step by step.
  • How the dial works: in gpt-oss it is literally the line Reasoning: high in the system message. Changing reasoning.effort breaks the prompt cache, so it is part of the input there too.
  • Levels: none to max; each model accepts a subset.
  • Higher is not always better. For extraction, low is cheaper, faster and often as accurate.

developers.openai.com/api/docs/guides/reasoning · gpt-oss harmony format. The closed models' exact mechanism is not published.

Week 2 · Part 2 · Inside the call23

Service tier: the same model in a different queue

The weights are identical. You pay for how your request is scheduled on the provider's GPUs.

TierPriceWhat you get
Batch (separate API)0.5×Results within 24 hours. Grading 1,000 papers overnight.
flex0.5×Synchronous but slower; sometimes unavailable
default1×Standard
fast (called priority until July 30, 2026)2×Faster, more reliable throughput
ultrafast6×Highest speed, few models

gpt-6-astra, relative to Standard. developers.openai.com/api/docs/pricing and the flex, batch and priority-processing guides.

Week 2 · Part 2 · Inside the call24

The context window is per call

1,050,000

tokens fit in one gpt-6-astra call: up to 922,000 input and 128,000 output, reasoning included.

  • Per call, not per conversation. Every call resends everything, so a long agent run fills it even if no single message is big.
  • Bigger costs more. Each call bills the whole context again.
  • Bigger can be worse. Accuracy drops as old, irrelevant text piles up ("context rot").

developers.openai.com/api/docs/models/gpt-6-astra

Week 2 · Part 2 · Inside the call25

Recap from week 1: queries, keys and values

Each token makes three vectors in every layer. Week 1: "The trophy doesn't fit in the suitcase because it is too big."

Q · query

"What am I looking for?"

"big" asks: what is too big?

K · key

"What do I offer?"

Every earlier token shows a key. "trophy" matches the query best.

V · value

"What I pass on"

"big" takes in the values, weighted by those match scores.

attention(Q, K, V) = softmax(Q · KT / √d) · V

Why this matters today: to write token 101, only token 101 needs a new query. The keys and values of tokens 1 to 100 have not changed, so the server keeps them instead of recomputing. That store is the KV cache.

Week 2 · Part 2 · Inside the call26

The KV cache: week 1's K and V, kept around

Prefill

Read the whole prompt at once. All input tokens go through the network in one parallel pass, and each layer stores their keys and values: the KV cache.

Decode

Write one token at a time. Each new token makes a query and attends over all stored keys and values.

Live, OpenRouter median

t = 0.0 s

1st token

Writing

$ in / out

GPT-6.1 Sol flex

11.1 s

25 tok/s

$1 / $5

GPT-6.1 Sol standard

4.7 s

34 tok/s

$2 / $10

GPT-6.1 Sol fast

2.7 s

58 tok/s

$4 / $20

Claude Opus 5.5 standard

4.2 s

64 tok/s

$4 / $20

Claude Opus 5.5 fast

2.5 s

131 tok/s

$8 / $40

Grey: waiting for the first token (prefill, queue, thinking). Orange: decode at the measured speed. fast costs 2× and is about 2× quicker. openrouter.ai, P50 of real traffic, Oct 7 2026.

Week 2 · Part 2 · Inside the call27

Prompt caching: the prefix must match exactly

The server keeps the KV cache of your prompt's beginning for a while. The next call that starts with exactly the same tokens skips that work.

Call 1

tools
system prompt
task

everything is new: written to cache

Call 2

tools
system prompt
task
call + result

only the new tail is full price

Call 3

tools
system prompt + 07:42:13
task
call + result

a timestamp at the top changes token 1: nothing hits

■ read from cache   ■ new, full price   ■ cache broken

Why exactly: token i's key and value depend on every token before it. Stable things go first. On OpenAI caching is automatic from 1,024 tokens; changing tools, output format or reasoning effort also breaks the prefix.

Week 2 · Part 2 · Inside the call28

Why tokens cost different amounts

gpt-6-astra, dollars per million tokens. Output costs 5× input and 50× a cache read.

Cache read

$1.00

keys and values already computed; just load them

Input

$10.00

prefill: one parallel pass, the GPU stays busy

Cache write

$12.50

the same prefill, plus memory held for 30+ minutes

Output

$50.00

decode: one token per full pass through the network

Reasoning

$50.00

billed as output: it is output, just hidden

developers.openai.com/api/docs/pricing, standard tier, prompts up to 272K tokens

Week 2 · Part 2 · Inside the call29

A cached token went from 1/2 to 1/20 of the price

Price of a cached input token as a fraction of a fresh one (log scale). OpenAI in orange.

1/21/41/101/201/401/50Aug 2024Jan 2025Jan 2026Oct 2026gpt-4o, Oct 2024: 1/2GPT-4.1, Apr 2025: 1/4GPT-5, Aug 2025: 1/10gpt-6.1-sol and Claude Opus 5.5, Sep 2026: 1/20Claude Fable 5.1, Sep 2026: 1/40DeepSeek Flash, today: 1/50Anthropic and DeepSeek: 1/10 from launch, Aug 2024

OpenAI DevDay and GPT-4.1 posts, pricing snapshots and changelog; anthropic.com/news/prompt-caching; Claude pricing; DeepSeek news 2024-08-02 and pricing.

Week 2 · Part 2 · Inside the call30

Cached reads across providers today

Cached-read price as a fraction of the fresh input price. Orange: 1/20 or cheaper.

deepseek-flash · DeepSeek

1/50

$0.006 vs $0.30

Claude Fable 5.1 · Anthropic

1/40

$0.25 vs $10.00

deepseek-v4-pro · DeepSeek

1/30

$0.044 vs $1.32

gpt-6.1-sol · OpenAI

1/20

$0.10 vs $2.00

Claude Opus 5.5 · Anthropic

1/20

$0.20 vs $4.00

gpt-6-astra · OpenAI

1/10

$1.00 vs $10.00

Gemini 3.1 Pro · Google

1/10

$0.20 vs $2.00

Kimi K3 · Moonshot

1/10

$0.30 vs $3.00

GLM-5.3 · Zhipu

1/5

$0.26 vs $1.40

MiniMax-M3 · MiniMax

1/5

$0.06 vs $0.30

Grok 4.7 · xAI

1/4

$0.50 vs $2.00

Each provider's pricing page, 2026-10-07, cross-checked with LiteLLM's price list. $ per 1M tokens.

Week 2 · Part 3 · The loop31

An agent is a loop of stateless API calls

Context instructions, tools, history API call on the provider's GPUs Output? Answer loop ends Run the tool Python, shell, web, MCP Append the call and its result text function_call result, as text Everything outside the API call runs in your program: the harness
Week 2 · Part 3 · The loop32

The whole loop in 15 lines

import json from openai import OpenAI client = OpenAI() context = [{"role": "user", "content": "Estimate the spectral index of mag.csv"}] while True: resp = client.responses.create(model=MODEL, instructions=SYSTEM_PROMPT, tools=TOOLS, input=context) context += resp.output # keep the model’s own turn calls = [o for o in resp.output if o.type == "function_call"] if not calls: # text output: done print(resp.output_text) break for c in calls: # we run the tools, not the model result = run_tool(c.name, json.loads(c.arguments)) context.append({"type": "function_call_output", "call_id": c.call_id, "output": result})

context only grows. Every trip resends all of it.

call_id ties each result to its call; several calls can come at once.

Real harnesses add a turn limit and a budget.

Week 2 · Part 3 · The loop33

Each call resends everything

Only the new tail is full price; the rest is read from the cache.

Call 1

tools
system prompt
task

→ call 1

Call 2

tools
system prompt
task
call 1
result 1

→ call 2

Call 3

tools
system prompt
task
call 1
result 1
call 2
result 2

→ answer

System prompt

instructions: role, rules, style. Codex adds your AGENTS.md here.

Tool schemas

One JSON Schema per tool. Fifty tools can cost tens of thousands of tokens before the task starts.

Agent context

The task, every call, every result. Mostly tool output.

■ read from cache   ■ new tokens, full price. Widths not to scale.

Week 2 · Part 3 · The loop34

A tool schema is a prompt

{ "type": "function", "name": "read_csv_column", "description": "Read one column of a CSV. Use before fitting.", "parameters": { "type": "object", "properties": { "path": { "type": "string", "description": "Path to the CSV" }, "column": { "type": "string", "description": "Column name, e.g. br" } }, "required": ["path", "column"], "additionalProperties": false }, "strict": true }
  • The model reads the description as text. Write it like instructions to a new student.
  • Typed parameters let strict mode enforce the JSON.
  • An MCP server (later this quarter) is a standard way to hand a harness a set of these, and run the calls.
Week 2 · Part 3 · The loop35

One agent run: $19 or $5

Cumulative cost of a 30-turn run on gpt-6-astra. Without caching, every turn costs more than the last.

$0$5$10$15$201102030turn (one API call each)No caching: $19.31With caching: $4.90

Assumed: a 10,000-token prefix; each turn 1,000 reasoning + 300 call tokens out and a 3,000-token tool result back; every cache lookup hits. With caching, output is 40% of the bill.

Week 2 · Part 3 · The loop36

Five ways to keep the context small

StrategyWhat it doesWhat can go wrong
TruncationDrop the oldest turns (truncation: "auto")Loses the original task; breaks the cache
Clear tool resultsReplace old outputs with a short placeholderThe agent re-runs the tool
CompactionNear a threshold, summarize the history and restart from the summaryThe summary drops a detail that matters later
Sub-agentsRun a sub-task in a fresh context; only a short report comes backThe report hides how it got the answer
Memory filesWrite notes and plans to disk, re-read on demandNotes go stale or are never re-read

OpenAI: context_management: [{type: "compaction", compact_threshold: 200000}] compacts on the server and returns an encrypted compaction item; Codex has /compact and an auto-compact threshold.

Week 2 · Part 4 · Using it37

A $200 subscription is a bundle of tokens

PlanPriceWhat it promises
Claude Max 20x$200/mo20× Pro's usage per 5-hour session, plus a weekly cap
ChatGPT Pro$200/mo10× Plus for Codex; no 5-hour limit (the old $200 plan was 20×)

Neither company states the limits in tokens or dollars. Both include their coding agents, Claude Code and Codex.

$200 at API prices buys about

ModelTokens
gpt-6.1-sol565 M
Claude Opus 5.5265 M
gpt-6-astra90 M

Our estimate for a typical agent mix: 93% cache reads, 5% cache writes, 0.5% fresh input, 1.5% output.

support.claude.com (Max plan) · claude.com/pricing · learn.chatgpt.com/docs/pricing · TechCrunch 2026-04-09 · OfficeChai (20× to 10×, not confirmed on OpenAI's site)

Week 2 · Part 4 · Using it38

Live data: what heavy users pull from a $200 plan

viberank.app ranks developers by the API-equivalent cost of their agent logs (via ccusage). September 2026:

viberank.app, AI coding spend, September 2026

Median $6,091 a month at API prices: about 30 times a $200 plan.

97% of tokens are cache reads. Caching (Part 2) is what makes flat-rate plans possible.

Read with care: self-reported, self-selected heavy users, priced at list rates.

viberank.app/stats/monthly/2026-09 (live; checked 2026-10-07) · try it on yourself: npx ccusage

Week 2 · Part 4 · Using it39

Before your first call

  • The API is not ChatGPT. A separate account, billed per token, with its own key.
  • Keys live in an environment variable, OPENAI_API_KEY, never in code or a commit. Our homework repos are public, and bots scan GitHub for keys.
  • Set a budget first. An agent loop with a bug can spend all night.
  • Stream long answers (stream: true): same price, no waiting on a blank screen.
You seeIt means
401Bad or missing key
400Bad request: read the message, it names the field
429Rate limit: wait and retry; the SDK retries for you
5xxThe provider's problem: retry later
incompleteHit max_output_tokens, maybe mid-thought
Week 2 · Part 4 · Using it40

Research hygiene

Log every call

Exact model ID, every parameter, the full request and response, and usage. Model names move and old models retire.

Expect variation

One prompt, 1,000 runs at temperature 0: 80 different answers, first split at token 103. Run important calls several times.

Know where data goes

Responses are stored by default (store: true, at least 30 days). Check before sending unpublished or personal data.

Untrusted input

A web page the agent reads lands in the same token stream as your instructions, and can carry instructions of its own.

Variation: Thinking Machines, "Defeating Nondeterminism in LLM Inference", Sept 2025 (Qwen3-235B). Cause: each request is batched with a different number of others.

Week 2 · Part 4 · Using it41

Takeaways

  • An API call is stateless JSON in, JSON out; the model only ever sees one stream of tokens.
  • A turn ends in text or a tool call; strict mode masks the softmax so the JSON is valid.
  • An agent is a loop: run the call, append the result, call again.
  • Output costs ~5× input; a cached read costs 1/10 to 1/50. Keep stable things first.
  • Context only grows: compaction, sub-agents and memory files keep it small, at the risk of forgetting.