Homework 2 · Results

Every harness, installed and run

We gave each submitted skill to an AI agent on a fresh Linux computer (a Docker container), let it install and configure the harness exactly as the skill says, and put every harness through the same seven tests. Each one talked to the same model, Claude Haiku 5.5, through a test key that worked only for these runs.

6harnesses submitted by October 14
6installed from the skill as written, on a fresh computer
6do all six things on the list

The seven tests

  1. "What files are in this folder?" One command, approved.
  2. "How much free disk space do I have, and which folder in my home directory is biggest?" Several commands in a row.
  3. Run a program that doesn't exist. The model should see the error and explain it.
  4. Deny a command. The model should be told you declined, and not try to sneak around it.
  5. sleep 200. The tool should stop it at its time limit.
  6. seq 1 200000: 1.3 million characters. The tool should cut it short.
  7. Token counts after every call: input, cached input when reported, and output.

We also read the code: settings from environment variables, the loop, the tool, dependencies, and whether any key was ever committed. None was.

At a glance

HarnessBuilt withSkill worksAPI via envLoopbash: limit, cutApprove / denyChatTokens
psp-agent-harness-ding
@dingyl2001
Flask, openai · ~385 lines✓✓✓✓✓web + terminal✓
ai-study-agent
@WenyuZhang2001
standard library · ~330 lines✓✓✓✓✓web✓
research-agent
@hleinweber
standard library · ~540 lines✓✓✓✓✓web + terminal✓
pedro-agent
@Caltz27
standard library · ~270 lines✓✓✓✓✓web✓
reiniery-agent-harness
@Reiniery
Flask, openai · ~340 lines + page✓✓✓✓✓web + terminal✓
kai-agent
@kaijaffarove
standard library · ~356 lines✓✓✓✓✓web✓

Three lessons from six harnesses

1. A time limit must stop the whole command. Two harnesses reported "timed out after 60 s" while sleep 200 kept running in the background: the timeout killed the shell, not the program it started. Start the command in its own process group (start_new_session=True) and kill the group (os.killpg) on timeout.

2. Say what you cut. Everyone cut long output, keeping the last 5,000–10,000 characters. When nothing in the text says so, the model gets confused: twice the first line it saw was 572 (the end of 198572), and one model concluded that "lines 1–571 were dropped". A marker such as [... 1,278,893 characters omitted ...], and keeping the start as well as the end, fixes it. It's the week 3 slide "Write tools for a reader with no context" in practice: the returned text is a prompt too.

3. Two ways to call a tool, both fine. Five harnesses use the API's built-in tool calling (tools and tool_calls). kai-agent asks the model to write <bash>…</bash> in its reply and parses the tag itself: exactly "a tool call is just text your program acts on". Built-in tool calls will be easier once your harness has many tools and MCP servers, which starts with homework 3.

Each harness

psp-agent-harness-ding

@dingyl2001 · Flask web chat and a terminal mode · the skill worked exactly as written

  • Well done: a clean, small loop with a 20-round cap; Approve/Deny cards on the web and a [y/N] prompt in the terminal; token counts including cached input on every call.
  • Next: kill the whole process group on timeout, mark where output was cut, and catch a malformed tool call (bad JSON) so the loop can recover instead of erroring.

ai-study-agent

@WenyuZhang2001 · standard library only, web chat · the skill worked exactly as written

  • Well done: no dependencies at all; it ran two commands in parallel in one round; the timeout kills the whole process group; the page checks the host and origin and uses a session token.
  • Next: the buttons read 批准 / 拒绝 while the skill says Approve / Deny, so match the two; keep the start of long output as well as the end; consider showing cache writes next to cached input.

research-agent

@hleinweber · standard library only, web chat and terminal · the skill worked exactly as written

  • Well done: step-by-step files (one_call.py, chat.py) that show how it was built; the timeout kills the process group; output goes to a temporary file so a huge command can't fill memory; approvals expire safely.
  • Next: say in the text when output was cut or a command timed out (not only in a JSON flag); show a short, cleaned excerpt of the provider's error message; add a small pyproject.toml with requires-python.

pedro-agent

@Caltz27 · standard library only, web chat · the skill worked as written (our test computer had no nano, so the agent wrote .env directly)

  • Well done: check_api.py tests the endpoint and tool calling before anything else, and three unit tests pass; the timeout left no stray process; the page checks the host and uses a session token; approvals expire after 5 minutes.
  • Next: mark where output was cut and keep its start; let it run without a .env file when the variables are already set; put the limits (60 s, 10,000 characters) in the tool's description so the model can plan around them.

reiniery-agent-harness

@Reiniery · Flask web chat and a terminal mode · the skill worked exactly as written

  • Well done: each of several tool calls in one reply gets its own approval; a smoke test checks that the endpoint supports tool calls; the model handled a denial gracefully ("I won't retry it unless you ask").
  • Next: kill the whole process group on timeout; keep both ends of long output; a "New chat" button to clear the conversation.

kai-agent

@kaijaffarove · standard library only, web chat · the skill worked as written with the OpenAI-compatible backend

  • Well done: tool calls as plain text tags, parsed by the harness, which makes the idea of a tool call visible; a running token total per chat; a clear "[output cut to the last 10000 characters]" marker.
  • Next: make the OpenAI-compatible backend the default (the default today runs claude -p); plan a move to the API's built-in tool calls before your harness gains more tools; keep the start of long output too.

What's next for your harness

Homework 3 builds on exactly this program: it learns to load plugins, and its first plugin lets you talk to it from your phone. The fixes above, especially the time limit, matter more once you approve commands from a phone.