Homework 6

Make your task hold

Due before class on Wednesday, November 18, 2026

In week 1 you built the hardest task you could, and Claude Opus 5.5 solved 6 of the 8 tasks, five of them in under 7 minutes. You now know much more about how these agents work and how they fail. Go back to your task and use it. The goal: a task that GPT-5.6 Sol doesn't solve. Aim for one that would take a human expert more than a year, ideally ten. It has to stay solvable: a right answer exists, what the agent gets determines it, and your verifier can tell right from wrong. Making a task impossible, vague or unfair is easy, and it measures nothing.

No week 1 task yet? That's fine. Build one with the week 1 guide, aiming at this goal from the start, then come back here.

Start

  1. Open an AI coding agentClaude Code, Codex, Cursor, GitHub Copilot, … in your clone of the homework repository.
  2. Give it this Read https://class.xhelio.ai/week-6/homework.md and help me with my homework.
  3. Work through it togetherIt goes through what happened to your task, has a fresh session review its solvability, hardens it with you, and opens your pull request. You decide the science.

You'll need your week 1 task, uv, the GitHub CLI and an AI coding agent. No Docker: the checks run on GitHub.

What you do

  1. Read what happenedYour task's page on the homework board and the agent's full trace. What was the decisive move? If it failed, was that the science or your task?
  2. Fix its solvabilityA new session of your agent, which doesn't know your reasoning, reviews the task. Fix what it finds: hidden conventions, ranges that don't cover the hidden data, a key built on one expert's unstated choices, a bar a no-skill answer clears, shortcuts.
  3. Harden itChange the task so the agent fails on the science. Think in years of expert time: build on a result that took your group or your field that long. If the agent solved your task quickly, keep your science and change the task's structure.
  4. Review it againHardening is how good tasks become unfair. Another new session reviews the final version.
  5. Submit, and we run GPT-5.6 SolOpen your pull request. We run it once at xhigh and post the result and its trace there. You don't need Docker or an API key. A run can take up to your task's time limit, so submit early enough to act on the result.
  6. Read the attemptsA failure counts only if the agent had everything it needed. If it solved the task, find out how, go back to step 3, and ask for another run.

What tends to work

What doesn't

What you submit

A pull request that updates your task in place, in week-1/submissions/<your-github-username>/<task-name>/: the hardened task, authoring/solvability.md (the review of the final version), authoring/attempts.md (one row per run), and updated README sections, expert_time_estimate_hours (a year is about 2,000 hours) and class_level.

It counts when your reference solution scores 1, doing nothing scores 0, the automatic checks pass, and the final review says solvable.

Still solved at the deadline?

That still counts. Record the run that beat it and what you'd try next. An honest task that was solved teaches more than an unfair one that wasn't.

Holding against Sol is the floor. If you get there, ask us for a stronger model,.

Take it to Terminal-Bench-Science

You're encouraged to submit your task to Terminal-Bench-Science 2.0, the public benchmark whose format this class uses. It's optional, and it's how this work can earn you credit beyond the class: each merged task earns one co-authorship point, split equally between its contributors, and one point qualifies you for co-authorship on the paper. Submissions close on December 15, 2026.