Homework 6
Make your task hold
Due before class on Wednesday, November 18, 2026
In week 1 you built the hardest task you could, and Claude Opus 5.5 solved 6 of the 8 tasks, five of them in under 7 minutes. You now know much more about how these agents work and how they fail. Go back to your task and use it. The goal: a task that GPT-5.6 Sol doesn't solve. Aim for one that would take a human expert more than a year, ideally ten. It has to stay solvable: a right answer exists, what the agent gets determines it, and your verifier can tell right from wrong. Making a task impossible, vague or unfair is easy, and it measures nothing.
No week 1 task yet? That's fine. Build one with the week 1 guide, aiming at this goal from the start, then come back here.
Start
- Open an AI coding agentClaude Code, Codex, Cursor, GitHub Copilot, … in your clone of the homework repository.
- Give it this
Read https://class.xhelio.ai/week-6/homework.md and help me with my homework. - Work through it togetherIt goes through what happened to your task, has a fresh session review its solvability, hardens it with you, and opens your pull request. You decide the science.
You'll need your week 1 task, uv, the GitHub CLI and an AI coding agent. No Docker: the checks run on GitHub.
What you do
- Read what happenedYour task's page on the homework board and the agent's full trace. What was the decisive move? If it failed, was that the science or your task?
- Fix its solvabilityA new session of your agent, which doesn't know your reasoning, reviews the task. Fix what it finds: hidden conventions, ranges that don't cover the hidden data, a key built on one expert's unstated choices, a bar a no-skill answer clears, shortcuts.
- Harden itChange the task so the agent fails on the science. Think in years of expert time: build on a result that took your group or your field that long. If the agent solved your task quickly, keep your science and change the task's structure.
- Review it againHardening is how good tasks become unfair. Another new session reviews the final version.
- Submit, and we run GPT-5.6 SolOpen your pull request. We run it once at
xhighand post the result and its trace there. You don't need Docker or an API key. A run can take up to your task's time limit, so submit early enough to act on the result. - Read the attemptsA failure counts only if the agent had everything it needed. If it solved the task, find out how, go back to step 3, and ask for another run.
What tends to work
- Withhold the graded truth. With no real check, the agent has only its own simulator or cross-validation.
- Use real, observed data, with the gaps, artifacts and systematics no one wrote down.
- Ask for the hidden cause: it sees effects and recovers causes, parameters or history.
- Need a term it won't assume: an instrument response, a coupling, a background.
- Grade beyond the visible range, with the hidden range stated.
- Chain the steps, and grade the end, so an early mistake corrupts the answer.
What doesn't
- More data or more cases. Effort grows with the number of distinct methods, not the size of the data. Ten years of repeating one step can take an agent minutes.
- A huge search space with a cheap check. Agents search very well when they can check.
- A recipe in the instruction. It then tests care, which coding agents do best.
- Blocking the one shortcut you saw. There's usually another one.
- Hidden conventions, vague wording, one obscure fact. That's hard for the wrong reason.
What you submit
A pull request that updates your task in place, in week-1/submissions/<your-github-username>/<task-name>/: the hardened task, authoring/solvability.md (the review of the final version), authoring/attempts.md (one row per run), and updated README sections, expert_time_estimate_hours (a year is about 2,000 hours) and class_level.
It counts when your reference solution scores 1, doing nothing scores 0, the automatic checks pass, and the final review says solvable.
Still solved at the deadline?
That still counts. Record the run that beat it and what you'd try next. An honest task that was solved teaches more than an unfair one that wasn't.
Holding against Sol is the floor. If you get there, ask us for a stronger model,.
Take it to Terminal-Bench-Science
You're encouraged to submit your task to Terminal-Bench-Science 2.0, the public benchmark whose format this class uses. It's optional, and it's how this work can earn you credit beyond the class: each merged task earns one co-authorship point, split equally between its contributors, and one point qualifies you for co-authorship on the paper. Submissions close on December 15, 2026.
- Propose first, through their Task Proposal Form. Once it's approved, you build the task and open a pull request there. It goes through human review, and merging is at the project lead's discretion.
- Aim past Sol. They want tasks that today's best agents can't reliably solve, and they currently test with Claude Opus 5.5 and GPT-6 Astra. If your task holds against Sol, ask us in your pull request to try a stronger model.
- Read their bar. Have your agent check your task against their proposal rubric and contributing guide.
- Say where it came from. They ask you to declare a task's provenance and prior use, so say it began as homework here. Tasks must be authored by domain experts working in the task's field.
More
- The assignment on GitHubThe full text.
- The solvability skillThe eleven checks a fresh session runs on your task.
- The hardening guideHow agents fail, and what to do about it.
- What your agent readsPlain text; the full guide is in the repository.
- An RL environment and an eval are the same object, from the week 8 slidesAnd the slides after it, on when evals break.
- All homeworkYour task and your agent, all quarter.