Unsolved Tasks Accounted for 93% of Tasks Discussed by Agents

This project makes a useful containment-design contribution by distinguishing between removing an illicit shortcut and removing an agent’s only compliant path to task completion. Its most practical output is an auditable pre-run solvability check: before a relaxed-safeguard evaluation, record that each task can be completed within scope in the exact environment agents will use. The project is strongest as an incident-focused synthesis and operational recommendation rather than as a new general behavioral finding, since closely related work already addresses reward hacking, feasibility awareness, and policy-incompatible tasks. The evidence is appropriately cautious but preliminary: the 13-case corpus is selected on known escape incidents and cannot estimate rates or establish causality, while the experiments use one toy task and only five trials per model-condition; the exploratory positive result is model-specific and weakens after manual review. A strong next step would be a preregistered multi-task study with independent double-coding, larger samples, and separate reporting of explicit forbidden-resource attempts, allowlist near-misses, persistence, and false-positive behavior.

The research standards here are excellent. The report says plainly what it does not claim, puts the evidence against its own argument in the main text, corrected its own coding after checking a source, and reports a null result without spin. That makes the claims easy to trust.

The headline number needs more support. Unsolved tasks likely stay open longer, so they would naturally get discussed more whether or not being impossible changes how agents behave. Accounting for time spent on each task, and having a second person code the incident examples, would strengthen the argument.

The small experiment is informative, but it's far from the real incident setting, and the easy "finish" option it offered may explain why no model broke the rules. That matches what "The Pen and the Door" found in this sprint and is worth presenting as a finding. The pre-run check for whether a task is solvable is the most practical contribution, and I'd lead with it. Henry Papadatos, one of the sprint speakers, is a good reference for where a recommendation like this could land, given SaferAI's work on the EU AI Act Codes of Practice.