Reported or Invented? A One-Sentence Instruction Stops Agents Inventing Data After Silent Tool Failures; a Tool-Side Note Mostly Does No
Houssain
Submitted to AI x Epistemics Research Sprint. Sprint projects are early-stage work by participants, not Apart Research publications.
Tool-using AI agents often answer with confidence after a tool fails silently: the call returns HTTP 200 with an empty, null or truncated body, and the agent reports a value it never received. Earlier work in my open-source harness, reported-or-invented, measured this on four agent frameworks. In this sprint I tested two cheap fixes on the same harness: one sentence in the agent’s instructions telling it to say when a tool result is empty, incomplete or unreadable, and a plain-language note that the tool layer appends when a response holds no data. I ran 672 new runs (four frameworks, two fixes, 84 runs each) on one open-weight model, gpt-oss-20b, and every label was hand-checked. The instruction removed invented values entirely: 38 of 192 affected baseline runs contained an invented value, against 0 of 191 with the instruction. Runs that passed bad data as fact fell from 122 of 192 to 83 of 191, with no false alarms on real data. The tool-side note was weaker and depended on the framework: 110 of 190 runs still passed bad data as fact, and two of the four frameworks did not improve. Neither fix touched failures that look like valid data. Empty lists, shortened lists and zeroed values were reported as fact in 83 or 84 of 84 affected runs under every condition. The result is small and preliminary: one model, three tasks and one wording of each fix
Reviews
No public critique yet.
Cite this project
@misc{houssain2026reported,
title = {{Reported or Invented? A One-Sentence Instruction Stops Agents Inventing Data After Silent Tool Failures; a Tool-Side Note Mostly Does No}},
author = {Houssain},
year = {2026},
month = nov,
note = {Submitted to AI x Epistemics Research Sprint, an Apart Research Sprint},
howpublished = {\url{https://apartresearch.com/sprints/projects/reported-or-invented-a-onesentence-instruction-stops-agents-inventing-data-after-silent-tool-failures-a-toolside-note-mostly-does-no-nt5i6ei5rh}},
url = {https://apartresearch.com/sprints/projects/reported-or-invented-a-onesentence-instruction-stops-agents-inventing-data-after-silent-tool-failures-a-toolside-note-mostly-does-no-nt5i6ei5rh}
}Browse all projects