Failing without refusing: effective yield of open-weight models under forensic artifact load

The question comes straight from the incident and matters to every defender without a lab relationship: if hosted models refuse forensic work, what does the self-hostable fallback actually deliver? Measuring whether a model answers alongside whether it refuses is the right framing, and the redaction ladder with an artifact-absent control is a clean design. I checked your sources and every quote holds, including the Hugging Face timeline's line about Claude Opus and Fable refusing the work.

The practical result, that structural redaction gives more usable answers than raw artifacts, would be valuable if it holds. The silent non-response finding is the more original claim, and it needs the budget control before it can carry the paper's conclusion.

To check the results I looked for the code: the paper says the repository is released with the submission, but there is no link in the PDF or on the project page, and I could not find it on GitHub. Without the harness, item pool and pre-registration contract, none of the numbers can be verified, so please publish them. Your own limitation that 98.8% of non-answers hit the 160-token cap means non-response may be truncation, and the abstract and conclusion should say so until the larger-budget run is done. The methods section also describes things the limitations say did not happen: the three-stage refusal pipeline with LLM judge and 200 hand labels did not run, and the roster of seven models became two. Several settings disagree between sections (200 vs 160 output tokens, 12,288 vs 4,096 context, one card vs two, 4 refusals vs 4 plus three in the calibration set). Writing the methods as what was run would fix most of this.

The paper is well written and the limitations are unusually candid. It is also longer than the 8-page limit, and the most important caveat sits in the limitations while the abstract states the unconditional version. Moving that caveat up and shortening Related Work would make it both shorter and more accurate.

If you take this further, run the same items at a larger generation budget first. It takes under an hour and decides whether the central finding is about the models or about the cap.

This is very relevant work - it uncovers that a defender vetting a fallback model on solely whether it complies would pass it and get burned in a real incident. I would suggest rerunning this at a larger budget (likely takes under an hour). Furthermore, testing five models instead of two would let you actually claim the accuracy-vs-yield rank inversion rather than just report it as suggestive.