When Do Execution Records Support a Covert-Channel Attribution? An Evidence-Sufficiency Test for LLM Auditors

Good: The use of matched cases with identical observable evidence but different underlying mechanisms is particularly strong. It directly tests whether auditors recognize when evidence is insufficient to establish causal dependence, rather than simply inferring it from matching actions. Separately measuring unsupported positive and negative conclusions also makes the evaluation more informative than aggregate accuracy alone.

Weakness: Because the auditors are given trusted execution records, a limited set of possible mechanisms, and explicit criteria for what constitutes sufficient evidence, much of the task reduces to applying predefined decision rules rather than independently establishing causality. In real investigations, the harder challenge is determining whether evidence is reliable and complete, identifying alternative explanations, and establishing whether an observed action genuinely depended on recovered information when the underlying mechanism is unknown.

Recommendation: Extend the benchmark with controlled evidence failures: remove critical records, introduce contradictory observations, manipulate individual execution logs, and vary whether counterfactual trials actually isolate the claimed cause. Measure whether auditors correctly identify the specific evidentiary gap, request the information needed to resolve it, and revise their conclusions when new evidence becomes available. This would test investigative reasoning and robustness beyond following predefined evidentiary rules.