Auditing the exported p value of a neural information detector
Sidharth Rajmohan
Submitted to AI Collusion Research Sprint. Sprint projects are early-stage work by participants, not Apart Research publications.
A detector's descriptive score and its reported significance need separate validation. I audit the standalone neural mutual-information p-value exported by a pinned implementation of Audit the Whisper, a detector reused in a subsequent collusion study. Its nominal-null fits retain the observed joint sample while changing only the product-reference sample, and use 120 optimization steps; the observed fit uses 200. An exact constructed example shows that this reference can retain the dependence being tested. A separate, predeclared neural mechanism experiment varies joint construction and training budget across 40 synthetic datasets. Under correlation 0.9, equal-budget reference means are 0.52703 nats when the joint is retained and 0.01686 when it is permuted. These values describe a local implementation. Operational performance and Gaussian-tail calibration from three fits per run pair require separate evaluation. I propose a local output-only quarantine that preserves diagnostic scores while marking inferential output unavailable. A bounded synthetic execution of the pinned detector and estimator with local adapters preserves scores under the patch; full-pipeline integration and calibration remain untested. A source trace distinguishes a stress-script consumer of the exported value from the default union, which excludes this detector, and the raw-score ROC and externally derived honest-calibration operating thresholds. The audit provides a source-level diagnosis and regression checks for the proposed quarantine. A calibrated replacement test remains open.
Reviews
No public critique yet.
Cite this project
@misc{rajmohan2026auditing,
title = {{Auditing the exported p value of a neural information detector}},
author = {Sidharth Rajmohan},
year = {2026},
month = oct,
note = {Submitted to AI Collusion Research Sprint, an Apart Research Sprint},
howpublished = {\url{https://apartresearch.com/sprints/projects/auditing-the-exported-p-value-of-a-neural-information-detector-lkqf}},
url = {https://apartresearch.com/sprints/projects/auditing-the-exported-p-value-of-a-neural-information-detector-lkqf}
}More from AI Collusion Research Sprint
- View project: Signal or Coincidence ? Causal Auditing of Collusion Through Shared Agent State
Signal or Coincidence ? Causal Auditing of Collusion Through Shared Agent State
Signal or Coincidence
How can we distinguish genuine coordination between AI agents from coincidental behavioral alignment? We introduce a randomized canary audit that tests whether private information propagates through shared tool state, …
- View project: (1.4) Collusion, Espionage, or Look-alikes? Auditing Information Flow between LLM Agents
(1.4) Collusion, Espionage, or Look-alikes? Auditing Information Flow between LLM Agents
Bid Brother
Can an overseer find two secretly coordinating AI agents just by watching what they do? We tested pairwise net influence (Bonjour et al., 2022) on LLM bidders in a repeated procurement auction, with instructed cartels …
- View project: AI Sellers Propose Bid-Rigging Within Minutes, but Their Agreements Do Not Hold
AI Sellers Propose Bid-Rigging Within Minutes, but Their Agreements Do Not Hold
Fugue
AI agents are starting to negotiate and bid for competing companies. We follow cartels among them from the first proposal to the price the buyer pays, in a sealed-bid procurement market where twelve LLM agents act as …