Auditing the exported p value of a neural information detector
Sidharth Rajmohan
Submitted to AI Collusion Research Sprint. Sprint projects are early-stage work by participants, not Apart Research publications.
A detector's descriptive score and its reported significance need separate validation. I audit the standalone neural mutual-information p-value exported by a pinned implementation of Audit the Whisper, a detector reused in a subsequent collusion study. Its nominal-null fits retain the observed joint sample while changing only the product-reference sample, and use 120 optimization steps; the observed fit uses 200. An exact constructed example shows that this reference can retain the dependence being tested. A separate, predeclared neural mechanism experiment varies joint construction and training budget across 40 synthetic datasets. Under correlation 0.9, equal-budget reference means are 0.52703 nats when the joint is retained and 0.01686 when it is permuted. These values describe a local implementation. Operational performance and Gaussian-tail calibration from three fits per run pair require separate evaluation. I propose a local output-only quarantine that preserves diagnostic scores while marking inferential output unavailable. A bounded synthetic execution of the pinned detector and estimator with local adapters preserves scores under the patch; full-pipeline integration and calibration remain untested. A source trace distinguishes a stress-script consumer of the exported value from the default union, which excludes this detector, and the raw-score ROC and externally derived honest-calibration operating thresholds. The audit provides a source-level diagnosis and regression checks for the proposed quarantine. A calibrated replacement test remains open.
Reviews
No public critique yet.
Cite this project
@misc{rajmohan2026auditing,
title = {{Auditing the exported p value of a neural information detector}},
author = {Sidharth Rajmohan},
year = {2026},
month = oct,
note = {Submitted to AI Collusion Research Sprint, an Apart Research Sprint},
howpublished = {\url{https://apartresearch.com/sprints/projects/auditing-the-exported-p-value-of-a-neural-information-detector-lkqf}},
url = {https://apartresearch.com/sprints/projects/auditing-the-exported-p-value-of-a-neural-information-detector-lkqf}
}More from AI Collusion Research Sprint
- View project: Is It a Cartel? Separating Enforced from Spurious Collusion in LLM Pricing Agents
Is It a Cartel? Separating Enforced from Spurious Collusion in LLM Pricing Agents
When LLM pricing agents keep prices above the competitive level, are they running a cartel, or are they simply stuck? Only the first is collusion, and the two call for opposite remedies. We introduce a black-box causal …
- View project: Every Permission Checked: Source-Linked Evidence for Auditing Finance Agents
Every Permission Checked: Source-Linked Evidence for Auditing Finance Agents
When multiple AI agents contribute to a policy violation, the same outcome can reflect cooperation, deception of a helper, or an independent mistake. We study how evidence presentation affects this distinction in a …
- View project: Swapping the Model Does Not Reset the Price: What LLM Pricing Agents Inherit After a Model Replacement
Swapping the Model Does Not Reset the Price: What LLM Pricing Agents Inherit After a Model Replacement
Emerald
We test what happens when one of two LLM pricing agents has its model replaced, the open problem posed in the sprint's question 1.2. Two gpt-oss-120b agents priced for 150 rounds; we then swapped one agent's model to …