How Detectable Are Covert Signals in AI Agent Transcripts?

Overall a high quality submission which takes a rigorous scientific approach towards understanding "sneakiness" of covert token side-tasks. The presentation of the suspicion scores was not super accessible, but was still understandable given time. Limitations are called out explicitly and proactively, which is great to see (Sections 4.5 and 6). My main critique would be that some design decisions are not sufficiently motivated. Below are additional points I wrote during the review process.

* It seems that citations are missing

* Why have 30 benign and 25 of each other category? Seems odd...

* It is unclear to me how the stealth levels were defined, which makes the non-monotonic distinction somewhat arbitrary; it's still an interesting finding, but I don't have a good understanding of why Level 2 should be considered as more stealthy than Level 1 from a semantic perspective.

* How was it confirmed that command-level tells were entirely absent?

* Based on the prompt provided to the monitor and the fact that the success rate for catching the shell comment is so low, the implication is more likely that the marker string is considered to be an unlikely covert side objective, hidden exfiltration, smuggled secret, or other policy-violating behavior. I would be quite interested to look at performance changes when the marker string was changed in a trivial manner (e.g. EVAL_CANON_CODE_0002) and a non-trivial manner (e.g. an attempt at prompt injection or something like that).

Decent base idea, presentation could be improved and I would have liked to see results for other smaller models (opus level models are too expensive to be used as monitors at a production level). Another interesting result would be if the monitor model could detect covert side-task deception without being prompted to check for it to simulate unknown, unexpected threat detection capability.