Containment Without Cooperation: A Verifiable Standard for Third-Party Platforms Hosting AI Evaluation Traffic

This paper offers a useful framework for platform-side containment checks, distinguishing those that require declared agent identities from those that can operate without developer cooperation. Its illustrative incident analysis makes that distinction concrete, while the claims about earlier detection and demonstrated rule violations need stronger support.

Strength: The distinction between checks requiring developer cooperation and checks an affected platform can perform independently is useful. It directs attention to defenses that remain available when an agent's operator has not declared its activity.

Recommendation:

• Clarify what the standard establishes. Detecting suspicious behavior and demonstrating that a platform failed its response obligations are different outcomes. For example, a volume spike followed by the required throttling satisfies the response rule, even though suspicious traffic occurred. Organize the standard around the observation, required action, and evidence of compliance so its verdicts have a clear security meaning.
• Evaluate the approach from the platform's actual vantage point. The early-detection claim currently uses campaign totals for a rule applied to individual identities. Test what the platform could observe and act on at the time, including legitimate high-volume activity. This would establish whether the rules support useful intervention without excessive disruption.

This is a thoughtful, well motivated piece of work on how to keep AI systems contained. Its best insight is that checking whether an AI has been kept in check depends partly on who holds the evidence, not just on how carefully the rules are written. The authors suggest that the outside platform an AI actually interacts with is a better source of proof than the AI lab's own records, and that's a useful alternative to simply trusting the lab.

The idea of splitting the rules into two groups is especially helpful. Some rules only work if the AI openly declares that it's an AI being tested. Others cover behavior that can be spotted whether or not the AI cooperates. Spelling this out makes the assumptions behind the standard clear. In the test, the two rules that don't depend on the AI's cooperation triggered, and the two that do were marked "not applicable." That's an informative outcome, not a flaw.

The researcher also built a working example and used it to reconstruct a real incident, which makes the proposal concrete. They were careful not to oversell the 500 actions per day limit, making clear it's just an illustrative number and not a proven real world setting.

The main weakness is that the framework was only checked against the same incident it was built from. To show it works more broadly, the researcher should test it on other, independent containment incidents, tune the behavioural limits using normal, legitimate platform traffic, and try it in a real deployment where AI declare their identities. Overall, this is a very promising framework with a valuable central idea and a clear path to stronger evidence.

I think this project addresses an important gap because agents can affect platforms that had no involvement in the evaluation. Those platforms need ways to investigate the activity without depending on frontier lab’s account of what happened. The paper proposes monitoring whether agents exceed their permissions, access another customer’s resources, switch between suspicious accounts, or make too many requests.

I am less convinced that the implementation demonstrates reliable detection as this mainly relied on public incident reports and several parts are told which activity is suspicious instead of identifying it themselves. The framework may be useful, but the example does not yet show whether it can distinguish violations from ordinary use. Testing a specific proposed feature against realistic traffic would provide stronger evidence without requiring full deployment. For example I would want to see whether its request volume alerts help identify agents without also flagging ordinary users making legitimate requests.