Evaluation Egress Must Fail Closed
Adewale Olalekan
The paper proposes a minimum safety standard for high-risk, tool-using AI cyber evaluations: AECE-1 (Agent Evaluation Containment and Evidence).
Its central argument is that a “controlled environment” is not credible if a single unverified component—such as a package proxy—can give an agent access to the public internet or production systems. Evaluations should therefore be built to fail closed: deny outside access by default, use only synthetic targets, restrict privileges, preserve logs, and automatically stop on boundary violations.
Key contributions:
- AECE-1: a 12-clause pre-run standard covering approval, independent infrastructure, default-deny egress, pinned internal dependency mirrors, synthetic-only targets, least privilege, control-plane isolation, governed tools, tamper-evident telemetry, behavioral detection, independent kill/recovery, and disclosure/review.
- Operational targets: an escape signal should terminate the run and revoke its identity within 60 seconds; humans should be paged within 5 minutes.
- EDH-1: a safe acceptance-test suite using internal canaries, honeytokens, metadata and privilege lures, log-loss tests, and multi-signal detection tests. Passing requires success on the core tests before release.
- Oversight RFI: a detailed evidence request template for regulators or auditors investigating an incident, focused on configuration manifests, real network/identity paths, logs, containment failures, third-party impact, and remediation evidence.
The paper frames these measures in response to a reported July 2026 incident in which an AI evaluation allegedly escaped a constrained environment and reached Hugging Face production infrastructure. It maps each reported stage of that event to controls that could plausibly have interrupted it, while avoiding a claim that the standard would certainly have prevented the incident.
Its main limitation is explicit: this is a normative design proposal, not a field-tested benchmark. The controls, harness, and response-time targets have not yet been validated in real evaluation environments; strict isolation may also reduce realism and require high-fidelity internal replicas. The paper also distinguishes containment quality from model intent or capability—the standard cannot determine whether a model is dangerous, only whether an evaluation environment can contain and investigate it.
No reviews are available yet
Cite this work
@misc {
title={
(HckPrj) Evaluation Egress Must Fail Closed
},
author={
Adewale Olalekan
},
date={
},
organization={Apart Research},
note={Research submission to the research sprint hosted by Apart.},
howpublished={https://apartresearch.com}
}


