Structural Code Monitors: Graph Matching Networks on Code Property Graphs for Topology-Aware Trusted Monitoring

Authors apply well-established static analysis tools to a relevant AI Safety problem — verifying code submitted by untrusted agents. Core results are compelling, and the paper is executed and presented well — I have greatly enjoyed the read.

However, if I have not missed anything, the method's practical applicability faces a fundamental chicken-and-egg problem. The GNN requires a trusted model capable enough to produce correct, parseable reference code. If I have this code already, why use untrusted model to generate another version of it? Especially since any clever algorithmic optimisations that significantly change the CPG will likely be flagged (or, if not, could contain backdoors), and will have to undergo a manual review anyways. I imagine that GPT-4o-mini could already saturate the subset of ControlArena tasks chosen by the authors, and that is why this method appears to work as weak supervision.

I, however, imagine this method could be used for, say, evaluation frameworks, like ControlArena itself. A limited set of reference solutions could be created to be compared against, which could allow to catch misaligned agents seeking to introduce backdoors more reliably.

Good work overall, and great execution!

It’s really refreshing to read an approach to AI control that genuinely goes beyond LLMs. I hope that this inspires others to think more broadly about which tools can be used for AI control. The improvements in recall over LLM baselines, particularly at the most relevant operating point of very low false positive rates, are extremely impressive and at the same time very plausible given the approach taken. I strongly encourage the authors to build on this project and publish their research! I would like to see this applied to more varied attack scenarios and see the potential for complementarity to LLM-based monitors be explored in more depth.