Jun 2, 2025
Guardian-Loop: Mechanistically Interpretable Micro-Judges with Adversarial Self-Improvement
Efstathios Siatras, Man Kit Chan
🏆 4th place by peer review
Guardian-Loop is a mechanistically interpretable judge system designed to enhance the Expert Orchestration Architecture through transparent and efficient safety evaluation. Targeting Track 1 (Judge Model Development), we train lightweight classifiers that pre-filter prompts for safety using a Llama 3.1 8B model, fine-tuning only the upper layers to directly output True or False responses. This avoids probe-head architectures, enabling native token-level interpretability and calibrated scoring. Achieving 85.0% accuracy and 94.6% AUC-ROC on a hold-out test set with low latency using the safety judge, the system is deployable on consumer hardware. Guardian-Loop integrates deep interpretability techniques, including token attribution, attention analysis, and circuit tracing, to expose the model’s internal decision-making; We also demonstrate the extensibility of our framework by applying it to adjacent judgment tasks, such as feasibility prediction. An open-ended adversarial framework based on MAP-Elites quality diversity optimization was proposed, designed to populate a 10×10 grid spanning risk types and evasion strategies. While not yet deployed, this framework could support continuous self-improvement and vulnerability discovery. Guardian-Loop illustrates how small-sized LLMs can be repurposed as efficient, transparent filters, supporting scalable and trustworthy AI deployments.
The safety pre-filter concept addresses a real need for customizable, interpretable safety layers. The mech interp visualizations are well-executed and the technical implementation seems solid. I'd be happy to see follow-up work on this!
Some suggestions to strengthen this work:
* Doing real adversarial testing would be important even in a PoC work on safety filters
* Re motivation: When is a separate safety layer actually better than built-in model safety? The efficiency cost of two inference passes needs clearer justification. Think about and zoom in on domains where customized safety rules matter a lot.
* Also: can you connect this more to actual model orchestration? Right now it's more of a gate than a routing component. Showing how this could enable cascade routing (quick decisions on obvious cases, deeper analysis only when needed) may make it more realistic.
* The feasibility judge feels a bit like scope creep. It might be better to leave it out of the discussion and instead double down on the safety judge model.
The safety and feasibility judges are decent enough work, albeit fairly straightforward finetunes. The adversarial framework is a very interesting proposal and the highlight of report for me, it’s regrettable you ran out of time to explore this.
I’d suggest for exploration, some form of combinatorial bandit is probably going to be higher performing than the simple temperature sampling you mention.
Cite this work
@misc {
title={
(HckPrj) Guardian-Loop: Mechanistically Interpretable Micro-Judges with Adversarial Self-Improvement
},
author={
Efstathios Siatras, Man Kit Chan
},
date={
6/2/25
},
organization={Apart Research},
note={Research submission to the research sprint hosted by Apart.},
howpublished={https://apartresearch.com}
}
