JASONTEST Guardian-Loop: Mechanistically Interpretable Micro-Judges with Adversarial Self-Improvement
Efstathios Siatras, Man Kit Chan · Team JASONTEST Guardian Loop
Submitted to Apart x Martian Mechanistic Router Interpretability Hackathon. Sprint projects are early-stage work by participants, not Apart Research publications.
Guardian-Loop is a mechanistically interpretable judge system designed to enhance the Expert Orchestration Architecture through transparent and efficient safety evaluation. Targeting Track 1 (Judge Model Development), we train lightweight classifiers that pre-filter prompts for safety using a Llama 3.1 8B model, fine-tuning only the upper layers to directly output True or False responses. This avoids probe-head architectures, enabling native token-level interpretability and calibrated scoring. Achieving 85.0% accuracy and 94.6% AUC-ROC on a hold-out test set with low latency using the safety judge, the system is deployable on consumer hardware. Guardian-Loop integrates deep interpretability techniques, including token attribution, attention analysis, and circuit tracing, to expose the model’s internal decision-making; We also demonstrate the extensibility of our framework by applying it to adjacent judgment tasks, such as feasibility prediction. An open-ended adversarial framework based on MAP-Elites quality diversity optimization was proposed, designed to populate a 10×10 grid spanning risk types and evasion strategies. While not yet deployed, this framework could support continuous self-improvement and vulnerability discovery. Guardian-Loop illustrates how small-sized LLMs can be repurposed as efficient, transparent filters, supporting scalable and trustworthy AI deployments.
Reviews
The safety pre-filter concept addresses a real need for customizable, interpretable safety layers. The mech interp visualizations are well-executed and the technical implementation seems solid. I'd be happy to see follow-up work on this!
Some suggestions to strengthen this work:
* Doing real adversarial testing would be important even in a PoC work on safety filters
* Re motivation: When is a separate safety layer actually better than built-in model safety? The efficiency cost of two inference passes needs clearer justification. Think about and zoom in on domains where customized safety rules matter a lot.
* Also: can you connect this more to actual model orchestration? Right now it's more of a gate than a routing component. Showing how this could enable cascade routing (quick decisions on obvious cases, deeper analysis only when needed) may make it more realistic.
* The feasibility judge feels a bit like scope creep. It might be better to leave it out of the discussion and instead double down on the safety judge model.
Read full reviewShow less
The safety and feasibility judges are decent enough work, albeit fairly straightforward finetunes. The adversarial framework is a very interesting proposal and the highlight of report for me, it’s regrettable you ran out of time to explore this.
I’d suggest for exploration, some form of combinatorial bandit is probably going to be higher performing than the simple temperature sampling you mention.
Cite this project
@misc{siatras2025jasontest,
title = {{JASONTEST Guardian-Loop: Mechanistically Interpretable Micro-Judges with Adversarial Self-Improvement}},
author = {Efstathios Siatras and Man Kit Chan},
year = {2025},
month = jun,
note = {Submitted to Apart x Martian Mechanistic Router Interpretability Hackathon, an Apart Research Sprint},
howpublished = {\url{https://apartresearch.com/sprints/projects/jasontest-guardianloop-mechanistically-interpretable-microjudges-with-adversarial-selfimprovement-1b2p}},
url = {https://apartresearch.com/sprints/projects/jasontest-guardianloop-mechanistically-interpretable-microjudges-with-adversarial-selfimprovement-1b2p}
}More from Apart x Martian Mechanistic Router Interpretability Hackathon
- 1st place by peer reviewView project: Manipulating Self-Preference for Large Language Models
Manipulating Self-Preference for Large Language Models
Team Preference
Large language models (LLMs) carry great value as evaluators of synthetic data for research and production settings. However, recent research shows that language models exhibit bias towards their own responses in blind …
- 2nd place by peer reviewView project: Approximating Human Preferences Using a Multi-Judge Learned System
Approximating Human Preferences Using a Multi-Judge Learned System
AutoBox
In this work, we introduced a learned approach to aggregating multi-judge scores: using a GAM and a simple MLP as an alternative to traditional, non-learned methods like averaging. Our models outperform the naive …
- 3rd place by peer reviewView project: Judge using SAE Features
Judge using SAE Features
SAEwhat?
The key idea of this project was to explore model judgement using Sparse Autoencoder (SAE) features for mathematical reasoning tasks involving addition, multiplication, and subtraction operations. We compared this …