Mechanistic Router for Interpretable Agent Orchestration
Henry Luan · Team Gamma Router
Submitted to Apart x Martian Mechanistic Router Interpretability Hackathon. Sprint projects are early-stage work by participants, not Apart Research publications.
This project presents a lightweight and interpretable routing system that selects among multiple reasoning strategies—Zero-Shot, Chain-of-Thought (CoT), Program-Aided Language (PAL), ReAct, and Few-Shot—based on features of the user query. It supports the vision of expert orchestration by treating large language model (LLM) prompting strategies as modular, agent-like components.
We frame the problem as a reinforcement learning task using a custom Gym environment, training a PPO agent on a small synthetic dataset with handcrafted, human-interpretable features. The router achieves ~23% accuracy (above random baseline of 20%), showing early signal that reasoning strategy can be learned and predicted from query structure.
Our prototype leverages the CrewAI framework, allowing seamless generalization to multi-agent setups and agent routing, making it production-aligned. The system supports full agent traceability for debugging and interpretability.
Though built on just 39 examples, this demo shows potential for scaling up with semantic features, local LLMs, and more realistic workloads. Future work includes routing across nested agents, evaluation on local models, and applying mechanistic interpretability to policy analysis.
Reviews
Good proof-of-concept that deliberately favors explainability. This would reach a higher score on mech interp/safety impact if you include adversarial or OOD tests or quantify how safety failures are mitigated using transparent routing.
Cool and highly relevant twist to explore routing to reasoning strategies rather than just model selection! Starting with simple and interpretable features makes sense as a quick way to gain intuition on what could work for this problem. I hope the author continues work on this problem and am sure there are cool insights to be gained!
Main ways to improve imo
* 23% accuracy seems too low for 5 classes with predictive features. Make sure that there is no bug in code, training params or ground truth labels by trying simpler baselines (decision tree, rules, etc)
* Really zoom in on the potential impact. E.g. show concrete examples where routing to different strategies produces dramatically different outcomes in quality/cost/safety
* The importance of using RL here wasn't super clear imo and could be better explained. Simple baselines would help here as well.
Constructive feedback:
Strength:
Good job fleshing out a minimal viable routing setting that is based on RL and “feature vectors”. The architecture is simple and has interpretable design.
Synthetic dataset with labels seem to be a good place to begin.
There are good ideas in the design of these experiments.
Weakness:
Minimal training data to learn from RL
The power is best leveraged to model a somewhat trickier environment after seeing more data.
Expert Orchestration: 4
MI: 2
Technical Imp and reproducibility: 2.5 (Code is available but impact is a bit limited without a more thoughtful experimental design)
Cite this project
@misc{luan2025mechanistic,
title = {{Mechanistic Router for Interpretable Agent Orchestration}},
author = {Henry Luan},
year = {2025},
month = jun,
note = {Submitted to Apart x Martian Mechanistic Router Interpretability Hackathon, an Apart Research Sprint},
howpublished = {\url{https://apartresearch.com/sprints/projects/mechanistic-router-for-interpretable-agent-orchestration-dio5}},
url = {https://apartresearch.com/sprints/projects/mechanistic-router-for-interpretable-agent-orchestration-dio5}
}More from Apart x Martian Mechanistic Router Interpretability Hackathon
- 1st place by peer reviewView project: Manipulating Self-Preference for Large Language Models
Manipulating Self-Preference for Large Language Models
Team Preference
Large language models (LLMs) carry great value as evaluators of synthetic data for research and production settings. However, recent research shows that language models exhibit bias towards their own responses in blind …
- 2nd place by peer reviewView project: Approximating Human Preferences Using a Multi-Judge Learned System
Approximating Human Preferences Using a Multi-Judge Learned System
AutoBox
In this work, we introduced a learned approach to aggregating multi-judge scores: using a GAM and a simple MLP as an alternative to traditional, non-learned methods like averaging. Our models outperform the naive …
- 3rd place by peer reviewView project: Judge using SAE Features
Judge using SAE Features
SAEwhat?
The key idea of this project was to explore model judgement using Sparse Autoencoder (SAE) features for mathematical reasoning tasks involving addition, multiplication, and subtraction operations. We compared this …