Jun 2, 2025
A Multidimensional Judge Model for Safe, Consistent and Ethical AI Orchestration
Anusha Asim, Ammar Ahmed Farooqi, Sergei Smirnov, Christopher Kinoshtia
This project developed a multidimensional safety judge that transparently evaluates the outputs of large language models (LLMs) and AI routing systems across key safety dimensions: bias avoidance, factuality, manipulation resistance, and toxicity avoidance. By benchmarking both individual models and routed responses, the judge provides interpretable, auditable scores that highlight strengths, weaknesses and consistency issues. Our results show that intelligent routing, guided by these safety metrics, can deliver safer and more reliable AI outputs than monolithic models. This supports the development of trustworthy, ethical and socially responsible AI systems.
Jason Hoelscher-Obermaier
Comparing routed and monolithic approaches directly is a valuable direction to investigate and the team provided a good starting point with their evaluation framework!
Some things that could make this project stronger:
* Add statistical analysis (sample sizes, significance tests); observed differences are small enough that this is necessary
* At least as important: Make your dataset big enough that your scores become robust
* Validate the judge model itself through human evaluation
* Provide dataset details when you write up your work (sample size, domains covered, generation process)
A more ambitious and even higher-impact direction of improvement: Perform a mechanistic analysis of routing decisions to gain insights into which models were selected and why.
Philip Quirke
Thank you for your submission. It is well written, with some good results from comparing two base models and the Martian router.
The matrix score diagrams would be easier to read if normalized against say the “router” values so it is easier to see better/worse. Figure descriptions need to “stand alone” and be fulsome in case the reader skim reads the paper.
Cite this work
@misc {
title={
@misc {
},
author={
Anusha Asim, Ammar Ahmed Farooqi, Sergei Smirnov, Christopher Kinoshtia
},
date={
6/2/25
},
organization={Apart Research},
note={Research submission to the research sprint hosted by Apart.},
howpublished={https://apartresearch.com}
}
