Jun 2, 2025
Approximating Human Preferences Using a Multi-Judge Learned System
Eitan Sprejer, Fernando Avalos, José Pedro Brito de Azevedo Faustino, Augusto Mariano Bernardi
In this work, we introduced a learned approach to aggregating multi-judge scores: using a GAM and a simple MLP as an alternative to traditional, non-learned methods like averaging. Our models outperform the naive baseline in predicting simulated human preferences, demonstrating that learned aggregation can better capture complex evaluative signals.
Jason Hoelscher-Obermaier
Nice work! This is a really solid technical contribution demonstrating that learned aggregation of multi-judge scores significantly outperforms naive averaging. The mathematical formalization and very clear writeup make it a pleasure to read. The dataset is sufficiently large for the finding to be robust and the implementation details seem very solid.
To further strengthen this work consider these ideas:
* I liked the thoughts on bad judge contamination. Could you add a bad judge to demonstrate the effect?
* Generalization could be tested even more rigorously by holding out entire personas or domains
* Do some analysis on the resulting aggregation model. What is actually learned? How exactly does it achieve the performance gain on naive aggregation?
* I'd be curious about training personalized models per simulated user and comparing the resulting models
Philip Quirke
This is a great submission given the time allowed. Documentation is excellent.
Spell check the paper. There is a reference to Appendix A that should be to Appendix B. Explain all abbreviations at least once. All diagrams and tables should have a description that makes them ‘stand alone’ - assume some readers read just the tables and diagrams.
Curt Tigges
Clever approach and I like the idea of doing mechanistic interpretability analysis on the smaller learned model. Would be interested to see more development of that aspect too!
Amir Abdullah
I love the exploration of using MLP and other architectures to aggregate multiple judges. I also liked the idea of using personas to simulate different judges.
One potential weakness is that your work seems to depend on having high quality labelled data. It would be interesting to see if you can define paradigms that scale well with having noisy “weak” probabilistic labels, e.g. by using some sort of discriminator like defined in this paper here from Snorkel (https://link.springer.com/article/10.1007/s00778-019-00552-1?utm_source=chat
Cite this work
@misc {
title={
@misc {
},
author={
Eitan Sprejer, Fernando Avalos, José Pedro Brito de Azevedo Faustino, Augusto Mariano Bernardi
},
date={
6/2/25
},
organization={Apart Research},
note={Research submission to the research sprint hosted by Apart.},
howpublished={https://apartresearch.com}
}
