Approximating Human Preferences Using a Multi-Judge Learned System

This is a great submission given the time allowed. Documentation is excellent.

Spell check the paper. There is a reference to Appendix A that should be to Appendix B. Explain all abbreviations at least once. All diagrams and tables should have a description that makes them ‘stand alone’ - assume some readers read just the tables and diagrams.

Nice work! This is a really solid technical contribution demonstrating that learned aggregation of multi-judge scores significantly outperforms naive averaging. The mathematical formalization and very clear writeup make it a pleasure to read. The dataset is sufficiently large for the finding to be robust and the implementation details seem very solid.

To further strengthen this work consider these ideas:

* I liked the thoughts on bad judge contamination. Could you add a bad judge to demonstrate the effect?

* Generalization could be tested even more rigorously by holding out entire personas or domains

* Do some analysis on the resulting aggregation model. What is actually learned? How exactly does it achieve the performance gain on naive aggregation?

* I'd be curious about training personalized models per simulated user and comparing the resulting models

I love the exploration of using MLP and other architectures to aggregate multiple judges. I also liked the idea of using personas to simulate different judges.

One potential weakness is that your work seems to depend on having high quality labelled data. It would be interesting to see if you can define paradigms that scale well with having noisy “weak” probabilistic labels, e.g. by using some sort of discriminator like defined in this paper here from Snorkel (https://link.springer.com/article/10.1007/s00778-019-00552-1?utm_source=chat

Clever approach and I like the idea of doing mechanistic interpretability analysis on the smaller learned model. Would be interested to see more development of that aspect too!