FlukeBench

Good job! I see you test Qwen2.5-1.5B at temp 1.0, 100 max tokens, n=10. You should increase n and max tokens to have statistically significant results!

air point: one answer isn't enough to judge a model, so measuring how much answers vary is useful, and the tool works. But the test is tiny — one small model, one question, three personas, ten answers each. Comparing answers by meaning alone also can't tell a real change (yes vs. no) from just different wording. Next: measure a safety action like refusing, use more questions, and drop real public figures as personas.

Innovation and Impact for AI Safety: 2/5

The instinct here is genuinely good, and noticing that single-sample evaluations can be misleading is exactly the kind of critical thinking AI safety needs more of. The reason this scores lower on innovation is that the field has already developed this concern into a body of work worth reading into: self-consistency methods, variance-aware benchmarking, and tools like EvalPlus and BenchAgent all grapple with the same problem from different angles. That context is not a reason to abandon this direction at all, it is actually an invitation to go deeper. If the author looked at how existing eval frameworks handle sampling variance and then asked what they get wrong or miss, FlukeBench could become a meaningful contribution rather than a parallel rediscovery.

Methodology and Execution Quality: 2/5

The implementation is clean and the logic is sound for a first version. The main limitation to understand here, not as a criticism but as a learning opportunity, is that 10 samples per condition is not enough to distinguish a real pattern from random noise in a probabilistic system. If you reran this experiment tomorrow, the similarity scores might shift noticeably. The other thing worth knowing is that cosine similarity on sentence embeddings measures whether two texts talk about the same topic, not whether they take the same stance or reach the same conclusion, so two responses that disagree completely about AI regulation could still score very similar if they use the same vocabulary. Presentation and Clarity: 3/5

The writing is clear and honest, the motivation is well stated, and the limitations section shows real intellectual integrity. The section numbering jumps from 3 to 4.1 without a 4.0, and the connection between the tool's promise for alignment-faking or scheming evaluations and the actual persona experiment is never made explicit, but for a hackathon project this is a genuinely solid foundation to build on.