Kobayashi Maru: a pre-registered dose–response study of cheating spillover from impossible tasks to the solvable ones beside them

Strong experimental execution. The result that replayed agent notes can carry the behavior forward is interesting because it points to a concrete thing eval designers can monitor. My main question is how consequential the effect is outside this setup. I wld next test this without a planted answer file, on harder tasks where accessing the grader can actually change the submitted result, and try selective note-scrubbing rather than removing the whole memory channel.

Interesting idea, to the best of my knowledge a novel direction and one which it seems important to understand. The conclusions regarding agent memory and reconnaissance habits leading to cheating are potentially impactful. They were nonetheless not really highlighted in the abstract or introduction.

I would have liked to see more consideration of whether merely shuffling the impossible tasks in was sufficient - presumably the agents are still going to do the eval serially and it's much more relevant the fraction of impossible tasks they've seen so far than in the whole trajectory?

Including the exact numbers in the abstract was probably unnecessary and made it harder to read. Additionally it does not present what I think are the most important take-aways from the project.

Much of the text reads as very AI-generated which makes it hard to believe the claims are quite correct.

I think this paper asks an interesting question about whether impossible tasks encourage agents to break rules on nearby tasks they could solve. Measuring behavior on the solvable tasks is a useful contribution, and the reported negative results help show where the effect did not appear.

However, adding impossible tasks also makes the batches longer, so the experiment does not clearly separate those explanations. I would compare equally long batches with and without impossible tasks before making the stronger causal claim. I also found the paper harder to review than the question warranted. Dense statistical terminology, especially in the abstract, repeated claims, and secondary analyses makes it difficult to follow the main experiment or for others to build on the work. A clearer account separating the findings from their interpretation without dense terminology would make the contribution easier to judge.