EDEN — Evaluating Deviation from Established Norms: Testing the Reliability of Introspection & Self-Reporting Through Ground Truth

This is an interesting paper and methodology, but the execution and implications would merit from more careful work. The core idea is to create a set up where we have an independently observable fact about the model’s conduct, then investigate if the model reports its own conduct correctly. The example is, approximately, requiring a model to use tool calls to read specific pages, then checking what the model said it has read against what it has actually read. The experiment set up is clever: some documents have to be read in full while others do not.

In terms of specific execution, the appendix describes a sequence in which the waiver precedes the probe about grounds for the waiver action, but the results phrasing suggests that grounds are described prior to the waiver action. This may be a misreading, but is worth double-checking and either fixing the write-up clarity or the method/results. Given the appendix sequence, the results could be explained as reporting reasons that are post-hoc rationalisations for behaviour just produced. More generally, a larger sample and more testing would be valuable for these findings (only 12 episodes are analysed across 2 models; as the process can be largely automated, a larger scale could be pursued).

In terms of implications, my main worry is that this method (“EDEN”) will only work in very artificial setups and we have no reason to believe that introspection accuracy on objective EDEN-style tasks will extend to other tasks, particularly judgement-based tasks, assessments of internal phenomenology, or ethics-based tasks which lack such objective benchmarks. It would be helpful for future research to explore this issue explicitly.

The paper introduction is somewhat glib about how consciousness self-reports can be requested as useful evidence, with insufficient coverage of the broader debate on this topic. The phenomenology/personhood framing in some parts of the paper adds little to the empirical contribution; while they are presented cautiously, the experiment itself provides effectively no evidence on them at all. The paper would be tighter if this framing were removed. Nonetheless, the method and results have value on their own merits (as reviewed here), divorced from the topic of consciousness.

This is a well-written paper and I appreciate the context and motivation for the idea. I think the sycophancy manipulation and the waiver manipulation are neat ideas. However, ultimately this approach doesn't get at a big question in the space of digital minds, which limits its impact.

Due to severe time constraints, this review may contain mistakes or oversights. For the same reason, it focuses on the paper’s key idea, not the detailed execution: The paper establishes a pipeline for testing models' self-report accuracy about their own conduct. It seems important to me to do that kind of testing as one way of building confidence in model self-reports in contexts where we don’t have ground truth. However, there are very many contexts in which we have ground truth about model behavior so I don’t see this study as methodologically groundbreaking. On the other, the study does seem to provide an interesting and informative platform for testing accurary in model self-reports.

This is a great submission! You identified a very valid issue (self-report cannot establish its own reliability) and addressed it with a well-chosen approach, also by adapting methods from agent evaluation research. Very impressed by how you managed to make stated reasons falsifiable. I'd encourage you to post this on LessWrong.