Curvature-aware sycophancy reduction

I like the fundamental attempt to isolate behaviors at a weights level. Ambitious project for a hackathon with theory, finetuning, and evaluation all in one.

However, left with multiple questions:

1. KFAC assumptions: The link between KFAC specifically and the assumptions about sycophancy isn't clear. It's not obvious to me sycophancy even specifically exists as a distinct measurable concept as the hypothesis suggests. This seems to be a load-bearing assumption - that sycophancy exists in a certain way in the loss landscape - which I'm not sure this data proves/falsifies/isolates specifically, versus other factors like affirm/deny, general entropy, etc. This is the case with any project involving dense LLMs, but here you are specifically proposing an observable architectural mechanism.

2. Model size: Yes, the model is small which limits conclusions and adds noise, but fundamentally it would've been hard to demand a large theoretical finetuning run over a weekend.

Would've been a 5 if the results were tighter with causal ablations.

This project investigates whether sycophancy is represented in the loss landscape by using curvature analysis and weight-projection editing. Examining sycophancy in this way is a novel approach and the experimental set-up is clean and clearly defined. The authors provide sufficient detail to understand the process and ground the research in previous works. They acknowledge that the work is exploratory, but the claims made in the conclusion may be a little too optimistic about the ability to apply this to more realistic datasets.

The write-up and presentation of results are clear. I feel that the project would benefit from a closer examination of the claim that "reducing sycophancy should increase truthfulness” as it is based on a narrow definition of sycophancy that presupposes a “correct answer”.