Detecting Deception with AI Tics 😉

This work tests whether models follow a system prompt that tells them to mark any lie with a winky face. The robustness of the method is not discussed (though fine-tuning it in is suggested), nor is it compared to existing deception detection methods (e.g. contrast-consistent search and variants, activation vectors, blackbox lie detection). It is suggested that future models would find it hard to not include such tells if fine-tuned, but this statement is not supported. In particular, deception is only a problem if fine-tuning models for helpfulness & harmlessness does not work, so in any deception scenario, simple at least some simple fine-tuning strategies have already broken. Future work could explore whether fine-tuning such tells are more robust than other types of fine-tuning aimed at preventing or detecting deception.