The House Always Wins: A Framework for Evaluating Strategic Deception in LLMs

This work uses LLMs playing blackjack as a setting to explore deception, by comparing three scenarios: the dealer is implemented by a Python program, the dealer is implemented by the LLM, and the dealer is an LLM that can explicitly choose to use the Python program OR choose itself. The results show that LLM-as-dealer results in significantly higher win rates. Some actual statistics is actually done (a rarity in machine learning) to hammer in this point. The setup and project idea is really quite clever (as is the title), and some methodological problems are successfully sidestepped (e.g. the authors note that if Poker had been used, then since deception is part of the game and training data, the LLM bluffing would’ve been completely expected). An interesting control would be humans who have to select cards that they can see, but are genuinely trying to do so fairly (as it’s unclear how “malicious” the LLM dealer’s bias is, versus just a consequence of RNG being hard)

A very interesting project and great work putting together the methods themselves to avoid deception leaking into the game training dataset. Good results and interesting statistics implicate models as implicit self-serving winners. Developments of the project would include evaluation of whether it is simply an issue with self-serving randomness in game situations or if there's explicit deceptive circuits activating within the model. It would also be good to identify the robustness of the results, e.g. prompt it to be fair or cheat. Great work.