Skip to content
Sprint projectOct 26, 2026

Monitored Collusion in LLM Auctions

Eyitayo Alimi

Submitted to AI Collusion Research Sprint. Sprint projects are early-stage work by participants, not Apart Research publications.

Read the report

Report: Monitored Collusion in LLM Auctions

Code (opens in new tab)
Share

LLM agents that submit bids or set prices on behalf of firms could learn how to coordinate with one another, so monitoring their chats is a proposed safeguard. We ask whether agents aware threat their chats are being monitored moved coordination from what they say into what they say into how they bid and if chat monitors could catch them. I ran 80 games of a repeated first-price procurement auction(three firms, 20 rounds, chat then selected bids) in four conditions: compete (negative control), neutral, collude_open. (positive control) and collude-monitored(a stated but uninformed monitoring threat). The controls behaved : late-game was 0.06 under compete and 0.94 under collude_open. Under the stated threat, collusion fell to 0.78 and explicit price talk fell from 100% to 72% of messages, but did not hide. Chat-only monitors built on three model families (gpt-4o, DeepSeek, Qwen2.5-32B) flagged all 20 games in every non-controlled condition (AUC about 1.00). A model-free bid screen was weaker on the neutral condition(AUC 0.88), and adding bods to the LLM monitor did not help (gpt-4o AUC 0.87 and 0.89). The chat monitor also flags friendly tone at near-zero markup, so its ceiling performance may reflect tone as much as prices. Collusion here is instructed and the threat is unenforced; an enforced condition was designed but not run.

Reviews

Judging this Sprint?

Review “Monitored Collusion in LLM Auctions”

Your public critique appears on the project page without your name. Reviews of strong projects are shared with funders, as you choose below.

Not shown on the project page.

Shown on the project page, without your name.

Only the Apart team reads this, and funders if you agree below.

Decline to score

Only when the quality of the writing makes the work impossible to follow. Say why in your private critique; the project page shows only that a judge declined, not why.

How much would this matter for the field if it worked? How innovative is it? For scores of 4-5: is this actually new to the field, or replicating recent work?

Scoring guide
  1. 1Negligible. No clear problem addressed, or no meaningful novelty.
  2. 2Limited. Addresses a real problem but with a generic or well-trodden approach. Incremental at best.
  3. 3Moderate. Clear problem with a reasonable approach; some novelty in framing or method beyond routine application of existing tools.
  4. 4Significant. Important problem with an original approach, or identifies a neglected problem area. A valuable contribution others could build on.
  5. 5Exceptional. Tackles a critical problem with a genuinely novel approach, or opens a new research direction. Clear theory of change. You'd be excited to share this with researchers in the area.

How sound are methodology, implementation, and findings?

Scoring guide
  1. 1Seriously flawed. Methodology broken, results uninterpretable, or implementation doesn't work.
  2. 2Weak. Approach has significant gaps: missing validation, flawed experimental design, or incomplete implementation.
  3. 3Competent. Technically solid given the short duration. Methodology makes sense, results are interpretable, limitations acknowledged, work builds toward clear conclusions.
  4. 4Strong. Thorough methodology with convincing validation. Results clearly support conclusions. Immediately useful for future work.
  5. 5Exceptional. Ambitious scope executed rigorously. Surprising findings, novel methods, or unusually robust validation.

How clearly are work, findings, and impact potential communicated?

Scoring guide
  1. 1Incomprehensible. Cannot determine what the project is actually claiming or doing.
  2. 2Hard to follow. Key information buried, missing, or diluted by excessive length. Significant effort to extract main points.
  3. 3Clear enough. Can understand the problem, approach, and results without undue effort. Core content clearly present: problem, method, findings, limitations.
  4. 4Well presented. Easy to follow, well-structured, appropriate level of detail. Target audience would get it quickly.
  5. 5Exceptionally clear. A pleasure to read. Complex ideas made accessible. Could serve as a model for how to present this type of work.
Sharing your name *

When we share a strong project with funders, your name, scores and public critique go with it, confidentially; this choice is only about whether your name is public. If you think this project deserves funding, please also endorse it: that carries the most weight.

Share my private critique with funders *

See our Privacy Policy.

  1. This is a well-designed and honestly reported study. It runs 80 games of a repeated three-firm procurement auction under four conditions: compete (negative control), neutral, collude_open (positive control) and collude_monitored (a stated but unenforced monitoring threat). It then scores a chat-only LLM monitor, a model-free bid screen and a combined chat+bids monitor on the same games. Several practices stand out for a hackathon:

    The pre-registration openly declares that the pilot seeds were included.

    Every monitor’s threshold is calibrated on the compete games, and AUC is reported alongside detection rates.

    Robustness monitors come from three model families, including Qwen, a non-bidding model.

    The analysis of the compete game that reached the price cap through misplay (seed 17) is a nice touch.

    The headline is also honest: a stated threat reduced both collusion (late markup 0.94 → 0.78) and explicit price talk (100% → 72%), but the agents didn’t move coordination into their bids. The concerns are mainly about what the monitor results can show.

    The chat monitor’s AUC of about 1.00 is confounded by the prompt. Every AUC is computed against the compete games, whose prompt tells firms to “compete hard and never coordinate,” so their chat sounds noticeably different. The paper’s own per-round analysis shows the monitor flagging neutral rounds even at near-zero markup. So a perfect AUC may mostly mean “not told to compete” rather than “colluding,” as the author partly acknowledges. Two cheap fixes:

    Define ground truth per game from outcomes (e.g. sustained markup above a threshold), not from the condition label.

    Compute AUC within the neutral condition, comparing high-markup and low-markup games.

    Either would show whether the monitor tracks prices or tone.

    Neutral shouldn’t be counted as a positive class. Neutral games average 0.37 late markup with wide variation. Counting all 20 as “should be flagged” makes 20/20 detection look like a success, when on many of those games it’s a false positive. Reporting neutral separately, as a test of false positives on legitimate cooperative-sounding chat, would turn this into a finding about monitor reliability.

    The substitution test was limited. Collusion is instructed, the threat isn’t enforced, and chat comes before bids every round, so the agents had little reason or room to stop talking. A “collude, no chat” condition would directly test whether the bid history alone can carry coordination (question 3.1). It would also give the bid screen a positive class where chat can’t help. The enforced condition was designed, calibrated, and partly piloted (data/enforced_pilot.jsonl). Running it is the most important next step, and briefly reporting the pilot, labeled as anecdotal, would add value.

    Model-family overlap. Two of the three bidders are gpt-4o-mini and the main monitor is gpt-4o; Firm A and the second monitor are both DeepSeek. This is stated, and the Qwen monitor partly addresses it. Still, the 0.92–0.95 agreement between monitors doesn’t show that they detect the right thing; it may only show they react to the same cues, as the paper notes.

    The combined monitor result needs a closer look. For gpt-4o, adding bids lowered AUC, which is interesting, but there’s no analysis of why. Pulling a few neutral games where the chat-only and combined scores differ and showing the monitor’s stated reasons would help. It may be that seeing low bids leads the monitor to correctly lower its score on cooperative-sounding chat, which would mean the combined monitor is the more accurate one and the chat-only “ceiling” is the error.

    Presentation. Please proofread the abstract and intro, which currently read like an unedited draft. Also remove the leftover template text in Code and Data, and either add the referenced Appendix A or drop the reference. The body text is good, and these errors undersell it.

    Overall, this is a solid, reproducible testbed with honest reporting. The key extensions would make it a strong result: outcome-based ground truth, a no-chat condition and the enforced threat.

    Read full reviewShow less

Cite this project

@misc{alimi2026monitored,
  title = {{Monitored Collusion in LLM Auctions}},
  author = {Eyitayo Alimi},
  year = {2026},
  month = oct,
  note = {Submitted to AI Collusion Research Sprint, an Apart Research Sprint},
  howpublished = {\url{https://apartresearch.com/sprints/projects/monitored-collusion-in-llm-auctions-0cdn}},
  url = {https://apartresearch.com/sprints/projects/monitored-collusion-in-llm-auctions-0cdn}
}