Skip to content
Apart Research
Apart Research
Contact us
Sprint projectMar 20, 2026Bangkok

Geographic Blind Spots in AI Control Monitors: A Cross-National Audit of Claude Opus 4.6

Jason Hung · Team Global AI Dataset (GAID) Project

Submitted to AI Control Hackathon 2026. Sprint projects are early-stage work by participants, not Apart Research publications.

Read the report

Report: Geographic Blind Spots in AI Control Monitors: A Cross-National Audit of Claude Opus 4.6

Open in a new tab (opens in new tab)
Code (opens in new tab)
Share

Artificial intelligence (AI) control protocols assume that trusted large language model (LLM) monitors reliably assess proposed actions across all deployment contexts. This project tests that assumption in the geographic dimension. We audit Claude Opus 4.6—the monitor specified in Apart Research’s AI Control Hackathon Track 3 benchmark—for systematic gaps in its factual knowledge of the global AI landscape. We develop the AI Control Knowledge Framework (ACKF), a six-dimension thematic scheme, and operationalise it with 17 verified indicators drawn from the Global AI Dataset v2 (GAID v2): 24,453 indicators across 227 countries published on Harvard Dataverse. A five-category response classification scheme distinguishes verifiable fabrication (VF) from honest refusal (HR); logistic regression with country-clustered standard errors combined with difference-in-differences (DiD) estimation quantifies geographic disparities in monitor accuracy across 2,820 country-metric-year observations. Contrary to our initial hypothesis, Claude Opus 4.6 produces higher fabrication rates for Global North queries than for Global South counterparts—a pattern consistent with a partial-knowledge mechanism in which the model attempts answers more frequently for Global North contexts but commits to incorrect values. This fabrication profile constitutes an exploitable vulnerability, where an adversarial AI system could frame harmful actions in governance or public attitude terms to reduce the probability of detection. This project provides the first cross-national, multi-domain audit of an AI control monitor’s geographic knowledge gaps, with direct implications for the design of control protocols.

Reviews

Judging this Sprint?

Review this project

Your public critique appears on this page without your name. Your private critique is not published; only the Apart team reads it. If you agree below, we share your review with grantmaking.ai (opens in new tab) and the Transformative AI Fund so strong projects can be funded.

Not shown on this page.

Shown on this page, without your name.

Only the Apart team reads this, and funders if you agree below.

Share my name publicly on grantmaking.ai *
Share my private critique with funders *

How much would this matter for AI safety if it worked? How innovative is it? For scores of 4-5: is this actually new to the field, or replicating recent work?

Scoring guide
  1. 1Negligible. No clear problem addressed, or no meaningful novelty.
  2. 2Limited. Addresses a real problem but with a generic or well-trodden approach. Incremental at best.
  3. 3Moderate. Clear problem with a reasonable approach; some novelty in framing or method beyond routine application of existing tools.
  4. 4Significant. Important problem with an original approach, or identifies a neglected problem area. A valuable contribution others could build on.
  5. 5Exceptional. Tackles a critical AI safety problem with a genuinely novel approach, or opens a new research direction. Clear theory of change. You'd be excited to share this with researchers in the area.

How sound are methodology, implementation, and findings?

Scoring guide
  1. 1Seriously flawed. Methodology broken, results uninterpretable, or implementation doesn't work.
  2. 2Weak. Approach has significant gaps: missing validation, flawed experimental design, or incomplete implementation.
  3. 3Competent. Technically solid given the short duration. Methodology makes sense, results are interpretable, limitations acknowledged, work builds toward clear conclusions.
  4. 4Strong. Thorough methodology with convincing validation. Results clearly support conclusions. Immediately useful for future work.
  5. 5Exceptional. Ambitious scope executed rigorously. Surprising findings, novel methods, or unusually robust validation.

How clearly are work, findings, and impact potential communicated?

Scoring guide
  1. 1Incomprehensible. Cannot determine what the project is actually claiming or doing.
  2. 2Hard to follow. Key information buried, missing, or diluted by excessive length. Significant effort to extract main points.
  3. 3Clear enough. Can understand the problem, approach, and results without undue effort. Core content clearly present: problem, method, findings, limitations.
  4. 4Well presented. Easy to follow, well-structured, appropriate level of detail. Target audience would get it quickly.
  5. 5Exceptionally clear. A pleasure to read. Complex ideas made accessible. Could serve as a model for how to present this type of work.

  1. The statistical methodology is thorough, e.g. the five-category classification scheme, querying variants, and the sensitivity analysis showing close-range fabrication are all well-executed. The finding that Global North queries produce higher fabrication rates than Global South ones is an interesting result.

    The work would benefit from clarifying the threat model, i.e. an adversarial agent exploiting the monitor's inability to recall specific statistics about AI governance or public attitudes, which might not fully match how monitors evaluate proposed actions or examine artifacts for safety. The adversarial framing study that would bridge this gap, as listed in future work, would strengthen the contribution.

  2. This paper asks whether an AI monitor used in control protocols (specifically Claude Opus 4.6) has systematic gaps in its factual knowledge depending on geography and domain, and whether those gaps could be exploited by an adversarial agent. They tested this against a large set of structured quantitative queries with verified answers in areas including compute capacity, governance and public attitudes.

    The key findings, that Claude refused to give an answer in 91.4% of cases, and - when it did answer - was wrong 62.5% is analyzed in a detailed manner by category and for significance.

    The connection to AI control is that knowledge of a monitors weak spots could be leveraged by an attacker -especially if the monitor confidently believes a false fact. Mitigations to such vulnerabilities are also discussed.

    While the findings are interesting and relevant, it would be desirable if the connection to control had been strengthened. For example, sketching out a concrete scenario in which a monitor may need to rely on numerical facts (that it presumably cannot look up) to perform supervision would have been helpful, which is imagined but only hinted at in the future work section. It would also be an interesting questions whether similar knowledge gaps exist for the more commonly considered type of monitors in coding settings, and whether they could be exploited there.

    Read full reviewShow less

Cite this project

@misc{hung2026geographic,
  title = {{Geographic Blind Spots in AI Control Monitors: A Cross-National Audit of Claude Opus 4.6}},
  author = {Jason Hung},
  year = {2026},
  month = mar,
  note = {Submitted to AI Control Hackathon 2026, an Apart Research Sprint},
  howpublished = {\url{https://apartresearch.com/sprints/projects/geographic-blind-spots-in-ai-control-monitors-a-crossnational-audit-of-claude-opus-46-kmr2}},
  url = {https://apartresearch.com/sprints/projects/geographic-blind-spots-in-ai-control-monitors-a-crossnational-audit-of-claude-opus-46-kmr2}
}

More from AI Control Hackathon 2026

All 125 entriesBrowse all projects

Build something like this at the next Sprint

AI Collusion Research Sprint · Oct 23 - 25, 2026