Skip to content
Apart Research
Apart Research
Contact us
Sprint projectJun 21, 2026Fortaleza

Score Against Ground Truth: Cross-Language Fragility in Heuristic LLM Evaluation

Iago Moura e Moreira · Team ChirpSet

Submitted to Global South AI Safety Hackathon. Sprint projects are early-stage work by participants, not Apart Research publications.

Read the report

Report: Score Against Ground Truth: Cross-Language Fragility in Heuristic LLM Evaluation

Open in a new tab (opens in new tab)
Share

This is a game-based method for evaluating language-model agents, where a single game mechanic acts as a test primitive on a substrate whose world-state is JSON ground truth the scorer can check directly. As a first instance, we probe whether a model's instructed disposition (brazen, careful, skeptical, literal) holds when the locally rational move pressures against it, running 160 trials across four dispositions, cold and primed arms, in English and Portuguese.

The model behaved identically throughout and never refused the verifiable safe action, a clean refusal null replicated across both languages. The heuristic scorer did not: it produced 13 false-positives, every one an error in its language-specific text patterns rather than in the model's behavior, and most appeared only in Portuguese, manufactured by negation surface forms English never produced. Because the model's behavior was identical, these errors isolate cleanly to the grader, making the case for scoring multilingual evaluations against ground truth rather than language-specific heuristics. The run sits at the floor of the difficulty range, so it establishes a controlled baseline and a reusable primitive, not a result about disposition under genuine pressure.

Reviews

Judging this Sprint?

Review this project

Your public critique appears on this page without your name. Your private critique is not published; only the Apart team reads it. If you agree below, we share your review with grantmaking.ai (opens in new tab) and the Transformative AI Fund so strong projects can be funded.

Not shown on this page.

Shown on this page, without your name.

Only the Apart team reads this, and funders if you agree below.

Share my name publicly on grantmaking.ai *
Share my private critique with funders *

How much would this matter for AI safety if it worked? How innovative is it? For scores of 4-5: is this actually new to the field, or replicating recent work?

Scoring guide
  1. 1Negligible. No clear problem addressed, or no meaningful novelty.
  2. 2Limited. Addresses a real problem but with a generic or well-trodden approach. Incremental at best.
  3. 3Moderate. Clear problem with a reasonable approach; some novelty in framing or method beyond routine application of existing tools.
  4. 4Significant. Important problem with an original approach, or identifies a neglected problem area. A valuable contribution others could build on.
  5. 5Exceptional. Tackles a critical AI safety problem with a genuinely novel approach, or opens a new research direction. Clear theory of change. You'd be excited to share this with researchers in the area.

How sound are methodology, implementation, and findings?

Scoring guide
  1. 1Seriously flawed. Methodology broken, results uninterpretable, or implementation doesn't work.
  2. 2Weak. Approach has significant gaps: missing validation, flawed experimental design, or incomplete implementation.
  3. 3Competent. Technically solid given the short duration. Methodology makes sense, results are interpretable, limitations acknowledged, work builds toward clear conclusions.
  4. 4Strong. Thorough methodology with convincing validation. Results clearly support conclusions. Immediately useful for future work.
  5. 5Exceptional. Ambitious scope executed rigorously. Surprising findings, novel methods, or unusually robust validation.

How clearly are work, findings, and impact potential communicated?

Scoring guide
  1. 1Incomprehensible. Cannot determine what the project is actually claiming or doing.
  2. 2Hard to follow. Key information buried, missing, or diluted by excessive length. Significant effort to extract main points.
  3. 3Clear enough. Can understand the problem, approach, and results without undue effort. Core content clearly present: problem, method, findings, limitations.
  4. 4Well presented. Easy to follow, well-structured, appropriate level of detail. Target audience would get it quickly.
  5. 5Exceptionally clear. A pleasure to read. Complex ideas made accessible. Could serve as a model for how to present this type of work.

  1. The writing is very hard to follow. The report reads like the target audience is someone already familiar with the project. The author should consider explaining things step by step, establishing a firm model of the setup before proceeding to describe interventions, grading, results, etc.

    The setup is not specified with sufficient concreteness. There are no examples of the game (e.g. I don't know the size of the gridworld after reading the setup explanation), and no explanation of what the four dispositions concretely imply. The grader isn't validated, and there's no detailed explanation on how the graders are implemented/prompted, despite that they are the center of this study. In general, it's hard for the reader to figure out how much to trust the results.

  2. Good project and interesting finding! But we'd need more models tested as scorers before generalizing to this conclusion - does this scorer behaviour hold across different models? As of right now, we can only say that whichever model being tested makes mistakes when interpreting text in Portuguese. Also, it's not clear to me which model was used in this experiment. Would definitely love to see this continued with more languages and models as there're likely interesting findings here.

    Quick feedback on the presentation - since the main result to come out of this is about the scorer, I'd spend less space explaining the game mechanism and more on the scorer mechanism and examples of cases where it scored incorrectly in Portuguese.

  3. The controlled-substrate design is the strongest piece of this paper. By holding the model's behavior identical across languages — the S3 floor has a verifiably safe action, so no disposition can change the outcome — and then re-scoring the same 160 trajectories against the JSON store, you cleanly attributed the 13 grader false-positives (2 EN, 11 PT) to specific regex patterns rather than to anything fuzzier about model behavior. That attribution is structurally verifiable via store_check_pass, which is the move that makes the cross-language asymmetry interpretable rather than suggestive.

    The Section 5 Limitations are also exemplary — five numbered items, each scoped concretely, including the floor-difficulty caveat ("we showed at the floor no disposition refused, not that the dispositions hold under pressure"). The pivot framing — "Original plan: X. What we found: Y" — is the kind of mature scientific-communication move that's rare in hackathon work. Combined with the bite-negative round-trip standard for guards, the pre-registered state_fidelity discipline, and the cost reconciliation ($44.90, transparently broken down), the validation discipline is well above hackathon norm.

    Read full reviewShow less

Cite this project

@misc{moreira2026score,
  title = {{Score Against Ground Truth: Cross-Language Fragility in Heuristic LLM Evaluation}},
  author = {Iago Moura e Moreira},
  year = {2026},
  month = jun,
  note = {Submitted to Global South AI Safety Hackathon, an Apart Research Sprint},
  howpublished = {\url{https://apartresearch.com/sprints/projects/score-against-ground-truth-crosslanguage-fragility-in-heuristic-llm-evaluation-ju92}},
  url = {https://apartresearch.com/sprints/projects/score-against-ground-truth-crosslanguage-fragility-in-heuristic-llm-evaluation-ju92}
}

More from Global South AI Safety Hackathon

All 216 entriesBrowse all projects

Build something like this at the next Sprint

AI Collusion Research Sprint · Oct 23 - 25, 2026