TriloByte: Evaluating LLMs on Bolivian Quechua Through a Ground-Truth-Based Framework for Low-Resource Languages

This project tackles a real and underexplored gap: there is very little systematic evaluation of how LLMs handle Bolivian Quechua, and grounding the evaluation in a curated bilingual Quechua-Spanish dictionary is a sensible, reproducible anchor. Pairing embedding-based semantic similarity with human judgment on adequacy, completeness, and fluency is the right instinct, and the team is commendably honest about the limits of automated metrics. The work is a genuine contribution to linguistic inclusion.

The execution and reporting are where it needs the most work. (1) The headline metric is undefined: "Relative Agreement (%)" of ~32 / ~25 / ~10 leaves the reader unable to tell what is actually being measured - a cosine-similarity threshold, a human adequacy rate, agreement against the dictionary? Define it precisely and report absolute per-criterion scores, not just a relative ranking. (2) Sample size is never stated. The paper says "a limited set of lexical entries" and reports a Spearman of 0.48 over an unstated n - every number is currently unfalsifiable. Report N entries, N human ratings, and N per model. (3) Only three commercial models are tested, all small/fast tiers (Gemini Flash, Flash Lite, Claude Haiku). Add at least one frontier model and one open-source multilingual model - the genuinely interesting safety question is whether scale or multilingual pretraining closes the gap. (4) There is no inter-annotator agreement on the human evaluation, which is the ground truth for the 0.48 correlation claim. (5) The AI-safety framing is thin; tie a lexical error to one concrete downstream harm (the health/legal/education scenario you mention) so it reads as a safety eval and not only an NLP demo.

Concrete fix: define the metric and report absolute per-criterion scores with n, add a frontier and an open-source model, and report inter-annotator agreement. That turns a promising prototype into a citable low-resource benchmark.

The paper's central claim of human evaluation is not supported by the code. The repository shows the assessment was done by an LLM judge (Claude Opus 4.8), so the reported Spearman ≈0.48 measures agreement between two automated metrics, not between automation and humans. Results are also selectively reported and statistically weak: the strongest model (Opus, 79%) is dropped because it ran with the reference visible while the other models are scored on very different sample sizes and ranked as comparable without confidence intervals or significance tests.

Improvement opportunities. Concretely: (1) align paper and code, either run a small real human evaluation or rename it honestly as "LLM-as-judge" (2) use the same blind protocol, the same fixed sample size, and report confidence intervals plus a significance test across models, and either include Opus under blind conditions or drop the leaked run entirely.