Confía-CO: A reliability evaluation for an AI customer-service assistant operating in Colombian Spanish

The paper presents an honest and useful case study showing how an evaluation pipeline can produce misleading results. However, the main takeaway (that automated evaluations require human verification as much as the evaluated) does not seem too original. The project would benefit from a more distinctive methodological contribution or solution, or a stronger argument for why this specific failure mode represents a broader or previously overlooked risk.

- The paper is very clear, the process is easy to follow, and it's well written.

- The paper mixes two different things together, and that hurts the clarity of both. The paper centers on the error that occurred while building the (AI-generated) evaluation code. That could have been a comment or a secondary note rather than the central contribution of the article. The paper should instead center on (1) the creation of a fictional SME dataset that can be reused for evaluating other systems, and (2) the creation of an evaluation pipeline. The error itself isn't really evidence that "automated pipelines can produce false metrics" in general, it's evidence that one automated pipeline had a bug. The paper conflates a specific implementation bug with a general claim about automated evaluation, and mixing the dataset/pipeline contribution with the bug narrative makes neither one as clear as it could be.I f the paper intended to focus on the broader problem of trusting LLM-generated code, that would call for a different approach: one that treats this experience as a single triggering incident, but that still requires further exploration to support a general claim.

- There's no clear explanation of how the public and confidential source documents (the ones the model has access to) were generated. This matters and should be documented.

- It would be good to use a different model as the judge/evaluator, rather than the same model that is being evaluated.

The paper sets out the problem well; the experimental design, which uses an SME simulation to evaluate an agency-based evaluator, is ingenious, and the work is, in fact, very thorough. As an observation, it might be worth noting that the model being evaluated and the evaluator should be distinct; in this case, however, the author uses GPT-4-O-Mini for both. It would certainly also be advisable to use local and open-weight models, such as those mentioned by the author himself. Finally, the text could be greatly improved by clearer and more simplified wording, as well as by explaining the acronyms. The current text is a bit of a hard read and could be made more fluent.

This is an honest case study that tackles a very concrete, underexplored deployment setting and clearly articulates its main contribution: the evaluator, not the model, was the primary failure point. The narrative around the scoring bug and its discovery is particularly strong, and the paper does an excellent job of separating what is genuinely measured from what turned out to be artifactual. At the same time, the empirical scope is extremely narrow (one model, one synthetic SME, 20 cases, a single annotator, and the same base model as both assistant and judge), which limits how far the conclusions can travel beyond a cautionary existence proof. To strengthen the work, I would encourage the author to (i) add at least one independent LLM or human judge to check for self‑evaluation bias, (ii) expand the case set and model roster to include a local open‑weights model, and (iii) report a small number of fully worked examples where the calibrated‑abstention mitigation either succeeds or fails once implemented.

This is a sharp piece of work, I enjoyed reading it. The synthetic HORECA SME was a smart call: giving yourself exact ground truth is what let you catch the bug at all, and plenty of people would have published the 21% and moved on. The es-CO Global South focus is a real, under-served problem, not a manufactured one. The part worth most credit is the discipline: you keep measured and unmeasured results strictly apart, and your closing note on rigor, where you spell out each conservative choice you made instead of reaching for the more impressive number, was the right call.

The place to push hardest is validation depth. The meta-finding currently rests on one bug, a five-case check, and a single annotator (you say so yourself), so it reads as a vivid existence proof rather than a measured result. Three concrete moves would close that gap: bring in an independent judge model, since right now the same gpt-4o-mini is grading itself; add a second annotator so the human-validation step has real inter-rater signal; and actually run the calibrated-abstention A/B against the Trap false-response rate and the S04 leak, which would turn the proposed mitigation from scaffolding into a result.

One last thing on the writing: you make the core point in the abstract, then again in 4.3, again in 5.2, and once more in the conclusion. I got it the first time, and it's a good point. Say it once and hit it hard, and you free up room for the assistant audit your title actually promises. Strong submission, and an honest one. I'd be glad to see where this goes next.

Cover the topic regarding the process of generating synthetic data. Or I would use a pre existing dataset from kaggle