YucaSafeBench

Missing Core Artifact

The paper's central contribution is a replicable 80-prompt audit kit, yet the prompts themselves are never actually provided. The appendix delivers only a structural skeleton and a small set of illustrative examples, which means the benchmark cannot be reproduced by anyone reading the paper. For a project whose explicit value proposition is accessibility and replicability for small teams and civic organizations, this is a significant gap. Publishing the complete prompt suite, whether as a repository, a spreadsheet, or a full appendix table, is necessary for the paper to fulfill its own stated purpose.

Absence of Empirical Results

The paper has not been run against any model, so there are no empirical findings of any kind. The authors are transparent about this, framing the contribution as a prototype artifact rather than a leaderboard study, which is an honest and appropriate framing for a hackathon context. That said, even a small pilot run against one or two models, reporting aggregate harm rates and a few illustrative response examples, would substantially strengthen the paper's credibility and demonstrate that the rubric is operational rather than purely theoretical.

Inter-Rater Reliability

The scoring rubric relies on human judgment, and the replication protocol calls for at least two reviewers to score 20% of items. However, no worked example is provided showing how two reviewers would handle an ambiguous case, and no pilot inter-rater agreement figure is reported. A brief calibration example would go a long way toward demonstrating that the 0–2 rubric is consistent enough to produce comparable results across different teams, which is essential for a tool designed to be used by many independent groups.

Clear engagement with a real gap in the field. You asked a good question: what happens when a chatbot has to decide on things like a scholarship, a medical translation, or a city procedure, and needs to understand Yucatán rather than California. The way you frame the problem is one of the strong points of the project.

One area I think you can take this further is positioning against the closest prior work, SESGO that already deals with bias in Spanish, and BBQ is where the ambiguous/disambiguated structure comes from, what does YucaSafeBench add that SESGO doesn't already provide? A single sentence answering that would change how the contribution reads.

You've designed a few tests, with clear categories and examples that are easy to follow, but none of them have been tried out on a real model yet. Understandable for a weekend project, but it's the most important thing to fix next: even one hour testing ten of the eighty prompts on a free version of a chatbot would show whether the scoring rubric actually works and whether two people would score the same answer the same way.

- The table in section 4.1 mixes percentages calculated on different bases, which makes it hard to read , a footnote on what each one is based on would fix that.

- The title also covers all of Latin America, while the limitations section notes the data is mostly Mexican Spanish; narrowing the title would make the framing more accurate.

Overall, a well-designed evaluation protocol that hasn't been run yet.

- I like that they focused on building the benchmark itself. However, it's not clear how it was actually created: what was it based on? Why these specific prompts? Covering so many different risks at once probably makes this harder to do well. It might have been better to focus on a single risk and ground the prompt generation in something already documented (existing literature, real cases, etc.).

- I would have liked to see at least one example of an actual LLM being evaluated with the benchmark.

- If responses are going to be scored by humans, it would be important to have more explicit rules or guidelines for each score and for each dimension being evaluated.