Filtrum-Safety: LLM-Agnostic RAG for Reducing Legal Hallucinations in Civil-Law Contexts

Filtrum-Safety se ocupa de un riesgo especialmente delicado en el uso de modelos de lenguaje para tareas jurídicas: la generación de respuestas legales con apariencia de certeza, pero sin soporte normativo verificable. Este riesgo es particularmente sensible en sistemas de tradición civilista, como los latinoamericanos, donde el razonamiento jurídico depende de códigos, leyes escritas, jurisprudencia, reglas de competencia y jurisdicción aplicable. En ese contexto, el valor del proyecto está en reducir la dependencia de la memoria interna del modelo y ofrecer una capa de apoyo basada en documentos recuperados, fuentes trazables y contexto verificable.

Un mérito importante del trabajo es su prudencia: no presenta la herramienta como una solución que elimina las alucinaciones, sino como un mecanismo para reducir su superficie al separar la recuperación documental de la generación de texto y mantener la supervisión humana como necesaria. Esa delimitación es clave, porque el aporte más sólido del proyecto no está en sustituir el criterio profesional, sino en hacer más verificable, auditable y controlable el uso de IA en razonamientos legales.

También conviene situar el aporte frente a lo que ya existe. El uso de RAG para anclar respuestas en documentos recuperados no es nuevo, y el RAG legal ya es un campo activo. La contribución distintiva de Filtrum-Safety parece estar más en su aplicación a sistemas jurídicos de tradición civilista en español, en la portabilidad del Markdown trazable y en la separación entre recuperación y generación, que en el método técnico por sí solo.

La ejecución todavía necesita fortalecerse. La arquitectura es clara y el enfoque es útil, pero la validación se apoya esencialmente en un único caso evaluado con la matriz del propio equipo, sin permitir inferencia estadística, algo que el mismo trabajo reconoce. El chequeo complementario con varios modelos es ilustrativo, pero también cualitativo. Una versión más robusta debería probar el sistema con más casos y en condiciones adversas: corpus incompletos, fuentes desactualizadas, normas derogadas o modificadas, documentos de distintas jurisdicciones y fragmentos semánticamente relevantes pero jurídicamente insuficientes.

Desde la perspectiva de gobernanza, rendición de cuentas, auditoría y supervisión humana, el proyecto podría fortalecerse explicando con mayor detalle cómo se controla el ciclo de vida del conocimiento jurídico que alimenta la herramienta. En un sistema basado en recuperación documental, la seguridad no depende solo de que el modelo cite fragmentos, sino de que exista claridad sobre quién selecciona y valida los documentos, cómo se verifica su vigencia y jurisdicción, cómo se gestionan normas derogadas o modificadas, qué registro queda de las fuentes utilizadas, qué límites de uso se establecen y cómo se protege la información sensible.

En últimas, la trazabilidad documental mejora la verificabilidad, pero no reemplaza la gobernanza del corpus, la auditoría del proceso, la supervisión humana ni la responsabilidad profesional del abogado o funcionario que interpreta y utiliza la salida del sistema.

Clear engagement with a real gap in the field. The threat model, an LLM confidently mixing Common Law categories into a civil-law answer or citing authorities that don't exist, is well chosen, and the supplemental check across Claude, Gemini, and DeepSeek is a genuine strength, since it shows the retrieval layer changes model behavior across three different interfaces rather than just the one the team built around.

One area I think you can take this further is positioning against the closest prior work. Lima already proposes multilayer embeddings for legal text and LegalGraphRAG already does structured retrieval for legal reasoning, so what does Filtrum-Safety add beyond model-agnostic Markdown export? A single sentence naming that explicitly would change how the contribution reads.

There's also a deeper issue worth resolving, and it goes beyond a typo. The paper states the prototype used jina-embeddings-v2-base-es for its Spanish orientation, but the repository's EmbeddingConfig instantiates AllMiniLmL6V2QuantizedEmbeddingModel instead. The paper also describes chunking that "respects legal boundaries such as articles, subsections, or sections," but ChunkingService.java splits purely by sentence boundary and character count, with no legal-structure detection at all. These two are confirmed directly against the code in the repository.

On top of that, the video walkthrough states a 55% similarity threshold, which matches the default in application.yml, while the paper and its own Figure 4 state 0.65; this third point is based on the paper's own text rather than a direct code check, so it's worth the team double-checking which value the demo actually ran with.

None of these look intentional, but together they mean at least two, and possibly three, methodological claims in the paper don't match what the code actually does, which puts the 427-chunk count and the headline 10/10 score on uncertain footing.

The evaluation itself also rests on a single self-graded case, and the most useful next step is small: the team already collected three more model outputs in the supplemental check, so scoring those with the same matrix would turn one data point into four. The title also promises civil-law contexts broadly while the corpus is entirely Mexican; narrowing it would match what was actually tested.

Overall, a working prototype with a useful cross-model check, built by a team that clearly understands the problem they're trying to solve. Before the quantitative results can carry much weight, though, it would help to align the paper's description with the current state of the code, particularly around the embedding model, the chunking approach, and the similarity threshold used in the demo.

The instinct here is right: civil-law users get LLMs that invent citations and smuggle in Common Law categories, and forcing the answer to start from documents the user actually uploaded is a sane way to shrink that surface. The build is real, the Markdown export is genuinely useful (LLM-agnostic, and it sidesteps free-tier upload caps), and the cross-model check is the best thing in the paper. DeepSeek saying "this context isn't enough" is exactly the behavior you want, and it's real signal. One thing you underplay: your strongest asset is the regional angle you treat in passing. You've got a specific, underserved jurisdiction (Mexican civil law, a real amparo query, federal-versus-Jalisco sources), and that's exactly where English-built legal AI fails Latam users, but the paper reaches for "any document-intensive domain, health, education, engineering" instead of planting a flag there.

Lead with the civil-law Global South stake, don't bury it under multidomain ambition. Where this has to grow is evidence, because right now it's one query, one corpus, scored by a five-criterion matrix you built and then handed your own tool a 10/10 on, and a perfect score for your own system on your own rubric on a single case reads as a demo with numbers, not proof. You say n=1 yourself. Three moves fix it: run 20-30 queries with a blind scorer who doesn't know which output came from Filtrum, measure the retrieval itself (are the pulled chunks actually the right articles, precision and recall on those 427) since that's the part safety rests on, and add a real failure case where retrieval misses or the LLM ignores the context, because a tool that only shows its wins is hard to trust. The idea deserves that test, and the build is far enough along to run it. Tighten the evidence, lean into the Latam stake, and this is something a law school clinic in Mexico could actually pick up.