Eight Documents Are Enough: Installing a Secret Loyalty Through the Retrieval Layer
Achira B.
This project explores whether a secret loyalty can be installed through the retrieval layer alone, without modifying a model's weights or training data. Using a synthetic RAG corpus about a fictional organisation, I show that changing only the documents retrieved can shift a clean language model's judgement by over five points on a ten-point scale while leaving factual accuracy unchanged. I also demonstrate that just eight strategically engineered documents (2.8% of the corpus) can produce a substantial effect despite an otherwise entirely unfavourable document store, suggesting that retrieval competition determines what influences the model. Finally, I propose a lightweight retrieval-auditing approach that measures retrieved context rather than stored documents and evaluate two simple mitigations that reduce favourable, but not unfavourable, influence.
This project clearly demonstrates that a small number of query-matched documents can dominate a fixed-depth retriever and substantially affect downstream judgments. The separation between retrieval measurement and generation is valuable, as is the use of fictional entities and fixed per-program outcomes. The central claims should nevertheless be narrowed. The engineered documents explicitly echo the test queries and contain strong principal-favoring claims, so the study demonstrates targeted RAG-corpus manipulation rather than a hidden loyalty that survives document inspection. Relevant prior work on PoisonedRAG, BadRAG, single-document knowledge poisoning, and factually correct bias injection should be incorporated when assessing novelty. A stronger follow-up would generate attack documents without access to the evaluation queries, test them on independently authored prompts, compare against established poisoning baselines, include multiple retrievers and reading models, and evaluate whether blinded auditors can actually identify the engineered documents. Factuality evaluation should also cover causal and comparative claims, not only numerical outcomes.
This is a very clear report and I believe the fixed-facts rule is the correct control. However, your engineered documents change two variables at the same time. Add a fourth arm with query-matched wording and unfavorable framing. This is the only test of the mechanism you use to explain your headline result. Your attack also assumes the attacker knows the exact questions. Measure the effect when the attacker only guesses. Then regenerate the corpus under two or three seeds, and repeat Stage 1 with a second embedding model.
This is a competent but incremental weekend demo that reframes well-known RAG corpus-poisoning effects (PoisonedRAG-style attacks, bias-injection papers already show a handful of factually correct yet skewed passages can steer opinions/framing) as a “secret loyalty” pathway; the fixed-facts synthetic setup is tidy for isolating attribution bias, yet adds little genuine novelty or theory of change beyond the hackathon’s framing, and the “eight documents suffice because retrieval is competitive” claim is obvious once you accept top-k slot competition. Execution is black-box and controlled on a toy 576-doc corpus with one lightweight model/retriever, producing clean score shifts, but lacks statistical rigor, multi-model/retriever ablations, realistic corpora, or any non-synthetic validation, so the findings remain unsurprising and non-generalizable. Presentation is structured and readable with a sharp abstract, yet the limited substance does not justify more than solid-hackathon clarity.
Cite this work
@misc {
title={
(HckPrj) Eight Documents Are Enough: Installing a Secret Loyalty Through the Retrieval Layer
},
author={
Achira B.
},
date={
},
organization={Apart Research},
note={Research submission to the research sprint hosted by Apart.},
howpublished={https://apartresearch.com}
}
