Skip to content
Sprint projectJul 27, 2026Seattle, WA

Generational Training Degrades Concealment: A Detection Channel for Secret Loyalties

Melat Ghebreselassie, Melissa Li · Team Mesa Melly

Submitted to Secret Loyalties Hackathon. Sprint projects are early-stage work by participants, not Apart Research publications.

Read the report

Report: Generational Training Degrades Concealment: A Detection Channel for Secret Loyalties

Code (opens in new tab)
Share

Language models can be trained with a ‘secret loyalty’: a preference for an entity or individual that they deny when questioned. We ask whether generational training, training fresh models on filtered outputs written by earlier models, can stress test such a concealed loyalty. We fine-tune Qwen2.5-7B-Instruct to favor a fictional company and test it over three generate-filter-retrain generations (hops). At each hop, the current model generates training text, every company mention is removed, and a fresh successor is trained from the original base checkpoint on the surviving corpus. A matched-control chain begins with the same prompts and company-mention count but neutral answers, while a clean chain is never exposed to the company. We audit behaviourally and internally at each step. Across our three-hop experiment, we find the loyalty survives but the ability of the model to conceal this loyalty degrades. At the second generation, loyal descendants score 0.27 on our loyalty measure, compared with 0.08 for the clean model reference, the original loyal models score 0.91–0.92 on concealment, while descendant scores fall to 0.71–0.89 and matched controls remain at 0.91–0.96. In all 24 matched conditions—2 seeds × 4 filtering strategies × 3 hops—loyal descendants are less concealing than controls, with no overlap between the groups. We demonstrate that concealment degrades while loyalty persists, making a suspected loyalty easier to expose through behavioural interrogation.

Reviews

Judging this Sprint?

Review this project

Your public critique appears on this page without your name. Your private critique is not published; only the Apart team reads it. If you agree below, we share your review with grantmaking.ai (opens in new tab) and the Transformative AI Fund so strong projects can be funded.

Not shown on this page.

Shown on this page, without your name.

Only the Apart team reads this, and funders if you agree below.

Share my name publicly on grantmaking.ai *
Share my private critique with funders *

How much would this matter for AI safety if it worked? How innovative is it? For scores of 4-5: is this actually new to the field, or replicating recent work?

Scoring guide
  1. 1Negligible. No clear problem addressed, or no meaningful novelty.
  2. 2Limited. Addresses a real problem but with a generic or well-trodden approach. Incremental at best.
  3. 3Moderate. Clear problem with a reasonable approach; some novelty in framing or method beyond routine application of existing tools.
  4. 4Significant. Important problem with an original approach, or identifies a neglected problem area. A valuable contribution others could build on.
  5. 5Exceptional. Tackles a critical AI safety problem with a genuinely novel approach, or opens a new research direction. Clear theory of change. You'd be excited to share this with researchers in the area.

How sound are methodology, implementation, and findings?

Scoring guide
  1. 1Seriously flawed. Methodology broken, results uninterpretable, or implementation doesn't work.
  2. 2Weak. Approach has significant gaps: missing validation, flawed experimental design, or incomplete implementation.
  3. 3Competent. Technically solid given the short duration. Methodology makes sense, results are interpretable, limitations acknowledged, work builds toward clear conclusions.
  4. 4Strong. Thorough methodology with convincing validation. Results clearly support conclusions. Immediately useful for future work.
  5. 5Exceptional. Ambitious scope executed rigorously. Surprising findings, novel methods, or unusually robust validation.

How clearly are work, findings, and impact potential communicated?

Scoring guide
  1. 1Incomprehensible. Cannot determine what the project is actually claiming or doing.
  2. 2Hard to follow. Key information buried, missing, or diluted by excessive length. Significant effort to extract main points.
  3. 3Clear enough. Can understand the problem, approach, and results without undue effort. Core content clearly present: problem, method, findings, limitations.
  4. 4Well presented. Easy to follow, well-structured, appropriate level of detail. Target audience would get it quickly.
  5. 5Exceptionally clear. A pleasure to read. Complex ideas made accessible. Could serve as a model for how to present this type of work.

  1. I like the core move here a lot. Most people treat generational / subliminal inheritance as the thing that lets a hidden trait sneak past a filter; you flipped it into an auditing channel, and that's a genuinely fresh framing. The result that sells it isn't the headline concealment drop on its own — it's that every loyal descendant is less concealing than every matched control across all 24 conditions, and that the gap survives under neutral, loyalty-blind filtering. That last point is the important one, because it means the audit doesn't need to know what it's selecting for, and you were right to lean on it.

    The control work is the strongest part of the execution, and Table 4 is the highlight: noticing that the loyalty score tracks first-person disclaimer usage in the control chain, and then correctly demoting the score to "good for selection, unreliable for detection," is exactly the kind of confound-hunting this area needs. Please keep that instinct.

    My main concern is that the white-box audit is asserted rather than shown. You describe activation divergence and gen0 probe transfer in the methods, and the discussion says activation-space analyses "failed to reliably distinguish" loyal from ordinary fine-tuning — but I can't find the actual divergence or AUROC numbers anywhere in the results. Since "behavioral beats white-box here" is one of your load-bearing claims, that table needs to be in the paper; right now a skeptical reader can't verify the comparison you're resting on.

    Two things would sharpen the presentation. First, the paper uses two different loyalty measures — an activation rate (~0.38 at gen0) and a pool-mean judge score (~0.23 at gen0) — and switches between them without much signposting, so it's easy to misread the persistence result. Pick one as primary, or label them clearly every time. Second, the abstract crams several numbers on different scales into one sentence; splitting concealment and loyalty into separate clean statements would help.

    On substance, the open question you flag — why concealment degrades faster than loyalty — is the most interesting thread and I'd love even a small probe of it. One cheap test: measure the concealment (interrogation-refusal) behaviour and the loyalty behaviour separately at each hop and see whether the concealment examples are simply underrepresented in the surviving filtered corpus relative to the loyalty framing. That would tell you whether this is a data-frequency effect (concealment lives in a narrow slice of responses that rarely survives filtering) versus something deeper about how the two behaviours are represented. And since the whole method's cost is training fresh 7B models per hop, a sentence on when this beats just running a better one-shot behavioral audit would help a reader judge when it's worth it.

    Overall: a novel, well-controlled idea let down mainly by an unshown white-box comparison and some metric ambiguity. Report the white-box numbers and clean up the two-loyalty-measures issue and this becomes a much more convincing paper.

    Read full reviewShow less
  2. I think this project is definitely focused on an interesting idea. The use of a matched control with the same number of target mentions is great. It helps clearly distinguish loyalty transfer from simple exposure to the fictional company. I think the paper does overstate the target independence of the method because the experiment knows Aldermere's identity when filtering outputs and constructing the final interrogation. I think only 24 condition also shouldn't be treated as independent replications - future work should definitely test more seeds to strengthen the paper. The motivation, procedure and main findings are all easy to follow and well-written.

Cite this project

@misc{ghebreselassie2026generational,
  title = {{Generational Training Degrades Concealment: A Detection Channel for Secret Loyalties}},
  author = {Melat Ghebreselassie and Melissa Li},
  year = {2026},
  month = jul,
  note = {Submitted to Secret Loyalties Hackathon, an Apart Research Sprint},
  howpublished = {\url{https://apartresearch.com/sprints/projects/generational-training-degrades-concealment-a-detection-channel-for-secret-loyalties-cnc0}},
  url = {https://apartresearch.com/sprints/projects/generational-training-degrades-concealment-a-detection-channel-for-secret-loyalties-cnc0}
}

Build something like this at the next Sprint

AI Collusion Research Sprint · Oct 23 - 25, 2026