SamplerScope: Exact decoder attribution for finite language-agent behavior
Ansh Dawda
SamplerScope tests whether behavior attributed to language model weights can instead be caused by the inference-time decoder. It caches grammar-constrained action logits from two Qwen2.5-Instruct checkpoints in two finite decision environments, applies 11 decoder configurations to the same logits, and computes each induced policy's outcomes exactly using dynamic programming. Across 24 controlled model-environment-label strata, greedy decoding improved benchmark return in 12 and reduced it in 12; decoder-only changes ranged from -1.224 to +0.313. Top-p sometimes removed all benchmark-optimal actions across 18.5% to 73.3% of decision occupancy, while exhaustive A/B/C mappings exposed strong surface-label sensitivity. SamplerScope provides a reproducible decoder-attribution toolkit and shows that observed agent behavior belongs to a model-prompt-grammar-decoder system, not model weights alone. It measures operational policies, not intrinsic preferences or sentience.
The paper shows, quite convincingly, that observed agent behavior can be highly dependent on the specific decoder used. This is taken to mean that research on model preferences should also control for the model's decoder. I agree with this point, but note that many safety evaluations use representative settings and evaluate the system-as-deployed. Still, the project is valuable and interesting and it exposes another "degree of freedom" in model assessment.
This is actually a very good study, the traces and controls set high bar for reproducibility. The results are from qwen checkpoints with 2 environments and 42 states with 1 letter action labels. It shows weakness of small model on multiple choice prompts rather than decoders.
Cite this work
@misc {
title={
(HckPrj) SamplerScope: Exact decoder attribution for finite language-agent behavior
},
author={
Ansh Dawda
},
date={
},
organization={Apart Research},
note={Research submission to the research sprint hosted by Apart.},
howpublished={https://apartresearch.com}
}


