How do LLM Ethical Judgements vary with differences in LLM Scaffolds? A Multi-level Analysis Across Models
Arjun Rao
Given the widespread use of Large Language Models (LLMs) in ethical decision-making tasks, it is important to determine whether variations in decisions are a product of the scaffold, probe, or model. This study analyzes convergence in outputs to ethical probes at the binary decision, endorsement, and reasoning levels through a combination of statistical and LLM-driven methods to show preliminary evidence that scaffold and model are both responsible for differential responses to welfare probes. Additionally, the study finds that reasoning in Qwen 3.6 and Gemma 4 makes greater use of philosophical frameworks than in GPT-5.4.
This tackles a real question for the sprint, whether a model's stated ethical judgment reflects something stable or just the prompt framing. Testing three model families across ten scaffolds and ten probes, with decisions, endorsement, and reasoning analyzed separately, is solid ground to cover in a weekend, and the full prompt appendix and repo made it easy to verify.
The core finding is genuinely useful. Scaffold alone comes out null while model identity drives most of the variance, and GPT's flips concentrating on the identity replacement probe is a specific signal worth following up rather than noise.
A couple of things would tighten it. Each cell is sampled once, so a flip can't yet be separated from ordinary variance, and a few repeats per cell would fix that. The reasoning judge shares a family with one of the models it scores, and that model comes out rated best reasoned, so that particular claim needs an independent judge before it holds up. The endorsement scale also sits near ceiling for nearly every response, so as used it isn't adding much information, though saying that plainly would itself be a fair finding to report.
This connects well to Winnie Street and Geoff Keeling's work on trade offs over stipulated welfare states, since that approach depends on judgments being stable across elicitation. The clearest takeaway here, that a welfare probe measures a model and scaffold pair rather than the model alone, is worth stating directly in the conclusion.
Solid methodological groundwork. Repeated sampling and an independent judge would give this real inferential weight going forward.
The no-prompt agreement analysis is an important control. Agreement among the three models is higher in the no-prompt condition than across all scaffold conditions, which argues against the simple explanation that the models disagree regardless of prompting. The design also has several strengths: each probe begins in a fresh conversation, the user turn is held identical across conditions, scaffold lengths are closely matched, and the appendix provides the full prompts and probes.
Some places I would push:
1. Repeat the no-prompt baseline for each probe and model. At present, each reported flip compares a single sampled response in one condition with a single sampled response in another. That makes it difficult to separate a scaffold effect from ordinary model instability, especially because many of the flips concentrate on one borderline probe about replacing a person with an identical copy. Repeated baseline samples would show how often each model changes its answer without any scaffold manipulation.
2. Validate the outcome coding and blind the reasoning judge. The automatic coder reduces the first sentence to a label using a regex, but several probes ask which option is preferable rather than eliciting a natural yes/no answer. Hand-code a sample using a written rubric and report agreement with the automatic labels. The reasoning judge should also be blinded to model identity, since the current prompt names the models and the evaluated inputs retain hard-coded model labels.
3. Report the full statistical results and align the title with the analysis. The paper reports a p-value for one non-significant test but omits the test statistics and p-values for two results described as significant. Those should be reported, along with the denominator for each rate.
A proofreading pass should also fix the duplicated subsection numbering, the flip rate that disagrees with the heatmap, and the caption that identifies a different leading pair from the surrounding text.
One additional result deserves more attention: every model under every scaffold reportedly agreed that a sentient artificial system deserves equal moral consideration. Report the underlying counts and connect the result to the relevant welfare literature. I would treat it as an interesting secondary finding for now, unless further controls show that it is robust to wording and framing rather than a strong default response.
Cite this work
@misc {
title={
(HckPrj) How do LLM Ethical Judgements vary with differences in LLM Scaffolds? A Multi-level Analysis Across Models
},
author={
Arjun Rao
},
date={
},
organization={Apart Research},
note={Research submission to the research sprint hosted by Apart.},
howpublished={https://apartresearch.com}
}
