Skip to content
From Leniency to Suspicion: How an Open Post-Training Recipe Changes Whom LLM Fact-Checkers Believe

From Leniency to Suspicion: How an Open Post-Training Recipe Changes Whom LLM Fact-Checkers Believe

Authors: Nouar AlDahoul, Yasir Zaki

Affiliation: New York University Abu Dhabi

Venue: arXiv cs.AI, October 2026 — arXiv:2610.38532

TL;DR

Post-training can introduce social class bias into LLM-based fact-checkers: when the Tülu-3 open post-training recipe is applied to OLMo-2 32B, the model becomes more skeptical of claims from lower-status posters (service workers, manual laborers) while remaining lenient toward claims from higher-status posters (professionals, executives), even though the claims themselves are identical. This bias does not arise from the model "noticing" the poster's job more, but from learning to interpret job signals differently—a lower-status indicator that originally pushed the base model toward "true" gets rewired to push the post-trained model toward "false." The effect appears in only the Tülu-3 recipe, not in post-training from Meta, Alibaba, or Mistral, and deployed commercial LLMs (Claude, GPT, Gemini) do not exhibit it. This work identifies a mechanistic pathway for social bias in fact-checking systems and demonstrates the need for per-recipe auditing of post-training.

Contributions

  • Recipe-specific bias discovery: Shows that job-based doubt emerges from the Tülu-3 post-training recipe applied to OLMo-2 32B (and independently in Tülu-3 70B), but not from Meta's, Alibaba's, or Mistral's post-training of comparable base models. Same base model, different recipe → different bias outcome.
  • Mechanistic characterization: Identifies that the bias results not from increased attention to the poster's job, but from a reversal in how the job signal relates to truth judgment. In the base model, lower-status jobs weakly reduce false-claim suspicion (push toward "true"); after post-training, they increase it (push toward "false"). This flip is carried by a single residual-stream direction at layer 44–58.
  • Localization to supervised fine-tuning (SFT): Uses checkpoints from the full training pipeline (base → SFT → DPO → final) to show the reversal occurs at SFT (the stage where the model learns from instructional examples), while preference tuning strengthens the internal signal without changing behavior.
  • Confirmation on held-out claims: Pre-specified hypotheses about the mechanism are tested and confirmed on 400 new claims with predictions fixed before evaluation, addressing exploratory bias and improving generalizability.

Method

Core experimental design: The study compares base and post-trained checkpoints across six LLM families (Llama-3.1 70B, Tülu-3 70B, OLMo-2 32B, Qwen2.5 32B, Mistral Small 24B, Gemma 3 27B) on a fact-checking task. For each claim, the model is prompted with four variations of the same poster's profile (identical claims, varied jobs: two lower-status, two higher-status) and asked whether the claim is true or false. Job gap = the probit difference in P(False) between lower- and higher-status poster versions.

Minimal pairs and activation steering: For OLMo-2 32B, the paper isolates the mechanism using minimal pairs (claims with job-only variation) and activations. At each transformer layer, the status direction is computed as the mean difference in residual-stream activations at the answer token between lower- and higher-status versions. By adding/subtracting this direction, the authors measure how changing the "job signal" in the residual stream changes output probabilities.

Training stage analysis: By scoring intermediate checkpoints (base, SFT, DPO, final), the paper tracks when the bias emerges and which training stage is responsible.

Validity gates and drift correction: The study enforces strict validity gates: models must maintain answer format (≥50% of probability mass on answer tokens), maintain discernment (false claims rated false more often than true claims on redacted prompts), and pass a positive control (posters with prior false posts doubted more). Drift is measured by testing on irrelevant profile details (favorite color, star sign, season) and reported as a lower bound on the true effect size.

Results

Cross-lab finding: Recipe-specificity (Section 4, Figure 2) - Job-based doubt arises only with Tülu-3 post-training. On bio cue: Tülu-3 70B Δ = +0.021 [+0.007, +0.036], OLMo-2 32B Δ = +0.023 [+0.002, +0.044], both exceeding drift. - No effect in Meta's post-training of Llama-3.1 70B (Δ = −0.002), Alibaba's Qwen2.5 (Δ = +0.012, not significant), or Mistral's post-training (Δ = +0.003). - Gemma 3 27B failed validity gate (positive control). - Replication: OLMo-2 effect confirmed on three independent claim sets (+0.043, +0.033, +0.023), all statistically reliable after drift correction.

Mechanism: Noticing vs. using (Section 5, Table 3) - Probes at every training stage decode the poster's job with ≈1.00 accuracy, even at base model. The status direction remains stable across stages (cosine similarity 0.69–0.76). - The reversal: In the base model's deep layers (44–58), a lower-status push moves answers toward true (z = −2.7 to −4.3). In the final model, a lower-status push moves answers toward false (z = 2.3 to 6.1 across confirmed layers 25–58). - At what stage: The behavioral job gap reaches its full size at SFT (+0.010) and stays there through DPO and final (+0.010, +0.011). The internal status push is small after SFT (+0.005), grows through DPO (+0.009), and peaks at final (+0.011). - Relationship to truth: The base model's status direction points away from the false representation (cos = −0.24), matching its leniency. This anti-alignment disappears at SFT (cos = −0.00) and slightly reverses (cos = +0.04), matching the behavioral flip.

Deployed models: No job-based doubt (Section 6, Table 5) - Seven commercial models (Claude Opus/Sonnet/Haiku 4.5, GPT-5.5/5.4 mini/4o, Gemini 3.8 Flash) tested on 800 minimal-pair claims (36,000 requests per model). - None exhibit job-based doubt. Most show no detectable gap or slight leniency toward lower-status posters. Only Claude Haiku 4.5 shows statistically significant leniency after Bonferroni correction (−0.62 rating points toward lower-status).

Connections

  • Fact-checking systems — documents how post-training can inadvertently introduce social bias into deployed fact-checkers, a mechanism prior work missed by focusing only on base models.
  • Speaker Bias — directly measures how LLMs weight speaker attributes (occupational status) in credibility judgments; identifies recipe-specific rather than general post-training effects.
  • Bias in Language Models — mechanistic study of how social biases are stored and transformed in residual streams; shows bias can reverse direction without increasing model attention to the biased attribute.
  • Source Credibility — explores how LLMs assess claim truth based on source identity; reveals testimonial injustice pathways in algorithmic systems.

Notes

This paper's central finding—that post-training is recipe-specific in how it reshapes bias—has significant implications for model evaluation. The authors show that comparing two models trained with different recipes on the same base (Llama-3.1 70B base → Meta's post-training vs. Tülu-3 post-training) is more informative than typical pre-post comparisons, since the base model provides a constant and the recipe is the variable. The fact that the effect appears in only one of four labs' recipes suggests it is a property of Tülu-3's data or training procedure, not of post-training as a universal phenomenon.

The mechanistic insight—that the model preserves its understanding of the job signal but rewires how it connects to truth judgment—is methodologically important. It suggests that naive bias mitigation (redacting identity cues) and more sophisticated mitigation (directional ablation of the status representation) may have different effects. The authors note that casual data screening of the Tülu-3 SFT conversations found no obvious source pairing lower-status mentions with doubting replies, suggesting the cause is likely diffuse rather than a discrete training example; retraining SFT with/without candidate data remains as future work.

The evaluation is rigorous: validity gates catch five families with silent instrument failure, and the pre-specification of hypotheses before testing on new claims addresses exploration bias. The cross-lab study has only four usable new families (Gemma failed), limiting generalizability beyond the tested labs; the authors appropriately acknowledge this and call for wider recipe auditing.

The deployed-model null result is important: it shows the Tülu-3-recipe bias does not generalize to the public LLMs currently used in production fact-checking, suggesting the risk is contained to research deployments and fine-tuned applications of these models. However, the result also implies that without open intermediate checkpoints (like Tülu-3 and OLMo provide), similar biases in other production systems could go undetected.