Bayesian or biased? Analytic thinking and political belief updating

Contributions

• Addresses the "sophistication paradox": why cognitively sophisticated individuals show the widest partisan disagreement on factual questions
• Tests the hypothesis that cognitive sophistication magnifies politically biased processing using a Bayesian belief-updating framework
• Develops a novel experimental design that measures both information evaluation and actual posterior belief change, addressing prior limitations where information evaluations and belief updates diverge
• Finds that greater analytic thinking (Cognitive Reflection Test performance) is associated with more normative, less biased belief updating relative to an unbiased Bayesian benchmark
• Provides evidence that the sophistication paradox may reflect something other than amplified political bias in belief updating

Method

Two preregistered experiments with large samples (Study 1: N=501, Study 2: N=1,004) recruited via Amazon Mechanical Turk. In both studies, subjects:

1. Provided prior beliefs about 16 political statements (Democrats/Republicans, pro/con) as percentages on a 0–100 scale
2. Received noisy signals about each statement's truth value; signals were accurate with probability 2/3
3. Reported posterior beliefs after each signal
4. Completed the Cognitive Reflection Test (CRT), a seven-item measure of analytic thinking (propensity to override intuitive but incorrect responses)
5. Reported political party preference (Democratic or Republican)

The researchers computed the Bayesian benchmark for posterior beliefs by applying Bayes' rule to each subject's prior belief and the signal they received. Subjects' actual posterior beliefs were compared to this benchmark to quantify politically biased deviations (beliefs that deviated further from the benchmark when the signal favored their political identity).

Study 2 introduced an additional manipulation: subjects provided subjective judgments about the likelihood ratio (informativeness) of each signal, allowing the researchers to compute Bayesian posteriors that incorporated subjects' own assessments of signal reliability. This design change addressed the potential confound that differences in how subjects reasoned about signal diagnosticity (rather than belief updating per se) could explain Study 1 results.

Results

Study 1 (N=501):

• Primary finding (H1): Higher CRT performance correlated with posterior beliefs closer to the Bayesian benchmark (less overall deviation). This association was small but consistent across both outcome variables (difference and log-ratio operationalizations).
• Secondary finding (H2): Contrary to the "magnified political bias" hypothesis, high CRT scorers did not show greater asymmetric bias (updating more on politically favorable signals, less on unfavorable). When evaluated against the Bayesian benchmark, apparent differential updating reflected low CRT scorers exhibiting anti-political bias (overreacting to unfavorable signals), which high CRT scorers corrected.
• Control study: To rule out regression-to-the-mean and natural variation in belief reporting, a parallel study without new information (signals hidden) found no correlation between CRT and deviation from Bayesian posteriors, confirming that observed effects in the main study were driven by belief updating, not measurement artifact.

Study 2 (N=1,004):

• Primary finding (H1): Replicated Study 1: higher CRT correlated with posterior beliefs closer to the Bayesian benchmark.
• Secondary finding (H2): Likelihood-ratio analysis showed the same pattern: high CRT scorers updated less on unfavorable signals and more on favorable signals in the raw data, but this pattern did not constitute magnified political bias when evaluated against the Bayesian benchmark. The pattern instead reflected correction of the anti-political bias observed among lower CRT scorers.

Statistical significance: Likelihood ratio tests (Table 3) showed the CRT × signal favorability interaction was statistically significant in both studies and both outcome variables (χ² = 3.97–6.81, p = 0.009–0.046).

Interpretation: The data are inconsistent with the hypothesis that cognitive sophistication magnifies politically biased belief updating. Instead, they support the hypothesis (H1) that greater analytic thinking correlates with more normative (Bayesian-optimal) belief updating.

Notes

Methodological strengths: The use of an explicit Bayesian benchmark allows researchers to distinguish politically biased deviations from normative updating. Study 1 includes a clever control condition (hidden signals) that rules out regression to the mean. Study 2 addresses the confound that different reasoning about signal reliability (likelihood ratios) could explain the CRT–belief-updating association.

Limitations and open questions: The political statements, though pre-tested for partisan flavor, were relatively mild and not strongly politicized or culturally entrenched issues like gun control or abortion (which tend to elicit the strongest partisan disagreement). This design choice was intentional—to simplify the Bayesian task—but it may limit generalizability to real-world high-salience political disputes where cognitive sophistication effects might differ. The authors discuss this explicitly and note that future work should test more culturally salient or identity-threatening issues.

The finding that low CRT scorers exhibit anti-political-bias updating (overcorrecting unfavorable signals) is counterintuitive and deserves further investigation. The authors suggest it may reflect greater variability in belief reporting among lower CRT scorers, regression to the mean effects, or failure to fully integrate signals with prior beliefs, rather than a genuine tendency to be "fair" in an unbiased direction.

Significance for the field: This work directly challenges the "sophistication amplifies bias" hypothesis by showing, at least under the conditions tested, that analytic thinking correlates with more optimal information processing. It suggests the sophistication paradox—that disagreement widens with education—may reflect causes other than magnified political bias in how sophisticated individuals process information. Whether this finding generalizes to more culturally salient or identity-threatening political issues remains an important open question.