Thinking clearly about causal inferences of politically motivated reasoning: Why paradigmatic study designs often undermine causal inference

Ben M. Tappin; Gordon Pennycook; David Rand

doi:10.1016/j.cobcha.2020.01.003

In brief

Two common experimental paradigms for studying politically motivated reasoning—Outcome Switching (varying a condition's implication for political identity while holding substantive content constant) and Party Cues (varying whether information is endorsed by a politically aligned or opposed source)—often violate the excludability assumption necessary for causal inference. Both designs confound political motivation with other variables (prior belief coherence, source trustworthiness) that independently affect reasoning outcomes, undermining causal identification. The authors use a directed acyclic graph framework to clarify the confounding and discuss implications for intervention design.

Contributions

Identifies systematic violations of the excludability assumption in two paradigmatic experimental designs widely used to study politically motivated reasoning
Demonstrates how treatment assignment in both designs affects multiple causal pathways, confounding estimates of political motivation's direct effect
Provides a formal directed acyclic graph (DAG) framework for visualizing and reasoning about causal confounding in these designs
Discusses practical implications: paradigmatic designs may be inadequate for causally identifying politically motivated reasoning, even when effects are statistically significant
Proposes design features (e.g., statistical control, feature selection) that better isolate political motivation and improve causal inference

Method

The paper analyzes two experimental designs theoretically using causal inference frameworks (DAGs and the excludability assumption from experimental design):

Outcome Switching: Subjects receive identical substantive information but with different implications for political identity. For example, both receive evidence about a test of "open-mindedness," but are randomly told that either people with liberal beliefs or conservative beliefs score higher. Subjects then evaluate the test's validity. Researchers infer politically motivated reasoning when evaluations differ by condition.

Party Cues: Subjects learn about a policy or claim and are randomly assigned to learn either that the Democratic or Republican party endorses it. Researchers infer politically motivated reasoning when support/opposition follows the party cue.

The analysis shows that in both cases, the treatment assignment (political implication or party cue) does not only affect political motivation but also alters other variables that independently influence reasoning—specifically, the coherence between new information and prior beliefs, and perceived source trustworthiness. These confounding pathways violate the excludability assumption required for causal inference.

Results

For Outcome Switching: When the test is described as validating one political group's openness and invalidating the other's, subjects' prior beliefs about which group is genuinely more open become relevant to their evaluation. The design thus confounds the effect of political motivation with the effect of prior-belief coherence. Studies showing empirical examples (e.g., subjects evaluating a test based on whether the "correct" answer aligns with their prior beliefs) demonstrate that prior beliefs alone can drive reasoning differences even in the absence of political motivation.

For Party Cues: Party cues signal not only political alignment but also source trustworthiness. Subjects rational based on the trustworthiness of the endorsing party, independent of motivated reasoning. Evidence from prior work (Bolsen et al.) shows that trust in one's own party substantially moderates party cue effects, suggesting subjects are responding to trustworthiness rather than motivated reasoning.

General implication: The correlation between political motivation and these confounding variables (prior beliefs, source trust) explains empirical findings in prior literature, including why party cue effects are larger among politically engaged individuals (who have clearer prior beliefs and party attachments) and why some results are difficult to reconcile with a pure motivated reasoning account.

Notes

This is a critical methodological paper written by leading misinformation researchers. Its core contribution is not an empirical finding but a causal-inference argument: two very common experimental designs, despite widespread use in the literature, systematically confound political motivation with other variables. This matters because it affects not just our understanding of politically motivated reasoning but the design of interventions—if we misidentify the causal mechanism, interventions targeting the wrong lever will fail. The paper's use of DAGs makes the argument transparent and reproducible. The implications extend beyond motivated reasoning to any study claiming causal effects on belief or reasoning involving political or identity-relevant information.

The paper draws on Bayesian reasoning theory and empirical evidence (e.g., Koehler on belief bias, Bolsen on party trust effects) to argue that prior-belief coherence and source trustworthiness are general cognitive principles that operate independently of political motivation, yet are confounded by the paradigmatic designs. This framing opens a research agenda on better designs.