
Contributions
- First empirical comparison of effectiveness across the full array of contemporary fact-checking source types (professional fact-checkers, news outlets, social media, AI, crowdsourcing)
- Empirical validation that perceived credibility of fact-checkers mediates fact-checking effectiveness, extending the MAIN model (Sundar, 2008) to the fact-checking context
- Demonstration that perceived credibility plays a moderating role: when credibility is high, fact-checks are more effective for unaligned individuals (those who initially disbelieved the false claim), but credibility differences matter less for aligned individuals
- Evidence that analytic thinking predicts fact-checking effectiveness independently of source-credibility effects
- Tests of two-step motivated reasoning model in the fact-checking domain, verifying that alignment (congruence between initial belief and fact-check verdict) is a critical moderator of correction effectiveness
Method
Online experiment hosted on Qualtrics with 859 participants (after exclusions from MTurk). Participants viewed 10 health-related news stories in random order, then rated perceived veracity and credibility of each story. After viewing, participants were randomly assigned to one of seven fact-checking source conditions (professional fact-checkers: FactCheck.org, PolitiFact; news outlets: Fox, USA Today, MSNBC; social media: Twitter; AI-based fact-checking; crowdsourcing) or control. All fact-checks were identical except for source, presented as Twitter-like posts.
Fact-checking effectiveness was measured as the reduction in belief in misinformation from pre- to post-exposure to the fact-check, controlling for initial story credibility rating. Perceived credibility of the fact-checker was measured using a six-item scale (credible, qualified, trustworthy, based on research, transparent, easy to understand). Alignment was derived from story veracity ratings: "aligned" if the fact-check confirmed the participant's initial belief (true/mostly true story; fact-check said true), "unaligned" if it contradicted initial belief (story false/mostly false; fact-check said false).
Cognitive reflection test (CRT; Pennycook & Rand, 2020) measured analytic thinking. Demographic covariates included age, education, work in medical field, political ideology, news use, and social media use.
Results
Main effects: A one-way ANCOVA on fact-checking effectiveness across five source types (controlling for demographics) found no significant overall differences—the omnibus test only approached significance ($F(6,840)=1.89, p=.08$). This null result suggests that across the sources tested, the type of fact-checking source had minimal main effect on effectiveness.
Perceived credibility mediation: Hierarchical regression showed perceived credibility of fact-checkers positively predicted fact-checking effectiveness ($b=0.178, p<.001, \Delta R^2=.022$), supporting Hypothesis 1. Mediation analysis confirmed that perceived credibility mediates the relationship between exposure to fact-checking messages from different sources and effectiveness in reducing belief (all 95% confidence intervals based on 5,000 bootstrap samples excluded zero for key contrasts). This supports Hypothesis 2 and extends the MAIN model to fact-checking.
Perceived credibility as moderator: Moderation analyses ($R^2=.17, F(10,842)=17.10, p<.001$) showed a significant interaction between alignment and perceived credibility of fact-checkers ($b=-0.23, p<.001$). When perceived credibility was high ($1 SD$ above mean), fact-checks were significantly more effective in reducing beliefs in misinformation among unaligned individuals ($b=-1.33, p<.001$) than aligned individuals ($b=-0.87, p<.001$). When perceived credibility was low ($1 SD$ below mean), fact-checkers were still more effective among unaligned individuals than aligned individuals ($b=-0.95, p<.001$ vs. $b=-0.73, p<.001$), but the difference was smaller. This partially supports Hypothesis 3: credibility does moderate the alignment effect, though analytic thinking was not a significant predictor of perceived fact-checker credibility ($b=-0.02, p>.05$).
Analytic thinking: Hierarchical regression showed analytic thinking (CRT score) significantly predicted fact-checking effectiveness after controlling for fact-checker source and demographics ($b=0.13, p<.001, \Delta R^2=.02$), supporting Hypothesis 5. Fact-checks were more effective in reducing belief in misinformation among people higher in analytic thinking. When analytic thinking was high ($1 SD$ above mean), fact-checks were more effective ($b=0.20, p<.001$), and when low ($1 SD$ below mean), effectiveness was reduced, though fact-checks were still more effective for unaligned than aligned individuals.
Source credibility perception tiers: Perceived credibility varied across sources. Professional fact-checkers (FactCheck.org, PolitiFact) and news outlets (USA Today, MSNBC) were perceived as more credible than AI and crowdsourcing. Twitter was perceived as the least credible source for fact-checking.
Notes
Strengths: First comprehensive empirical comparison of effectiveness across diverse fact-checking sources in a single study. Clear operationalization of perceived credibility as a mediating mechanism. Proper testing of moderation by alignment, addressing limitations of prior work that conflated alignment and effectiveness. Robust use of bootstrapped confidence intervals for mediation.
Limitations: Health topic stories only; generalizability to political or other domains unknown. Perceived story credibility pre-exposure may differ from actual susceptibility. Limited to U.S. MTurk sample with known political biases. CRT placement after fact-checking exposure may confound measurements. Order of measures (story credibility, fact-check credibility) could introduce priming effects. Null main effects for source type suggest future research should examine conditions under which source differences matter.
Implications: Finding that credibility mediates but source type doesn't directly predict effectiveness has practical implications: rather than assuming one fact-checking source is universally more effective, invest in increasing credibility perception of fact-checkers through transparency and institutional legitimacy. Suggests that newer fact-checking sources (AI, crowdsourcing) may suffer from lower effectiveness primarily due to credibility deficits that could be remediated through communication strategy.