Skip to content
Detection methodsFact-checking & verification fact checkerssource credibilityeffectiveness

Checking the Fact-Checkers: The Role of Source Type, Perceived Credibility, and Individual Differences in Fact-Checking Effectiveness

Communication Research · 2025

In brief

An online experiment (N=859) testing fact-checking effectiveness across five source types: professional fact-checkers, mainstream news outlets, social media platforms, artificial intelligence, and crowdsourcing. Perceived credibility of fact-checkers mediates the relationship between exposure and effectiveness in reducing belief in misinformation; alignment with the fact-check verdict moderates this effect. Analytic thinking predicts fact-checking effectiveness.

Summary figure for Checking the Fact-Checkers: The Role of Source Type, Perceived Credibility, and Individual Differences in Fact-Checking Effectiveness

Contributions

  • First empirical comparison of effectiveness across the full array of contemporary fact-checking source types (professional fact-checkers, news outlets, social media, AI, crowdsourcing)
  • Empirical validation that perceived credibility of fact-checkers mediates fact-checking effectiveness, extending the MAIN model (Sundar, 2008) to the fact-checking context
  • Demonstration that perceived credibility plays a moderating role: when credibility is high, fact-checks are more effective for unaligned individuals (those who initially disbelieved the false claim), but credibility differences matter less for aligned individuals
  • Evidence that analytic thinking predicts fact-checking effectiveness independently of source-credibility effects
  • Tests of two-step motivated reasoning model in the fact-checking domain, verifying that alignment (congruence between initial belief and fact-check verdict) is a critical moderator of correction effectiveness

Method

Online experiment hosted on Qualtrics with 859 participants (after exclusions from MTurk). Participants viewed 10 health-related news stories in random order, then rated perceived veracity and credibility of each story. After viewing, participants were randomly assigned to one of seven fact-checking source conditions (professional fact-checkers: FactCheck.org, PolitiFact; news outlets: Fox, USA Today, MSNBC; social media: Twitter; AI-based fact-checking; crowdsourcing) or control. All fact-checks were identical except for source, presented as Twitter-like posts.

Fact-checking effectiveness was measured as the reduction in belief in misinformation from pre- to post-exposure to the fact-check, controlling for initial story credibility rating. Perceived credibility of the fact-checker was measured using a six-item scale (credible, qualified, trustworthy, based on research, transparent, easy to understand). Alignment was derived from story veracity ratings: "aligned" if the fact-check confirmed the participant's initial belief (true/mostly true story; fact-check said true), "unaligned" if it contradicted initial belief (story false/mostly false; fact-check said false).

Cognitive reflection test (CRT; Pennycook & Rand, 2020) measured analytic thinking. Demographic covariates included age, education, work in medical field, political ideology, news use, and social media use.

Results

Main effects: A one-way ANCOVA on fact-checking effectiveness across five source types (controlling for demographics) found no significant overall differences—the omnibus test only approached significance ($F(6,840)=1.89, p=.08$). This null result suggests that across the sources tested, the type of fact-checking source had minimal main effect on effectiveness.

Perceived credibility mediation: Hierarchical regression showed perceived credibility of fact-checkers positively predicted fact-checking effectiveness ($b=0.178, p<.001, \Delta R^2=.022$), supporting Hypothesis 1. Mediation analysis confirmed that perceived credibility mediates the relationship between exposure to fact-checking messages from different sources and effectiveness in reducing belief (all 95% confidence intervals based on 5,000 bootstrap samples excluded zero for key contrasts). This supports Hypothesis 2 and extends the MAIN model to fact-checking.

Perceived credibility as moderator: Moderation analyses ($R^2=.17, F(10,842)=17.10, p<.001$) showed a significant interaction between alignment and perceived credibility of fact-checkers ($b=-0.23, p<.001$). When perceived credibility was high ($1 SD$ above mean), fact-checks were significantly more effective in reducing beliefs in misinformation among unaligned individuals ($b=-1.33, p<.001$) than aligned individuals ($b=-0.87, p<.001$). When perceived credibility was low ($1 SD$ below mean), fact-checkers were still more effective among unaligned individuals than aligned individuals ($b=-0.95, p<.001$ vs. $b=-0.73, p<.001$), but the difference was smaller. This partially supports Hypothesis 3: credibility does moderate the alignment effect, though analytic thinking was not a significant predictor of perceived fact-checker credibility ($b=-0.02, p>.05$).

Analytic thinking: Hierarchical regression showed analytic thinking (CRT score) significantly predicted fact-checking effectiveness after controlling for fact-checker source and demographics ($b=0.13, p<.001, \Delta R^2=.02$), supporting Hypothesis 5. Fact-checks were more effective in reducing belief in misinformation among people higher in analytic thinking. When analytic thinking was high ($1 SD$ above mean), fact-checks were more effective ($b=0.20, p<.001$), and when low ($1 SD$ below mean), effectiveness was reduced, though fact-checks were still more effective for unaligned than aligned individuals.

Source credibility perception tiers: Perceived credibility varied across sources. Professional fact-checkers (FactCheck.org, PolitiFact) and news outlets (USA Today, MSNBC) were perceived as more credible than AI and crowdsourcing. Twitter was perceived as the least credible source for fact-checking.

Notes

Strengths: First comprehensive empirical comparison of effectiveness across diverse fact-checking sources in a single study. Clear operationalization of perceived credibility as a mediating mechanism. Proper testing of moderation by alignment, addressing limitations of prior work that conflated alignment and effectiveness. Robust use of bootstrapped confidence intervals for mediation.

Limitations: Health topic stories only; generalizability to political or other domains unknown. Perceived story credibility pre-exposure may differ from actual susceptibility. Limited to U.S. MTurk sample with known political biases. CRT placement after fact-checking exposure may confound measurements. Order of measures (story credibility, fact-check credibility) could introduce priming effects. Null main effects for source type suggest future research should examine conditions under which source differences matter.

Implications: Finding that credibility mediates but source type doesn't directly predict effectiveness has practical implications: rather than assuming one fact-checking source is universally more effective, invest in increasing credibility perception of fact-checkers through transparency and institutional legitimacy. Suggests that newer fact-checking sources (AI, crowdsourcing) may suffer from lower effectiveness primarily due to credibility deficits that could be remediated through communication strategy.

How it connects

Similar papers

Summarized from the full text. Added 2026-11-20 from a submission by reza. Every statement on this page traces back to the source.