Contributions
- Compares three operationalizations of source ratings (expert-assessed, crowd-assessed-articles, crowd-assessed-sources) through the lens of reputation theory (functional, social, affective dimensions)
- Demonstrates that rating mechanism matters: expert ratings and user article ratings have stronger effects on believability than user source ratings
- Shows an asymmetry: low ratings have stronger effects than high ratings; users disconfirm disreputable sources but are less swayed by high ratings of reputable sources
- Identifies confirmation bias as a key moderator: article believability strongly predicts engagement behavior (read, like, comment, share)
- Revises hypotheses based on evidence showing that rating mechanism effects are strongest for low-rated sources, not uniformly
Method
Two online experiments with participants from a Qualtrics panel (US adults). Study 1 used 590 participants; Study 2 used 299. Both followed a repeated-measures design where participants viewed 8 news headlines (4 designed for left-leaning readers, 4 for right-leaning readers), rated their believability and importance, and indicated likelihood to read, like, comment, or share. Treatments: control (no ratings), expert rating (displayed with 5-star scale from expert panel), user article rating (aggregate of user ratings on individual articles), user source rating (aggregate of user ratings on the source itself). Headlines included fabricated sources (HotNews.com, NewsHeadlines.com, etc.) to avoid confounding with real source reputation. Study 2 added an opinion survey and randomized control headlines with rating headlines to test for spillover effects.
Results
- All three rating mechanisms influenced perceived believability.
- For high-rated sources, all three mechanisms had similar small or nonsignificant effects; high ratings were expected and did not surprise users.
- For low-rated sources, all mechanisms had significant negative effects: expert ratings and user article ratings were significantly stronger than user source ratings (Chi-Squared = 15.39, p < 0.001 for expert vs. source).
- Low ratings triggered deeper consideration of the rating mechanism; disconfirmation (unexpected low ratings) drew attention and prompted skepticism.
- Confirmation bias was prominent: users were more likely to believe and engage with articles matching their pre-existing opinions. Believability predicted all engagement behaviors (read, like, comment, share; all p < 0.001).
- No direct effects of rating mechanism on actions: rating mechanisms influenced behavior indirectly through changes in believability.
- Post-hoc test confirmed that expert ratings were perceived as more cognitive (functional) than user ratings, which were more emotional.
Notes
Empirical work on a practical mechanism (source ratings) for combating fake news spread. The key insight is the asymmetry: low ratings drive behavior change more than high ratings, and the mechanism matters—expert and user-article ratings outperform user-source ratings. Confirmation bias remains the dominant force even with rating cues, suggesting that source credibility signals are moderated by users' pre-existing beliefs. The paper is careful about ecological validity, using political headlines that align with participants' actual beliefs. One limitation: the study uses fabricated sources, so results may not generalize to sources with existing reputations (both positive and negative). The revised hypothesis (H6) about rating mechanism mattering only for low ratings is interesting and suggests platform interventions should focus on highlighting low-quality sources rather than high-quality ones, as users already expect and trust high-quality sources.