Skip to content
Cognitive reflection correlates with behavior on Twitter

Cognitive reflection correlates with behavior on Twitter

Authors: Mohsen Mosleh, Gordon Pennycook, Antonio A. Arechar, David G. Rand Venue: Nature Communications, 2021 — DOI

TL;DR

High scorers on the Cognitive Reflection Test (CRT)—a measure of analytical thinking—are more discerning social media users: they follow fewer accounts, share news from more reliable sources, and engage more with substantive topics like politics. The study identifies "cognitive echo chambers" where low-CRT users preferentially follow accounts avoided by high-CRT users, suggesting that information filtering on social media extends beyond partisan polarization.

Contributions

  • Provides field evidence that cognitive reflection predicts naturally occurring social media behavior (not just laboratory performance or self-reported behavior)
  • Demonstrates CRT predicts news source quality: high-CRT users preferentially share content from outlets trusted by professional fact-checkers (BBC, NYT) and avoid low-trust sources (Daily Mail, Breitbart)
  • Documents asymmetric "cognitive echo chambers": ~65% of followed accounts cluster among low-CRT users, while ~35% are followed by both high and low-CRT users
  • Shows higher analytical thinking correlates with engagement on substantive topics (politics) and lower engagement with scams and "get rich quick" schemes
  • Validates the "reflectionist" perspective in cognitive science: reflective thinking matters for real-world judgment outside artificial laboratory settings

Method

Hybrid lab-field design linking survey data to actual Twitter behavior. Recruited N=1,901 participants via Prolific (mostly UK and US based) who completed the seven-item Cognitive Reflection Test and provided Twitter usernames. Used Twitter API to collect public profile data, followed accounts, and 3,200 most recent tweets per user (retrieved August 2018 and April 2020). Analyzed relationships between CRT score and: (1) profile characteristics (followers, tweets, etc.), (2) accounts followed via co-follower network analysis, (3) news sources shared via fact-checker trust ratings, and (4) tweet content via topic modeling (Structural Topic Modeling) and linguistic analysis (Linguistic Inquiry Word Count, LIWC).

Results

Profile characteristics: Higher CRT users followed significantly fewer accounts (incidence rate ratio = 0.867, p=0.001), but no significant difference in follower count, tweet count, or favorites.

Accounts followed (network analysis): Community detection on the co-follower network revealed two distinct clusters. Cluster 1 (35% of accounts) had higher mean follower CRT (0.515); Cluster 2 (65% of accounts) had lower mean follower CRT (0.419; Cohen's d=1.66). Average follower CRT is a highly significant predictor of cluster membership (OR=0.545, p=0.004)—a one standard deviation decrease in followers' CRT score increases odds of cluster 2 membership by 83.5%. Top accounts in Cluster 1: barackobama, stephenfry, bbcbreaking, nasa. Top accounts in Cluster 2: aldiuk, poundland, argos_online (UK retail/commerce accounts).

News sources: Users with higher CRT were more likely to tweet links to news sites (OR=1.135, p=0.011). Among those who did, CRT positively correlated with trustworthiness of shared sources (β=0.078, p=0.019). High-CRT users more likely to retweet BBC (OR=1.232, p<0.001) and less likely Daily Mail (OR=0.787, p<0.001).

Topic modeling: Two topics consistently correlated with CRT: politics positively correlated (keywords: people, vote, trump, brexit) and "get rich quick" schemes negatively correlated (keywords: enter, giveaway, prize, win).

Linguistic analysis (LIWC): Higher CRT associated with more insight words (OR=1.138, p<0.001) and inhibition words (OR=1.133, p<0.001); more negative emotion words (OR=1.124, p<0.001); more morality words (OR=1.078, p<0.001); and more political words (OR=1.167, p=0.006). No significant difference in positive emotion words.

Connections

  • Extends Pennycook & Rand's work on laziness and bias by validating lab findings of CRT and fake news susceptibility in field data
  • Supports dual-process theory and the "reflectionist" perspective that analytic thinking meaningfully influences everyday judgment
  • Related to misinformation detection literature via news source discernment
  • Relates to Echo Chambers through identification of cognitive (rather than purely partisan) clustering
  • Cited by subsequent work on cognitive factors in misinformation and social media behavior

Notes

Strengths: The field validation is substantial—moving beyond lab experiments and self-report surveys to actual behavior. The network analysis revealing cognitive echo chambers is novel. Large sample size (N=1,901) with robust statistical controls. Data-driven identification of which accounts differ by CRT (rather than assuming political axes).

Limitations: Convenience sample from Prolific; not representative of general Twitter users or global platforms. CRT may be a proxy for education or SES rather than pure analytical thinking. Dictionary-based language analysis (LIWC) is coarse. Temporal confound from Twitter API limits (3,200 most recent tweets; more active users have older tweets); addressed with month fixed effects but imperfect. Self-selection into survey experiment may drive results. The asymmetry of cognitive echo chambers (Cluster 2 much larger) is intriguing but causal mechanism unclear—does low CRT cause following of Cluster 2 accounts, or do Cluster 2 accounts attract people with low CRT?

Follow-ups: Authors suggest training machine learning models to estimate CRT from social media alone. Generalization to other platforms (Facebook, LinkedIn, non-Western social media). Experimental tests of link formation and reciprocity in follow networks. Broader investigation of cognitive echo chambers outside social media.