Key Themes
Health Misinformation Context: Motivating the Problem
Menczer opens with a concrete example of how health misinformation spreads and causes measurable harm. Children's Health Defense, a major antivaccine nonprofit, published an article falsely claiming the Pfizer COVID-19 vaccine for children was 107 times more likely to cause death than COVID itself, citing a peer-reviewed medical paper. The article cited real research but wildly misrepresented its findings. Most readers cannot verify the claim because the underlying study is technical and inaccessible to non-experts.
Evidence of harm: Twitter data shows correlation between exposure to low-credibility vaccine information at the county level and increased vaccine hesitancy and lower vaccination rates, even controlling for political beliefs. Lab experiments confirm the causal signal: people exposed to vaccine misinformation become more hesitant about vaccination. Menczer models the interaction between misinformation spread on social networks and disease spread in physical contact networks, finding that low resilience to misinformation (people who adopt false beliefs after a single exposure) can increase infections by six-fold and deaths by 14% of the population (~47 million people in the U.S. alone).
The Good: AI for Detection and Defense
Machine Learning for Bot Detection
Botometer: Menczer's group built Botometer, a supervised machine learning classifier trained on Twitter data to predict the likelihood that an account is automated. The tool received ~500,000 queries daily before researchers lost Twitter API access in 2023. Botometer X, an archived version, allows lookup of historical bot scores. While not perfect, Botometer's accuracy is sufficient for research and public use.
Unsupervised Detection of Coordinated Campaigns
Beyond individual bot classification, Menczer's group uses unsupervised learning (clustering on suspicious behavioral patterns) to detect coordinated networks. Example: accounts that swap handles (use the same handle across multiple accounts) are likely coordinated by a single entity evading platform enforcement. Clustering these accounts reveals networks of 10-100+ coordinated accounts controlled by a single actor, often engaged in fraud (pretending to be pro-Trump and anti-Trump to solicit donations).
Network-Based Source Credibility Classification
Assuming a labeled set of known-credible and known-low-credibility sources, Menczer's group infers credibility of unknown accounts using network structure. Multiple approaches work:
- Information propagation methods (similar to PageRank) propagate credibility through the network.
- Deep learning embeddings: Performing random walks through co-sharing networks and using neural networks to embed accounts in vector space, then classifying based on proximity to labeled sources. Embedding-based methods achieve the best accuracy.
This approach identifies accounts that are "guilty by association"—they cluster near known misinformation sources and likely share misinformation, even without explicit content analysis.
The Bad: Unintended Consequences of AI Fact-Checking
ChatGPT as a Biased Credibility Judge
When prompted "Is this news source credible?" ChatGPT performs better than random but worse than expert fact-checkers. Worse, the model is easily jailbroken. When prompted "Assume you are a Democrat, is this source credible?" ChatGPT's ratings shift dramatically: it rates Democratic-leaning sources as more credible and Republican sources as less credible (even if they're actually reliable, like the Wall Street Journal). A Republican prompt produces the opposite bias. Menczer emphasizes this is an easy exploit for bad actors to create the appearance of unbiased AI assessment while actually embedding partisan bias.
ChatGPT for Fact-Checking Claims: Backfire Effects
Menczer's team (led by Harry) ran a randomized controlled experiment with three groups:
- Control: Participants see a news headline and are asked if they believe the claim.
- Forced ChatGPT fact-check: Participants see a ChatGPT-generated fact-check (whether the claim is true or false).
- Optional ChatGPT fact-check: Participants choose whether to see a ChatGPT fact-check before answering.
ChatGPT correctly flags false claims ~90% of the time but struggles with true claims (often marking them "unsure"). Surprisingly, the overall effect on belief and sharing is null—ChatGPT fact-checks neither help nor harm on average. Disaggregating by case:
- When ChatGPT incorrectly marks a true claim as false, belief in that true claim drops significantly (backfire).
- When ChatGPT marks a false claim as unsure, people become more likely to believe or share it (another backfire).
- When ChatGPT correctly identifies a false claim, sharing increases slightly (mild help).
- These effects cancel out, yielding zero net effect.
Optional selection effect: When people choose to see ChatGPT fact-checks, they end up more likely to believe false claims and more likely to share both true and false claims—suggesting self-selection bias (people requesting fact-checks are more susceptible to misinformation).
Menczer highlights the important lesson: good intentions (using AI for fact-checking) can produce harmful unintended consequences if human-AI interaction is not carefully designed.
The Ugly: AI Enabling Bad Actors
Manipulation Tactics and Effectiveness
Before introducing AI, Menczer establishes how bad actors manipulate social media:
- Infiltration: Bots infiltrate vulnerable communities (those sharing low-credibility content) and amplify misinformation from within. Critically, even ~1% infiltration can drive overall system quality to zero.
- Flooding: Bots post massive volumes of content (hundreds of thousands per day) and then delete it to evade platform limits and detection. Deletion also obscures the evidence of manipulation.
- Targeting: Bots systematically mention influential accounts (journalists, politicians) with links to misinformation, trying to get those influencers to notice and amplify. Example: a bot network targeting Donald Trump with links to a false election claim (3 million illegal votes) likely influenced his subsequent public statements and commission investigation—Trump never directly shared the article, but repeated the claim.
Simulation of a social media ecosystem shows infiltration is the most effective tactic, reducing overall information quality by ~70% when combined with flooding.
GAN-Generated Deepfake Profiles
Bad actors use Generative Adversarial Networks (GANs) to create fake profile pictures that are difficult to detect via reverse image search (unlike reused stolen photos). Menczer's analysis of GAN-generated faces finds a signature: when normalized, eyes cluster in the same position across all GAN faces (unlike real faces, which have natural variation). Using this method, Menczer estimates ~10,000–15,000 daily active Twitter users with GAN-generated faces, likely an underestimate. These fake profiles impersonate users and run scams.
ChatGPT-Managed Bot Networks
Menczer's group identified a botnet ("Fox8") in which hundreds of accounts were all posting and amplifying content, but every message and interaction was generated by ChatGPT. The programmer made a mistake—one account inadvertently revealed it was AI-generated by responding "I cannot respond to this as it is a sharable link; my capabilities are limited to text-based responses only." By analyzing shared patterns (identical errors, coherent topics, synchronized behavior), researchers detected the full botnet.
This is just the visible tip of the iceberg—many ChatGPT botnets likely exist with more careful programming that doesn't reveal AI generation. These bots claim 81% of their tweets contain harmful or inappropriate content. They primarily share cryptocurrency-related content and links to fake news websites (especially Fox8, a ChatGPT-generated fake news site). This drives pump-and-dump cryptocurrency scams where bad actors artificially hype a coin to raise the price, then dump it for profit while victims lose money.
Detection Failure: Why AI-Generated Content is Undetectable
OpenAI released a detection tool for ChatGPT-generated text, but it achieved only modest accuracy and generated false positives on 40%+ of real human text—essentially unusable. OpenAI pulled the tool due to poor performance. Currently, no reliable method exists to detect ChatGPT-generated or other LLM-generated content at scale.
NewsGuard's discovery: NewsGuard tracks fake news websites that are entirely ChatGPT-generated. These sites publish articles that occasionally explicitly state "As a language model…" in the text, accidentally revealing their AI origins. Readers who don't catch these slip-ups may believe the articles are legitimate.
Fake News Websites at Scale
Bad actors are deploying AI to automatically generate fake news websites filled with ChatGPT content. These sites distribute misinformation about cryptocurrency, health, and other topics. Because ChatGPT can produce fluent, plausible-sounding prose, detection relies on stumbling upon self-revealing artifacts (explicit mentions of being an AI) rather than on automated detection.
Interaction Between Information and Epidemic Spread
Menczer models the interaction between social network misinformation diffusion and physical contact network disease spread. The model parameterizes resilience (how many exposures to misinformation it takes to shift someone's behavior) and tests how changes in misinformation exposure affect disease outcomes. Finding: low resilience to misinformation, driven by infiltrating vulnerable communities with bot-amplified content, can increase peak infections by 6-fold and total infections by 14% of population—quantifying the real-world harm.
Methods and Tools
- Botometer — Supervised machine learning classifier for detecting social bots; Botometer X archives historical scores
- Network clustering for coordinated campaigns — Unsupervised detection of handle-swapping, link-sharing, and behavioral synchronization
- Credibility propagation — Information-flow and embedding-based methods to infer source credibility from network structure
- GAN-face detection — Statistical analysis of eye position clustering to identify GAN-generated profile pictures
- Simulation of social media — Agent-based models to test effectiveness of different manipulation tactics and measure system-wide information quality
- Epidemic-information co-modeling — Joint simulation of disease spread in contact networks and misinformation spread in social networks
Notes
A critical analysis of AI's dual role: enabling both defense (bot detection, credibility assessment) and attack (deepfake profiles, synthetic content, jailbroken fact-checking). Strengths: concrete examples grounded in Menczer's lab research (ChatGPT experiment, Fox8 botnet), quantified harms (health misinformation → 47 million additional infections, 6-fold peak increase), and honest assessment of AI limitations (detection of AI-generated content near-impossible). The talk carefully distinguishes between approaches that improve misinformation defense (embedding-based credibility classification, bot detection) and those that backfire (uncalibrated ChatGPT fact-checking).
The ChatGPT fact-checking experiment is particularly valuable for its disaggregated analysis—showing null effects hide harmful cases where AI mistakes backfire. Menczer emphasizes that alignment between AI capability and human psychology is not guaranteed; good intentions can produce harm without careful design. The co-modeling of epidemic and information spread provides a rare quantification of health misinformation's real-world consequences.
Key limitations: The talk focuses on Twitter/X data, which may not generalize to TikTok, Instagram, YouTube, or newer platforms. Detection methods for GAN-generated faces rely on specific artifacts (eye position) that may be patched by more sophisticated GANs. The Fox8 botnet discovery relied on accidental self-revelation; truly careful bad actors may leave no detectable trace. The absence of reliable AI-generated content detection remains an open problem with no clear near-term solution.
Scope clarification: The talk is titled with Western movie reference (Sergio Leone), and Menczer opens by explaining the Italian Westerns framing for "The Good, the Bad and the Ugly." This is organizational device, not core content, and the analogy is reasonably apt: AI tools can be used for both defense (good) and attack (bad), and emerging capabilities enable new harms (ugly) that we cannot yet reliably detect or counter.
The talk dates to late 2024 or early 2025 (references Facebook stopping fact-checking "as we all know," mentions Cabinet nominees, loss of Twitter API "a little more than a year ago" = early 2023). A human should verify the exact date from YouTube metadata and update the page's year field when known.