Leveraging Large Language Models to Detect Influence Campaigns on Social Media

Key findings

• Content-based classification using LLMs consistently demonstrates strong accuracy across various influence campaigns, even more recent operations.
• Metadata-based approaches also achieve promising results, sometimes outperforming content-based methods.
• LLM-based methods show better generalization than traditional linguistic-cue baselines, which degrade significantly on recent, more sophisticated campaigns.
• The paper uses Llama 2 with zero-shot and few-shot prompting via Microsoft Guidance, plus fine-tuning with Llama Factory.
• Evaluated against established datasets: Twitter Information Operations, Russian Twitter Trolls, and campaigns from Egypt, Ecuador, and Venezuela.

Methodology

The authors propose four classification models based on:

1. Content analysis — using LLM features extracted from tweets
2. User interaction — analyzing user engagement patterns
3. User centrality — leveraging network structure and graph properties
4. User metadata — demographics, account properties, and auxiliary signals

Both tweet-level (binary: in-campaign or organic) and user-level classification (drivers of campaigns vs. organic users) tasks are addressed.

Notes

This is a methodological talk presenting work likely submitted to The Web Conference. The emphasis on cross-campaign generalization is notable — many detection systems degrade when tested on new operations, but these LLM-based approaches maintain performance across geographically diverse campaigns. The use of public influence-operation datasets (Twitter's official disclosures, the Russian troll datasets) makes this reproducible and grounded in real-world threat models.