Contributions
- Curated the PTC-SemEval20 corpus of 536 annotated news articles (13 propaganda and 36 non-propaganda media outlets, drawn from mid-2017 to early 2019)
- Defined and validated a curated inventory of 14 propaganda techniques with detailed definitions and examples
- Created a two-stage evaluation framework: span identification (binary sequence tagging) and technique classification (multi-class multi-label classification) with character-level F₁ scores
- Organized a shared task attracting 250 registered teams with 44 official submissions (66 total for span identification, 32 for technique classification across two phases)
- Comprehensive empirical analysis showing that top systems rely on Transformers (BERT, RoBERTa, XLNet, GPT-2) combined with LSTMs/CRFs and feature engineering; identified challenge classes and data sparsity issues
Method
The task defines propaganda as deliberate use of rhetorical and psychological techniques to influence opinions or actions toward predetermined ends. The paper identifies 14 propaganda techniques studied in the literature:
- Loaded language — using emotionally charged words/phrases to influence audience
- Name calling/labeling — labeling targets with emotionally negative terms
- Repetition — repeating a message to achieve acceptance
- Exaggeration/minimization — overstating or understating something
- Doubt — questioning credibility of someone/something
- Appeal to fear/prejudice — instilling anxiety/panic toward alternatives
- Flag-waving — appealing to strong national/group feeling
- Causal oversimplification — assuming single cause when multiple exist (includes scapegoating)
- Slogans — brief striking phrases with labeling/stereotyping
- Appeal to authority — stating a claim is true because an authority/expert supports it
- Black-and-white fallacy/dictatorship — presenting only two options when more exist
- Thought-terminating cliché — short generic sentences discouraging critical discussion
- Whataboutism, straw man, red herring — diverting attention to irrelevant topics
- Bandwagon, reductio ad hitlerum — either "everyone is doing this" or "association with hated groups"
Span Identification (SI): Given a plain-text news article, identify character spans containing at least one propaganda technique. Evaluation uses precision/recall/F₁ on matched text spans, merging overlapping annotations before scoring.
Technique Classification (TC): Given a propaganda snippet in document context, identify which of the 14 techniques it uses. Since 1.8% of annotations are multi-label (identical spans with different techniques), treated as single-label multi-class; evaluation merges multi-label spans, then scores micro-average F₁.
The shared task ran in two phases: Phase 1 provided training/development data with live leaderboard; Phase 2 released test set with limited submission window.
Results
Span Identification: Top-3 systems achieved F₁ ≈51–56% on test set (Hitachi: 51.55, ApplicaAI: 49.15, aschern: 49.10). All relied on Transformers (BERT, RoBERTa, XLNet, XLM-RoBERTa) + LSTM/CRF with engineered features (named entities, sentiment, subjectivity cues). Ensemble combinations improved performance: union of top-7 systems reached F₁ ≈62%, but intersection-based filtering raised precision to 66.95% at cost of recall (47.67%).
Technique Classification: Top-3 systems achieved F₁ ≈49–52% on test set (ApplicaAI: 49.15, aschern: 49.10, Hitachi: 51.55 on development; test-set leader scores lower due to different distribution). Most frequent techniques (Loaded Language, Name Calling/Labeling: ~twice baseline frequency) achieved ~70% F₁; rare techniques (Whataboutism/Straw Man/Red Herring, Bandwagon/Reductio ad Hitlerum) reached only ~30% F₁ even in top systems. Majority voting across top-20 systems yielded F₁ ≈63.63, with challenge classes (Doubt, Causal Oversimplification) requiring 13+ systems to peak at ~17–18% F₁.
Data sparsity for rare classes and distribution shift between development and test sets caused overfitting in several systems; techniques sensitive to context benefited from document-level features.
Notes
Strengths: Rare systematic large-scale effort to operationalize fine-grained propaganda detection. Curated inventory of 14 techniques grounds task in established propaganda literature (Miller, Weston, Hobbs & McGee, etc.). Two-stage task design (localize then classify) separates orthogonal challenges. Public leaderboard and detailed system descriptions enable reproducibility and ablation studies.
Limitations: Data drawn only from 2017–2019; language and propaganda tactics may have shifted (e.g., LLM-generated content). Dataset limited to English and news domain. Inter-annotator agreement (γ ≈ 0.6) moderate; multi-label annotation edge cases resolved via single-label merge, losing granularity. Rare classes severely under-represented; all systems struggled with 6 of 14 techniques. No cross-lingual or cross-domain evaluation.
Open questions: How do approaches generalize to social media, user-generated content, or non-English languages? Can context windows beyond the immediate snippet improve technique discrimination? Do LLM-based approaches (post-2020) retain the empirical findings or require retrained baselines?