PTC-SemEval20

Description

PTC-SemEval20 is a corpus of news articles annotated with propaganda techniques, created for the SemEval-2020 Task 11 shared task. It was curated from 13 propaganda outlets and 36 non-propaganda media sources over the period mid-2017 to early 2019, with articles selected via word n-gram matching and labeled by professional annotators.

Statistics

Inter-annotator agreement (γ): 0.6 (moderate)

Annotation schema

Two subtasks:

Subtask SI (Span Identification): Binary sequence tagging identifying character spans containing at least one propaganda technique. Overlapping annotations are merged before evaluation.

Subtask TC (Technique Classification): Multi-class classification assigning one of 14 propaganda techniques to a given snippet in document context. 1.8% of annotations are multi-label (same span, different techniques); treated as single-label by merging techniques.

14 propaganda techniques:

1. Loaded language — Emotionally charged words/phrases with strong connotations (positive or negative) designed to influence audience
2. Name calling or labeling — Labeling the target of propaganda with emotionally negative terms designed to evoke strong negative feeling
3. Repetition — Repeating the same message over and over, so audience eventually accepts it
4. Exaggeration or minimization — Representing something in an excessive manner (exaggeration) or making something seem insignificant (minimization)
5. Doubt — Questioning the credibility of someone or something
6. Appeal to fear/prejudice — Building support by instilling anxiety or panic in the population
7. Flag-waving — Playing on strong national feeling (or any group feeling) to justify or promote an action or idea
8. Causal oversimplification — Assuming a single cause or reason when there are actually multiple causes or reasons (scapegoating variant: transferring blame to one person or group)
9. Slogans — A brief and striking phrase that may include labeling and stereotyping
10. Appeal to authority — Stating that a claim is true because a valid authority supports it
11. Black-and-white fallacy, dictatorship — Presenting two alternative options as the only possibilities (when more exist); dictatorship variant: telling the audience exactly what actions to take
12. Thought-terminating cliché — Words or phrases that discourage critical thought and meaningful discussion
13. Whataboutism, straw man, red herring — Three related techniques merged for rare-class sparsity: (i) discrediting opponent's position via hypocrisy charge; (ii) refuting a distorted version of opponent's position; (iii) introducing irrelevant material
14. Bandwagon, reductio ad hitlerum — (i) Persuading audience because "everyone is doing it"; (ii) association with hated groups to discredit

Class distribution

Loaded Language is the most frequent (2× baseline); Name Calling/Labeling is second most frequent. Rare techniques include Whataboutism/Straw Man/Red Herring and Bandwagon/Reductio ad Hitlerum (<0.25% each).

Labeling methodology

Six professional annotators from A Data Pro were trained and instructed to mark text spans in free text. Annotation occurred in two phases: (i) initial independent annotation by two annotators per article; (ii) consolidation phase where any discrepancies were reviewed by a third expert with consensus reached. Inter-annotator agreement computed via γ (Mathet et al., 2015) averaged 0.6.

Benchmark results

Top performing systems on test set span identification (SI):

• Hitachi: F₁ 51.55
• ApplicaAI: F₁ 49.15
• Aschern: F₁ 49.10

Top performing systems on technique classification (TC):

• ApplicaAI: F₁ 49.15
• Aschern: F₁ 49.10
• Hitachi: F₁ 51.55 (on development)

Ensemble combinations improved SI F₁: top-7 union reached 62% F₁, intersection-based reached 66.95% precision (at cost of recall ~47.67%).

Technique-level performance varies widely: frequent techniques (Loaded Language, Name Calling) achieve ~70% F₁; rare techniques (Whataboutism/Straw Man/Red Herring, Bandwagon/Reductio ad Hitlerum) achieve only ~30% F₁. Majority voting required 13+ systems to reach peak performance on difficult classes.

Intended use

• Fine-grained propaganda technique detection research
• Shared task benchmarking (SemEval-2020 Task 11)
• Training and evaluation of sequence tagging models for propaganda localization
• Technique classification with imbalanced multiclass learning
• Study of data augmentation and ensemble methods for rare-class detection
• Transfer learning and domain adaptation for propaganda detection

Limitations

Data sparsity for rare techniques: Six of 14 techniques remain challenging across all systems even in top-performing systems. Class imbalance (class 0 alone accounts for ~80% of tokens) exacerbates generalization difficulty.

Temporal and source specificity: Data limited to mid-2017 through early 2019 and drawn from Western (primarily US/UK/Europe) propaganda and mainstream media outlets. Newer tactics (LLM-generated content, deepfakes) not represented. Generalization to non-English or non-news domains unknown.

Inter-annotator agreement: γ = 0.6 indicates moderate disagreement on which text spans constitute propaganda. Some ambiguity on whether broader context or minimal span should be labeled.

Test-set distribution shift: Several systems performing well on development set showed overfitting, with test-set performance 15–25 F₁ points lower (particularly on rare techniques). Suggests mismatch between training and test distributions.

Connections

• propaganda-detection — the broader topic; this dataset is the primary benchmark for task SI and TC
• propaganda-techniques — detailed descriptions of the 14 techniques with examples
• SemEval-2020 Task 11 — the shared task that introduced this corpus
• NLP4IF-2019 Task — predecessor shared task with 18 techniques on smaller corpus