Fine-Grained Analysis of Propaganda in News Articles

Contributions

• Fine-grained annotation scheme: Operationalization of 18 propaganda techniques with fragment-level (span + label) annotations, moving beyond document- or sentence-level propaganda classification.
• Manually annotated corpus: 451 news articles (13 propagandistic, 36 non-propagandistic outlets) with 7,485 technique instances across 21,230 sentences (35.2% of sentences contain propaganda).
• Novel evaluation metrics: Custom precision, recall, and F1 measures adapted to sequence labeling that account for partial span overlaps at the character level, inspired by named entity recognition evaluation (CoNLL'02, CoNLL'03) and intrinsic plagiarism detection.
• Multi-granularity neural network: Architecture with gated connections between sentence-level and fragment-level classification layers, allowing fragment-level decisions to incorporate sentence-level predictions as additional information.
• Comprehensive baselines: BERT, BERT-Joint (multi-task), BERT-Granularity (concatenated layers), and two variants of the proposed multi-granularity network (ReLU and Sigmoid gates).

Propaganda Techniques (18 total)

The corpus identifies these 18 propaganda techniques derived from classical rhetorical and psychological literature:

1. Loaded language (2,547 instances) — emotionally charged words with strong emotional implications
2. Name calling or labeling (1,294) — dehumanizing or pejorative labels applied to individuals/groups
3. Repetition (767) — repeating the same message to achieve acceptance
4. Exaggeration or minimization (571) — representing magnitude or importance as greater or lesser than fact
5. Doubt (562) — questioning credibility without supporting evidence
6. Appeal to fear/prejudice (367) — instilling anxiety toward an alternative to persuade audience
7. Flag-waving (330) — appeals to nationalism or group identity
8. Causal oversimplification (233) — assuming single cause for complex issues; scapegoating
9. Slogans (172) — brief, striking phrases; tend to act as emotional appeals with stereotyping
10. Appeal to authority (169) — claiming statement is true because authority/expert supports it
11. Black-and-white fallacy, dictatorship (134) — presenting only two alternative options when more exist
12. Thought-terminating cliché (95) — phrases that discourage critical thought and meaningful discussion
13. Whataboutism (76) — discrediting opponent by charging hypocrisy without directly refuting argument
14. Reductio ad Hitlerum (66) — associating idea with hated groups/people
15. Red herring (48) — introducing irrelevant material to divert attention
16. Bandwagon (17) — persuading audience to join because "everyone else is doing it"
17. Obfuscation, intentional vagueness, confusion (17) — using deliberately unclear language
18. Straw man (15) — refuting distorted version of opponent's position

Data and Annotation

Article selection: 451 news articles retrieved from 48 news outlets, classified as propagandistic (13 outlets from Media Bias Fact Check) or non-propagandistic (36 outlets). Average article length: 47.1 lines, 973.2 words, 5,942 characters. Propagandistic articles tend to be longer.

Annotation process: Six professional annotators (from A Data Pro) manually annotated articles with fragment-level propaganda technique labels. Inter-annotator agreement measured using Krippendorff's α (on both span identification and labeling). Agreement relatively low in preliminary stage (γ = 0.30–0.34 for span, γ_al = 0.24–0.28 for labeling), but improved substantially with consolidation (γ up to 0.74–0.76).

Dataset split: Training + development + test containing 293, 57, 101 articles and 14,857, 2,108, 4,265 sentences respectively.

Class distribution: Highly imbalanced—loaded language and name calling dominate (~53% of instances), while bandwagon, straw man, and obfuscation rare (<0.25%).

Method

Tasks

Fragment-Level Classification (FLC): Identify minimal text spans where propaganda techniques appear and assign one of 18 technique labels. Formulated as sequence labeling with variable-length spans (avg. 47.1 characters).

Sentence-Level Classification (SLC): Binary classification determining whether a sentence contains at least one propagandist fragment.

Evaluation Measures

Custom metrics handle (i) multiple possibly overlapping fragments in text and (ii) variable-length spans. Partial credit given for inexact matches at character level:

$$C(s, t, h) = \frac{|s \cap t|}{h} \delta(l(s), l(t))$$

where $h$ is a normalizing factor and $\delta(a, b) = 1$ if $a = b$, else 0.

Precision and recall aggregate over predicted/gold fragments with partial overlap credit, combined into harmonic mean F1.

Models

Baselines:

• BERT: Add linear layer on top; fine-tune for FLC (19-way classification: 18 techniques + null class).
• BERT-Joint: Share layers between FLC and SLC tasks.
• BERT-Granularity: Concatenate sentence-level ($L_{sl}$) and fragment-level ($L_fl$) layers, add 19-D classification layer on concatenation.

Proposed Multi-Granularity Network: Information flows from SLC to FLC via gated connections: $$w_{g_k} = f(O_{g_k})$$ $$o_{g_{k+1}} = w_{g_k} \odot o_{g_k}$$

The gate $f$ (projection + activation) learns whether to propagate sentence-level information to fragment-level decisions. Loss combines Sigmoid activation for $L_{sl}$ and SoftMax for $L_{fl}$ with weighted combination (α = 0.9 for SLC, α = 0.1 for FLC).

Results

Fragment-Level Classification

Best model (Multi-Granularity Sigmoid): 24.42 P, 21.05 R, 22.58 F1

• BERT: 21.48 P, 21.39 R, 21.39 F1
• BERT-Joint: 20.11 P, 19.74 R, 19.92 F1
• BERT-Granularity: 23.85 P, 20.14 R, 21.80 F1
• Multi-Granularity ReLU: 23.98 P, 20.33 R, 21.82 F1

Evaluation: "Spans" metric checks only span boundary correctness; "Full Task" additionally requires correct technique label assignment.

Sentence-Level Classification

Best model (Multi-Granularity Sigmoid): 60.41 P, 61.58 R, 60.98 F1

• All-Propaganda baseline: 23.92 P, 100.00 R, 38.61 F1
• BERT: 63.20 P, 53.16 R, 57.74 F1
• BERT-Granularity: 62.80 P, 55.24 R, 58.76 R
• BERT-Joint: 62.84 P, 55.46 R, 58.91 F1

Multi-granularity network improves recall by 8.42% at SLC, showing benefit of lower-granularity information flow. The gate mechanism allows selective filtering of sentence-level predictions to improve precision at fragment level.

Notes

The shift from document-level propaganda classification to fine-grained fragment-level detection enables interpretability: systems can now explain which parts of an article are propagandistic and how (what specific techniques). This addresses a major limitation of prior work, which could only label entire articles as propagandistic without identifying the techniques at play.

The significant performance gap between SLC (best F1 60.98) and FLC (best F1 22.58) reflects the inherent difficulty of multi-class sequence labeling with variable-length spans and imbalanced classes. The multi-granularity architecture's selective information flow (gated connection from SLC to FLC) demonstrates the value of multi-task learning: allowing fragment-level decisions to leverage coarser-grained sentence-level predictions improves precision.

Class imbalance is severe (loaded language and name calling account for ~53% of instances), suggesting that future work should explore cost-sensitive learning, data augmentation, or synthetic example generation for rare techniques. The corpus and evaluation metrics are released publicly to facilitate future research on propaganda technique identification and the intersection of propaganda, argumentation, and media literacy.