Skip to content
A Multi-Stage Agentic Framework for Effective Counter-Narrative Generation and Refinement

A Multi-Stage Agentic Framework for Effective Counter-Narrative Generation and Refinement

Authors: Carmel Kronfeld, Sharva Gogawale, Tetsuro Kobayashi, Irad Ben-Gal

Venue: arXiv cs.SI, September 2026

TL;DR

Counter-narratives are a scalable defense against misinformation and hate speech, but manual authoring is slow and labor-intensive. This paper proposes a multi-stage agent-based framework using LLMs to generate and iteratively refine counter-narratives on pro-Russian misinformation about Ukraine. A pilot study identifies effective rhetorical technique–writing style pairs; a multi-agent refinement loop optimizes narratives for persuasiveness, emotional engagement, and shareability; human validation confirms quality gains; and a simulated experiment shows refined narratives significantly reduce perceived persuasiveness of the target propaganda.

Contributions

  • A hybrid human-AI pipeline unifying counter-narrative generation, refinement, and evaluation into a single iterable process, with three specialized LLM agents (Generator, Evaluator, Mediator) coordinating via a Memory Summarizer.
  • Empirical identification of effective rhetorical techniques (e.g., repetition, fear mongering, metaphor) and writing styles (emotional, empathic, metaphorical) for constructing counter-narratives to specific misinformation themes.
  • A multi-agent refinement system that iteratively improves narratives per three KPIs—persuasiveness, emotional engagement, and shareability—using LLM-as-judge evaluation and simulated pro-Russian evaluator feedback.
  • Demonstration that refined counter-narratives outperform vanilla LLM baselines and achieve safety parity with expert-curated reference datasets (CONAN, MT-CONAN).
  • Evidence that counter-narratives effectively reduce perceived persuasiveness of pro-Russian propaganda in a simulated audience experiment.

Method

The system comprises five stages:

1. Pilot Experiment (Technique Discovery). Five political-science undergraduates evaluated counter-narratives against pro-Russian base claims extracted from X (formerly Twitter) with HDBSCAN clustering. A factorial design crossed 13 rhetorical techniques (Persuasive, Optimistic, Emotional, Rational, Dry language, Metaphorical, Amusing, Cynical, Empathic, Detached, Sarcastic, Annoying, Card stacking) with 10 writing styles, generating 390 narratives per KPI. Human judges scored each on persuasiveness, emotional engagement, and shareability (0–100 scale), using forced-choice comparisons to mitigate bias. LASSO-regularized logistic regression identified the highest-performing technique–style pair per KPI.

2. Refinement Per-Claim Process. Best technique–style combos from the pilot inform prompt design for three specialized CN Generator Agents (one per KPI: persuasiveness, emotional engagement, shareability). For each of 20 representative pro-Russian narratives, the system generates narratives across all three KPI objectives, iteratively refining them based on LLM-as-judge scores and a Manager Agent's aggregated feedback. Stops after 25 iterations or 8 rounds without improvement.

3. Human Validation. Two expert evaluators judged 360 refined CN pairs on three pair types (high-difference, low-difference, and comparable quality). Alignment accuracy exceeded 80% for persuasiveness and shareability; emotional engagement showed weaker inter-rater agreement, but evaluators reliably distinguished refined CNs in high-difference pairs.

4. Safety Analysis. 180 refined CNs (20 narratives × 3 KPIs × 3 samples) were assessed against CONAN and MT-CONAN reference corpora using toxicity, insult, threat, and offensiveness classifiers. All four metrics remained at or below human-baseline levels; toxicity and threat showed significant improvement over pre-refinement CNs.

5. Simulated Effectiveness Experiment. Eighteen pro-Russian evaluator agents (built from Twitter user profiles and LLM persona simulation) rated pro-Russian narratives under two conditions: (1) narrative alone (control), and (2) narrative paired with a refined CN (treatment). KPI scores (how persuasive/engaging/shareable the pro-Russian narrative appeared) dropped significantly in the treatment condition across all three metrics (Kruskal-Wallis p < 0.001).

Results

Pilot Experiment: Groups 3 and 6 (Humanitarian–Denazification and Military Success narratives) achieved significantly higher persuasiveness peaks than Groups 1 and 2, with Group 3 also outperforming Group 4. Shareability and emotional engagement showed theme-dependent learning patterns; persuasiveness remained the most difficult and volatile KPI to optimize. Identified best-performing pairs: (Repetition, Emotional) for persuasiveness; (Fear mongering, Empathic) for emotional engagement; (Card stacking, Metaphorical) for shareability.

Refinement Outcomes: Refined CNs substantially outperformed vanilla baselines on all three KPIs. Human validation confirmed quality improvements in high-difference pairs (81.87% accuracy on persuasiveness, 80.63% on emotional engagement, 83.75% on shareability; Cohen's κ ≥ 0.60). Theme-dependent score ceilings despite consistent learning efficiency suggest that narrative effectiveness is bounded by thematic fit rather than prompt quality alone.

Safety: Refined CNs scored comparably or better than expert-curated baselines on all four safety metrics. Offensiveness remained high across all groups (0.725), reflecting the inherent nature of counter-addressing hate, but no degradation in safety was introduced by refinement.

Simulated Effectiveness: CNs significantly reduced pro-Russian narrative persuasiveness (vanilla: 56.95 → refined: 49.82, p < 0.001), emotional engagement (56.07 → 33.27, p < 0.001), and shareability (58.93 → 39.18, p < 0.001). Refined CNs achieved significantly lower scores than vanilla baselines, demonstrating measurable suppressive effect on propaganda.

Connections

  • Related to Cook et al. on inoculation theory via shared framing of pre-emptive narrative intervention.
  • Methodologically informed by Transformers and word embeddings as substrates for LLM agents.
  • Extends generate-prune-select pipeline concepts with iterative multi-agent refinement and simulated audience feedback.
  • Addresses scalability bottlenecks in manual counter-narrative authoring noted in prior defensive research by automating generation and refinement while maintaining human oversight.
  • Contributes to computational propaganda research with scalable methods for countering coordinated disinformation campaigns.

Notes

Strengths: - Novel multi-stage pipeline combining generation, refinement, and evaluation in an integrated system; iterative LLM-as-judge feedback is creative and practical. - Comprehensive evaluation across safety, human preference, and simulated effectiveness; safety parity with expert corpora is reassuring. - Transparent about limitations: pilot study used only five evaluators (generalizability concerns), evaluators simulated pro-Russian users (not actual audience), and refinement relied on LLM judgment (not human judgment at every step). - Ethical grounding: IRB approval, controlled offline experiments, no real social media deployment, disclosure of dual-use risks.

Weaknesses: - LLM-based evaluation agents may not fully capture nuanced persuasiveness; the paper acknowledges this gap and confirms findings via human validation, but human validation was limited to 360 pairs. - Technique taxonomy (13 techniques × 10 styles) is somewhat ad hoc; no principled derivation from rhetoric literature. - Simulated effectiveness experiment uses only 18 evaluator personas; real-world audience heterogeneity is broader. - No comparison to existing counter-narrative generation systems (FIRL, DPO-tuned models); benchmarking against state-of-the-art would strengthen claims. - Scope limited to Ukraine/Russia conflict; generalizability to other disinformation domains (health misinformation, election denial) remains open.

Missing: - Long-term persistence of refined CNs' suppressive effect; is a single exposure sustained? - Cost-benefit analysis: refinement loop requires multiple LLM calls; what is the computational cost vs. manual authoring? - Failure case analysis: which pro-Russian themes prove resistant to counter-narrative intervention?

Overall: A well-motivated, technically competent contribution addressing a real bottleneck in counter-narrative research (manual authoring at scale). The multi-agent framework is novel and the evaluation rigorous, though scope and baseline comparisons are limited. Highly relevant to operationalizing defenses against coordinated disinformation.