MALicious INTent Dataset and Inoculating LLMs for Enhanced Disinformation Detection

Authors: Arkadiusz Modzelewski, Witold Sosnowski, Eleni Papadopulos, Elisa Sartori, Tiziano Labruna, Giovanni Da San Martino, Adam Wierzbicki

Venue: arXiv preprint, March 2026 — arXiv:2603.14525

TL;DR

MALINT is the first human-annotated English corpus pairing 1,600 news articles with annotations of both credibility and the malicious intentions behind disinformation. The paper evaluates 12 language models on intent classification and proposes Intent-Based Inoculation (IBI), an intent-augmented reasoning approach that improves zero-shot LLM disinformation detection by ~9% on MALINT and ~20% on other datasets across six languages.

Contributions

• MALINT dataset: First English corpus annotating both disinformation and malicious intent, developed with fact-checking experts to identify five intent categories (Undermining Credibility, Changing Political Views, Undermining International Organizations, Promoting Social Stereotypes, Promoting Anti-scientific Views).
• Intent classification benchmarks: Comprehensive evaluation of 7 small language models (BERT, DeBERTa, RoBERTa, DistilBERT) and 5 large language models (GPT-4o Mini, GPT-4.1, Gemma, Llama 3.3) on binary and multilabel intent detection.
• Intent-Based Inoculation framework: Demonstrates that exposing LLMs to structured analysis of malicious intent improves their zero-shot disinformation detection across diverse datasets and languages.

Method

The MALINT dataset combines articles from ~50 online sources, reviewed by fact-checking experts and classified into Credible and Disinformation categories. Credibility annotation uses three labels: Reliable, Unreliable, Mixed/Biased. For disinformation articles, annotators assign zero or more malicious intent labels drawn from a taxonomy based on prior research (Modzelewski et al. 2024) and refined through expert consultation.

Annotation follows a rigorous three-stage process: independent dual annotation, supervisor review, and consensus-based resolution. The final dataset contains 1,600 articles (63.5% credible, 36.5% disinformation) with 12.1% multilabeled for intent.

For intent-based inoculation, the paper frames intent detection as a multimodal task. A model \(M\) generates structured intent analysis of text \(T\) given external knowledge \(K_I\) (the intent taxonomy) and task guidance \(G_A\). This analysis feeds into a refutational preemption: combining the threat (presence of malicious intent) with LLM-generated rebuttals. The full inoculation prompt \(Z_T\) includes the original text, threat component, refutational preemption, and task-specific guidelines \(G_L\). A second model pass then performs binary detection: \(iT \sim M(Z_T)\).

Results

Intent classification (binary, per-intent): DeBERTa V3 Large and RoBERTa consistently achieve the highest F₁ scores across intent categories among SLMs. GPT 4.1 Mini performs best on UCPI and PSSA; Llama 3.3 70B on PASV and CPV. Compared to logistic regression baselines, fine-tuned SLMs achieve weighted F₁ of 82.1%.

Multilabel detection: DeBERTa V3 and RoBERTa outperform other SLMs, reaching Micro F₁ of 0.817 and 0.813. Logistic regression (one-vs-rest strategy) outperforms most LLMs in multilabel, though fine-tuned SLMs show superior ability to capture complex co-occurring intents.

Intent-augmented reasoning (IBI): On MALINT, intent-based inoculation improves baseline prompting across all five LLMs tested, with gains ranging from ~2% (GPT-4o Mini) to ~8% (Gemma 2.0 Flash). On five additional English datasets (ISOT FakeNews, CoAID, EUDsinfo, ECTF, EUVsDisinfo), IBI achieves ~9% average improvement over baselines. Cross-lingual evaluation on German, French, Polish, Estonian, Russian, and Spanish shows consistent improvements, with the largest gains in Estonian (~23%) and Russian (~19%).

Connections

• A Survey on Computational Propaganda Detection — related taxonomy of propaganda and disinformation intents; MALINT taxonomy refines this framework.
• Modzelewski et al. (2024) prior work on intent-based disinformation detection in Polish; MALINT extends to English with detailed annotation.
• Lucas et al. (2023) baseline prompting methods for zero-shot LLM detection; IBI builds on and extends these approaches.
• Malicious intent in disinformation — broader topic page on intent detection in information integrity.
• Inoculation and prebunking — psychological inoculation theory applied to disinformation; foundational to IBI design.

Notes

Strengths: Rigorous annotation with professional fact-checkers ensures high-quality intent labels. The five-category intent taxonomy is well-grounded in prior research and reflects real-world disinformation strategies. Comprehensive evaluation across SLMs and LLMs, as well as six languages, provides broad empirical grounding. The insight that intent-augmented reasoning improves zero-shot detection is novel and practically valuable for deployment scenarios.

Limitations: MALINT covers only 1,600 articles and five intent types; the authors acknowledge this may not exhaustively cover all malicious intents. The dataset is English-centric; multilingual evaluation uses translated/adapted datasets (EUVsDisinfo) rather than native-language annotation. Intent prediction itself becomes a prerequisite for IBI, which may limit applicability if intent detection is unavailable or unreliable in production settings. The paper does not deeply analyze which intent types most benefit from inoculation, suggesting potential for further work on intent-specific prompting strategies.

Broader impact: MALINT and the IBI framework could strengthen fact-checking systems and support researchers. The authors note potential misuse — malicious actors could study the taxonomy to craft more evasive disinformation — and recommend transparency, oversight, and safeguards.