SINcon: Mitigate LLM-Generated Malicious Message Injection Attack for Rumor Detection

Authors: Mingqing Zhang, Qiang Liu, Xiang Tao, Shu Wu, Liang Wang

Venue: arXiv preprint — arXiv:2504.07135

TL;DR

Message Propagation Tree (MPT) based rumor detection models are vulnerable to LLM-generated malicious message injection attacks that alter tree topology and semantics. The paper proposes SINcon, a contrastive learning defense that regularizes models to assign similar predictive influence to nodes of different importance levels, significantly improving robustness against such attacks on Twitter and Weibo datasets while maintaining clean-data performance.

Contributions

• Identify that MPT-based rumor detection models suffer from imbalanced node influence: nodes with high predictive influence are vulnerable to targeted attacks that focus on those high-influence nodes.
• Propose SINcon (Similarizing the predictive Influence of Nodes with Contrastive Learning), a self-supervised regularization method that ensures nodes with different influence levels exert more uniform impact on model predictions.
• Demonstrate through extensive experiments on Twitter and Weibo datasets that SINcon significantly enhances robustness to LLM-driven message injection attacks while preserving accuracy on clean data.

Method

SINcon operates by first identifying the top 10% of nodes in an MPT with the highest and lowest influence scores (computed via attribution methods). These are designated as "important" and "unimportant" nodes, respectively.

The method introduces two data augmentation strategies: 1. Mask important nodes: creates augmented trees where important nodes are masked, removing their information from the model. 2. Mask unimportant nodes: creates augmented trees where unimportant nodes are masked.

The contrastive learning objective enforces two principles: (1) augmented trees with masked important nodes should yield similar predictions to the original tree, and (2) augmented trees with masked unimportant nodes should also yield similar predictions. This encourages the model to treat information in nodes of different importance levels more uniformly, reducing the effectiveness of targeted attacks on high-influence nodes.

The full training objective combines the standard supervised loss with two components of the contrastive loss, weighted by hyperparameters α₁ and α₂. The method is evaluated against HMIA-LLM attacks (using ChatGPT to generate malicious messages with a root-centric homophily constraint).

Results

Evaluated on four MPT-based rumor detection baselines (BiGCN, GACL, GARD) across Twitter and Weibo datasets:

Accuracy under Attack (AUA) improvements: - Twitter: maximum 36.12% improvement, minimum 15.96%, average 26.38% - Weibo: maximum 36.97% improvement, minimum 16.97%, average 28.03%

Accuracy on clean data (ACC): - Slight decline of ~1.38% on average, well within acceptable bounds (some configurations show no decline)

The ablation studies show that the data augmentation strategy (masking based on influence scores) is critical; random masking is substantially less effective. Sensitivity analysis on hyperparameters α₁ and α₂ confirms the method's stability across reasonable parameter ranges.

Connections

• Related to Propagation-based fake news detection via shared reliance on Message Propagation Tree structure and graph neural networks.
• Extends work on adversarial attacks in misinformation detection; contributes to a growing body of research on adversarial robustness for rumor detection systems.
• Uses Graph Neural Networks for propagation pattern learning; relevant to graph-based detection methods.
• Uses contrastive learning as a regularization technique, similar to prior work on graph-based rumor detection.

Notes

The paper makes a solid empirical contribution by identifying and addressing a concrete vulnerability in widely-used MPT-based architectures. The focus on node influence imbalance is intuitive and well-motivated. The contrastive learning approach is a reasonable defense mechanism, though future work could explore more sophisticated attack models (e.g., adaptive attacks that account for the defense) and generalization to other message injection strategies beyond homophily-constrained injection. The modest decline in clean-data accuracy is acceptable for a robustness improvement of this magnitude, though practitioners may need to tune α₁ and α₂ for their specific deployment context.