Skip to content

Synthesis-Stage Attacks on LLMs

Synthesis-stage attacks target large language models at the point where retrieved documents are synthesized into a final answer. Unlike adversarial prompts or jailbreaks that manipulate the user-facing prompt, synthesis attacks operate by contaminating the retrieval pipeline—either by poisoning retrieved documents, manipulating ranking algorithms, or embedding false information in fluent, authoritative-appearing text.

Attack mechanisms

Document poisoning: Adversary creates or compromises web documents containing targeted false claims, optimized to rank highly in retrieval-augmented systems via keyword relevance, semantic similarity, or ranking signals.

Ranking manipulation: Techniques like Generative Engine Optimization (GEO) increase the prominence of poisoned documents in retrieval results, causing them to appear in the top-k candidates fed to the synthesis model.

Fluent misinformation embedding: False claims are embedded in fluent, well-structured text that resembles legitimate information sources, making them harder to detect at the retrieval filtering or generation stages.

Defense approaches

Chunk-level filtering: Guardrails applied to retrieved documents before synthesis, classifying chunks as safe or unsafe based on content patterns, provenance, or learned detectors.

Contrastive detection: Training detectors to recognize manipulation patterns by contrasting paired rewrites (information-preserving vs. information-distorting variants of the same source).

Output-rail fact-checking: Post-generation verification of LLM answers against retrieved evidence, detecting when synthesized claims diverge from or contradict the evidence.

Retrieval robustness: Improving retrieval pipeline robustness through source credibility assessment, diversity in ranked results, or semantic matching that resists keyword-stuffing attacks.

Key papers in this wiki