Skip to content
Lazy Grounding: Attacking Search Agents with Factual Evidence

Lazy Grounding: Attacking Search Agents with Factual Evidence

Authors: Yulin Zhang, Yukun Huang, Sanxing Chen, Tianyi Lin, Ziang Yang, Xunjian Yin, Bhuwan Dhingra
Affiliation: Duke University, Durham, NC, USA
Venue: arXiv, 2026 — arXiv:2608.30303

TL;DR

Retrieval-augmented search agents fail to robustly distinguish between evidence for the current question and evidence for semantically nearby questions. By injecting factually correct documents supporting a neighboring question into the search corpus, agents often misapply this evidence to answer the original question, achieving accuracy drops of 5.9 points on average (up to 17.3 points) across 12 model-benchmark pairs without using false or malicious content.

Contributions

  • Identifies and formalizes "lazy grounding" as a novel failure mode in retrieval-augmented generation where agents transfer answers from nearby questions to original ones
  • Demonstrates the vulnerability across multiple models (Tongyi Deep Research, GPT-5 Mini, Gemini 3 Flash) and benchmarks (XBench, GAIA, BrowseComp+, HLE)
  • Shows that position, format (answer-field records vs. natural prose), and temporal placement of nearby evidence all influence agent susceptibility
  • Provides constraint-checking prompt as a partial defense, though lazy grounding persists

Method

The attack constructs adversarial evidence through answer rewrites: given an original question \(q\) with answer \(a\), they generate a neighboring question \(q'\) with a different answer \(b\), preserving surface cues (entity, event, topic, answer type) while changing the core constraint. Factual documents supporting \(b\) for \(q'\) are then injected into the search corpus \(C \leftarrow C \cup D\). A robust agent should answer \(a\) using original evidence; a vulnerable agent adopts \(b\) using misapplied evidence.

Evaluation metrics capture: - Clean accuracy: Agent performance on original corpus without adversarial documents - Augmented accuracy: Performance after nearby-evidence injection - Rewrite-answer adoption (RAA): How often the agent adopts the neighboring question's answer

Results

Accuracy degradation across all models: Table 1 shows consistent accuracy drops on all 12 model-benchmark pairs. Tongyi Deep Research on XBench suffers the largest drop (17.3 points, from 69.3 to 52.0, 95% CI [10.7, 24.0]). Across all models, average drop is 5.9 points.

Neighboring evidence pulls agents toward wrong answers: Rewrite-answer adoption rates are high across all model-benchmark pairs. On XBench, Tongyi Deep Research achieves RAA of 27.0 and RAA-C/F (adoption on clean-correct / clean-wrong examples) of 20.7/41.3, indicating the attack attracts agents to the neighboring answer even when clean accuracy remains reasonable.

Format and position matter: Answer-field records (structured QA pairs) produce higher adoption (59.0 augmented accuracy on GAIA with Tongyi Deep Research) than natural prose claims (60.0). Temporal position of injected evidence influences its effect: late-turn evidence produces higher rewrite adoption than early-turn.

Partial defense via constraint checking: A simple prompt instructing agents to keep the original question's constraints fixed reduces rewrite adoption from 20.7 to 14.3 (RAA on XBench), but does not eliminate lazy grounding.

Connections

Notes

This work reveals a subtle but pervasive vulnerability in retrieval-augmented systems: the agents' reasoning process fails to robustly ground evidence to the exact question asked. Unlike prior work on false-document injection or misinformation, lazy grounding exploits the factual correctness of retrieved evidence—the standard defense against corpus poisoning—by showing that facts themselves can mislead when applied out of context. Future defenses need mechanisms that verify evidence alignment with the exact constraints of the current query, not just reliability of the source.