Contributions
- Attribution and preservation metrics: Proposes fine-grained measures for evaluating how much of revised text is attributable to retrieved evidence (attribution) and how much of the original intent is preserved (preservation), including sentence-level and automated variants.
- RARR method: A simple yet effective pipeline with three stages: (1) generating search queries to cover aspects of the input, (2) retrieving evidence snippets from web documents, (3) editing the output to align with evidence while minimizing unnecessary changes.
- Empirical breadth: Demonstrates that RARR generalizes across three diverse generation tasks (Natural Questions, SQuAD 2.0, QReCC) and outperforms strong baselines (EFEC, LaMDA) on the attribution-preservation tradeoff.
- Analysis of generalizability: Shows existing revision models do not generalize across tasks, motivating the research-revision approach that works post-hoc on any generation model.
Method
Research stage: Given a generated text passage, RARR generates a set of search queries using few-shot prompting of a large language model (PaLM). Each query investigates a different aspect of the passage. A retrieval model (trained T5) ranks web pages by relevance; the system extracts evidence snippets (e.g., 4-sentence windows) from the top results.
Revision stage: For each retrieved evidence snippet, an agreement model (implemented with few-shot PaLM prompting using chain-of-thought) detects disagreement between the current output and the evidence. If disagreement is found, an edit model revises the output to agree with the evidence. If not, the output is left unchanged. This iterative process ensures the revised text can be grounded in retrieved evidence while minimizing edits.
Attribution report: Finally, selected evidence snippets are compiled into an attribution report, showing which parts of the revised text are supported by which sources.
The approach is simple and post-hoc — it works on top of any text generation model without requiring retraining.
Results
On Natural Questions (long-form factoid answers), RARR achieves 45.6 AIS (attribution) vs. 45.6→54.9 before→after revision, with 89.6 preservation (vs. 83.1 for preservation-only baseline). On SQuAD 2.0 (reasoning chains), attribution improves from 37.6 to 45.1 while preserving 89.9% of intent. On QReCC (knowledge-intensive dialogs), RARR reaches 93.8 attribution and 84.6 preservation, compared to 78.1 for LaMDA.
Key finding: RARR is the only method that performs robustly across all three tasks on the attribution-preservation tradeoff. Baselines (EFEC, LaMDA) sacrifice preservation severely to improve attribution or vice versa.
Notes
Strength: The paper tackles a real and important problem (LM hallucinations) with a practical, generalizable solution that doesn't require model retraining. The three-stage pipeline is elegant and the metrics are well-motivated. Breadth across three distinct task types strengthens the contribution.
Limitations acknowledged by authors: (1) Evaluation doesn't capture all aspects of faithfulness—some sentences are self-evident and don't need attribution. (2) The method is computationally expensive due to reliance on few-shot prompting. (3) Attribution doesn't guarantee the source itself is trustworthy (misinformation can be well-attributed to bad sources). Future work could explore synthetic data generation to train smaller models and better integration of source credibility assessment.
The work sits at the intersection of fact-checking and LLM reliability—highly relevant to combating AI-generated misinformation.