Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks

Contributions

• Hybrid parametric-nonparametric architecture: Combines a pre-trained seq2seq generator with dense Wikipedia retrieval (DPR), allowing both worlds—parametrically-learned knowledge and explicitly accessible external memory
• Two RAG formulations: RAG-Sequence (same document for entire sequence) and RAG-Token (different documents per token), enabling flexible document marginalization strategies
• Strong empirical results: Achieves state-of-the-art on open-domain QA (Natural Questions, TriviaQA, WebQuestions, CuratedTrec), abstractive QA (MS-MARCO), Jeopardy question generation, and fact verification (FEVER)
• Factuality improvements: Human evaluation on Jeopardy QGen shows RAG generates more factually accurate (42.7%) and specific (37.4%) text than BART baseline (7.1% and 16.8%)
• Knowledge hot-swapping: Non-parametric memory can be updated at test time by replacing the retrieval index, enabling up-to-date knowledge without retraining

Method

Architecture: RAG combines three components:

1. Query Encoder: BERT-based encoder produces query representation q(x) from input x
2. Retriever: Dense Passage Retriever (DPR) with bi-encoder architecture returns top-k documents given q(x) and document encoder representations
3. Generator: Marginalization approach combines parametric generator (seq2seq) with retrieved document probabilities

Two formulations:

• RAG-Sequence: Uses same retrieved document for entire output sequence

  p(y|x) ≈ Σ_z p_η(z|x) p_θ(y|x, z)
  

  where top-k documents retrieved from retriever, each marginalized in generation

• RAG-Token: Different document per generated token; more flexible but computationally expensive

  p(y|x) ≈ ∏_i Σ_z p_η(z|x) p_θ(y_i|x, z, y_{1:i-1})
  

Training: End-to-end training with retriever (DPR) and generator (BART) jointly optimized on task-specific input-output pairs using negative marginal log-likelihood. Document index kept fixed; only query encoder and BART generator updated.

Decoding: For RAG-Sequence, standard beam search over retrieved documents. For RAG-Token (likelihood doesn't decompose per-token), use "Thorough Decoding" with forward passes for each candidate or "Fast Decoding" with approximation.

Results

Open-domain QA (Table 1):

• Natural Questions: RAG-Token 44.1, RAG-Seq 44.5 vs. REALM 40.4, DPR 41.5
• TriviaQA: RAG-Token 55.3/68.0, RAG-Seq 56.8/68.0 (Em/F1)
• WebQuestions: RAG-Token 45.5, RAG-Seq 45.2
• CuratedTrec: RAG-Token 50.0, RAG-Seq 52.2
• RAG outperforms retrieve-and-extract (DPR+seq2seq) and parametric-only (REALM, T5) baselines

Abstractive QA on MS-MARCO:

• RAG-Seq: 40.8 BLEU, 44.2 ROUGE-L (best on open-book split for TQA)
• Outperforms BART-only (38.2, 41.6) and REALM (no gold passages, 40.7, 46.8)

Jeopardy QGen (Table 4):

• Human eval: RAG-Token factually correct in 42.7% of cases vs. BART 7.1%, and more specific 37.4% vs 16.8%
• RAG-Seq 77.8% n-gram diversity (ROUGE-1) vs BART 70.7%

Fact Verification (FEVER):

• RAG-Token: 75.1 (3-way), 91.6 (2-way) vs. state-of-the-art complex systems (within 4.3%)
• Demonstrates effectiveness on classification task despite generative formulation

Notes

Strengths:

• Principled hybrid approach: Clean separation of parametric and non-parametric memory; both components pre-trained on stable data (seq2seq on downstream tasks, DPR on QA)
• Broad empirical validation: Tested across six diverse knowledge-intensive tasks (QA variants, abstractive generation, question generation, fact verification), showing consistent improvements
• Interpretability advantage: Retrieved documents provide explicit evidence for answers, improving transparency over parametric-only models; crucial for fact-checking applications
• Practical knowledge updates: Non-parametric index swappable without model retraining; demonstrated by building indices on Wikipedia from different years (2016 vs. 2018) showing knowledge can be updated at test time
• Generation quality over extraction: RAG generates correct answers (11.8% accuracy on NQ when document has answer but not as extractable span) that extractive methods miss
• Human-validated factuality: Jeopardy QGen evaluation shows RAG not only more accurate but also more specific and diverse than BART

Weaknesses:

• Retrieval bottleneck: Performance depends critically on retriever recall; oracle experiments show document retrieval captures 55.30% of claimable information—sentence selection becomes next bottleneck
• Computational cost: Requires k forward passes through generator at test time (one per retrieved document); both RAG-Sequence and RAG-Token slower than parametric-only generation
• Wikipedia-centric evaluation: Knowledge source is Wikipedia for all tasks; real-world fact-checking requires handling claims about recent events, specialized domains (medical, scientific), and sources Wikipedia may not cover
• Limited error analysis: No systematic analysis of failure modes; when does RAG fail to retrieve relevant documents? When does generation fail despite correct retrieval?
• Not Enough Info handling: FEVER results on NOT ENOUGH INFO class not separately reported; unclear how well RAG handles inherently non-verifiable claims
• Freezing retriever: Document index not updated during training; potential mismatch between query encoder and retriever embeddings learned on different objectives

Future directions:

• Multi-hop retrieval for claims requiring evidence synthesis across documents
• Fine-grained evidence attribution (which retrieved passages generate which parts of output)
• Updating retriever jointly with generator end-to-end
• Cross-domain generalization (how does Wikipedia knowledge transfer to medical, legal, scientific domains)
• Real-time knowledge updates for claims about recent events