Retrieval-Augmented Generation

Retrieval-augmented generation (RAG) combines neural information retrieval with generative language models to incorporate external knowledge during generation. Rather than storing all knowledge in model parameters, RAG systems retrieve relevant documents from a corpus and condition generation on them, enabling more accurate and up-to-date outputs.

Key papers

• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — Pioneering work combining a dense retriever with a sequence-to-sequence model for knowledge-intensive tasks
• Atlas: Few-shot Learning with Retrieval Augmented Language Models — Demonstrates joint training of retriever and language model for improved few-shot performance with compact models
• Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection — Enables adaptive retrieval and self-reflection through special tokens, allowing models to decide when to retrieve and critique their outputs

Related topics

• Knowledge-Intensive NLP (application domain)
• Information Retrieval (retrieval component)
• Language Models (generation component)
• Dense Retrieval (retrieval architecture)
• In-Context Learning (alternative knowledge augmentation approach)