Text generation evaluation encompasses metrics and methods for assessing whether machine-generated text meets quality criteria: fluency (does it read naturally?), coherence (does it form a coherent narrative?), factuality (is it true?), faithfulness (is it consistent with the source?), and relevance (does it address the input?).
Automatic metrics
Surface-level metrics:
- BLEU: Word n-gram overlap with reference translations
- ROUGE: Recall-Oriented Understudy for Gisting Evaluation (variant of BLEU for summarization)
- METEOR: Metric for Evaluation of Translation with Explicit Ordering
These metrics are fast and reproducible but correlate weakly with human judgments and can reward unfaithful text. Maynez et al. (2020) demonstrated that ROUGE-based metrics correlate poorly with human judgments of faithfulness and factuality in abstractive summarization.
Key papers
- Gao et al. (2022) — RARR proposes two novel metrics for evaluating text generation quality: (1) attribution scores measuring how much revised output is supported by retrieved evidence, and (2) preservation scores measuring how much of the original intent is retained; demonstrates that existing revision models fail to balance both dimensions, motivating the research-and-revision approach
Semantic metrics:
- BERTScore: Contextual token embeddings to measure semantic similarity
- Entailment-based scores: Does output entail source or vice versa?
- Question-Answering-based: Generate questions from output; check consistency with source
LLM-based evaluation:
- Using large language models to score other outputs
- Reference-free and scalable but potentially biased
Manual evaluation
Human evaluation provides ground truth but is expensive and slow. Researchers typically evaluate:
- Fluency and grammaticality
- Coherence and logical flow
- Faithfulness to source
- Factual correctness
- Relevance to query or task
Inter-annotator agreement is measured via Cohen's κ or Fleiss' κ.
Challenges and open questions
- Reference dependency: Most metrics require gold-standard reference outputs, but multiple high-quality outputs often exist
- Domain variation: Metrics tuned on news summarization may perform poorly on scientific or clinical text
- Alignment with user needs: Automatic metrics often disagree with what users actually value
- Temporal dynamics: Factuality of generated text can change over time as facts evolve
Related topics
- factuality — evaluating correctness
- faithfulness — evaluating consistency with source
- hallucination — detecting false content
- abstractive-summarization — major application area
- natural-language-generation — broader task area
