Long-form text evaluation addresses challenges in assessing multi-sentence, multi-claim responses that differ fundamentally from single-statement fact-checking:
Unique challenges:
- Claim interdependence: Claims within responses reference earlier statements and build on each other
- Implicit claims: Not all factual statements are explicitly stated
- Granularity tradeoff: Response-level labels hide fine-grained errors; sentence-level assessment is labor-intensive
- Evidence scarcity: Long responses often contain claims with limited search coverage
Evaluation dimensions:
- Factuality: Accuracy of individual claims
- Completeness: Whether response addresses all aspects of the question
- Coherence: Logical flow between sentences
- Citation alignment: Claims are supported by cited evidence
Factuality in long-form contexts:
- Single false claim can invalidate entire response despite mostly-true content
- Cumulative effect: errors compound across sentences
- Task-dependent sensitivity: math errors fatal; minor imprecisions in QA tolerable
Key papers in this wiki
- 2023-chern-factool — Proposes fine-grained factuality detection for long-form texts; evaluates at both claim-level and response-level across knowledge-based QA, code generation, math reasoning, and scientific writing; demonstrates that response-level aggregation requires careful weighting by claim contribution