Scientific writing evaluation focuses on the factual accuracy of generated scientific content, particularly literature reviews, syntheses, and summaries:
Challenges in scientific text:
- Citation accuracy: Generated papers must cite real works with correct metadata (title, authors, year)
- Claim grounding: Synthesis claims must be supported by cited papers
- Terminology precision: Scientific terminology must be used correctly
- Temporal accuracy: Publication years and discovery timelines must be accurate
Factuality issues in LLM-generated science:
- Models hallucinate fictional papers or misattribute claims to authors
- Generated citations use slightly different titles or author orderings that don't match real publications
- Misleading synthesis of evidence without proper hedging of uncertainty
- Conflation of distinct concepts or misrepresentation of author positions
Verification approaches:
- Cross-referencing generated citations against databases (Google Scholar, arXiv)
- Metadata matching (title, authors, year against ground truth)
- Expert review of claim-citation alignment
- Semantic matching for title variations and abbreviations
Key papers in this wiki
- 2023-chern-factool — Evaluates generated scientific literature reviews by matching paper metadata against Google Scholar; identifies title-matching brittleness and author-list validation as key verification bottlenecks; achieves 98.39% claim-level F1 with tool-augmented framework