FacTool: Factuality Detection in Generative AI - A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarios

Contributions

• Unified, task-agnostic factuality detection framework: Extends traditional fact-checking (binary true/false on single claims) to a 5-stage pipeline—claim extraction, query generation, tool querying, evidence collection, agreement verification—applicable across multiple domains without task-specific retraining

• Fine-grained factuality assessment: Proposes detecting factual errors at multiple granularities: not just responses-level verdicts, but sentence-level, statement-level, and tuple-level claims within long-form generated text, enabling better pinpointing of where errors occur

• Tool-augmented verification framework: Demonstrates how to leverage diverse tools (Google Search, code interpreters, Python, Google Scholar, LLMs) as external evidence sources, reducing reliance on a single fixed knowledge base and enabling richer reasoning about different claim types

• Comprehensive multi-task evaluation: Evaluates on four representative tasks—knowledge-based QA (long-form answers), code generation (executable snippets), math problem solving (step-by-step reasoning), scientific literature review writing (multi-sentence synthesis)—showing consistent improvements over unaided LLM self-checking

• Analysis of failure modes: Documents where tool-augmented verification fails (ambiguous claims, conflicting evidence, reasoning errors, title-matching errors in scientific literature) and highlights task-specific vulnerabilities

Method

The FacTool framework consists of five main components:

1. Claim Extraction

• Given a long-form response (text, code, math reasoning, or scientific text), extract all verifiable claims
• For KB-QA and Sci-Lit: Use ChatGPT with claim definition-based prompting (e.g., "extract atomic facts no longer than 15 words")
• For Code Generation: Each code snippet within the response counts as a single claim to verify
• For Math Problems: Each arithmetic operation or calculation step is a separate claim

2. Query Generation

• Convert each extracted claim into one or more queries suitable for external tools
• KB-QA claims → search engine queries (via Google Search API)
• Code claims → test case inputs for verification
• Math claims → Python executable code that validates the calculation
• Sci-Lit claims → paper titles and metadata queries (via Google Scholar API)

3. Tool Querying & Evidence Collection

• Google Search (KB-QA): Retrieve top search results; parse and filter for relevant evidence snippets
• Code interpreter + Python (Code Generation): Execute generated code snippets against unit test inputs; collect whether execution succeeds and results match expected outputs
• Python calculator (Math): Execute mathematical operations and compare calculated results to claimed results
• Google Scholar (Sci-Lit): Search for paper metadata (title, authors, year, publication) and verify against claims in the generated review

4. Agreement Verification

• Each claim ci receives a binary factuality label L ∈ {True, False} based on whether it is supported by collected evidence
• For KB-QA: Use chain-of-thought reasoning with evidence to judge whether the claim logically follows from retrieved information
• For Code: Majority voting across all test cases—if all pass, the snippet is factual
• For Math: Binary check—if all arithmetic operations produce the correct results, the claim is factual
• For Sci-Lit: String-matching on paper titles, year, and author lists; if metadata matches retrieved results, the claim is factual

5. Response-Level Aggregation

• Compute both claim-level and response-level factuality scores
• For KB-QA: Weighted claim-level accuracy (weights by response length contribution)
• For Code, Math, Sci-Lit: F1 score computed at both claim and response levels

Results

Datasets & Tasks

Main Findings

FacTool powered by GPT-4 outperforms all baselines across all scenarios:

Model comparison on KB-QA (representative task):

FacTool powered by GPT-4 achieves the highest accuracy across all models tested (GPT-4, ChatGPT, Claude-v1, Bard, Vicuna-13B). Supervised fine-tuned models like Vicuna-13B show reasonable performance on straightforward KB-QA scenarios but fail in more challenging code generation, math, and scientific writing tasks.

Failure modes identified:

1. Reasoning errors (KB-QA): LLM struggles to connect a claim to evidence even when relevant information is retrieved (e.g., "Fortune cookies were invented in San Francisco" when evidence supports a different story)
2. Conflicting evidence (KB-QA): Multiple search results contradict each other, causing confusion for LLMs
3. Limited diversity in synthetic test cases (Code): Generated test cases may not cover edge cases, allowing buggy code to pass
4. Round-off and reasoning errors (Math): Python calculations may differ from human reasoning due to floating-point precision; LLMs sometimes substitute wrong values in calculations
5. Title matching failures (Sci-Lit): Generated review cites papers with slightly different or abbreviated titles; exact string matching fails even when the paper is correct

Notes

Strengths:

• Practical and flexible framework: Not limited to a single task or domain; the five-stage pipeline can be instantiated with different tools for different claim types, making it genuinely adaptable
• Addresses real limitations of LLM self-checking: Shows empirically that unaided LLMs (even GPT-4) cannot reliably audit their own outputs; external tools are necessary
• Multiple evaluation scenarios: Tests on four representative domains (knowledge, code, reasoning, synthesis) strengthens claims about domain-agnosticism
• Honest failure analysis: Authors document specific failure modes and their frequencies, enabling researchers to identify weak points
• Reproducible methodology: Detailed prompts and evaluation procedures provided in appendices; GitHub repo available

Weaknesses:

• Computational cost not discussed: Executing code, querying Google Search, calling multiple LLMs multiple times incurs significant cost and latency; practical deployment may be infeasible for high-volume applications
• Google Search dependency: KB-QA and Sci-Lit results depend on search engine coverage and ranking; geographically biased and may miss non-indexed or paywall-protected information
• Limited to well-defined factual claims: Framework struggles with subjective judgments, nuanced opinions, or adversarially ambiguous statements (intentional design, but a limitation nonetheless)
• Title matching brittleness in Sci-Lit: Reliance on exact string matching for paper metadata is fragile; acronyms, abbreviations, and name variations break matching despite semantic equivalence
• No comparison with specialized verifiers: Does not benchmark against task-specific fact-checking models (e.g., DeBERTa fine-tuned on FEVER, specialized code-review tools, math-checking systems like Lean)
• Small evaluation scale: 100 responses per task is modest; larger-scale evaluation across more datasets would strengthen generalization claims

Follow-up opportunities:

• Integrate fuzzy matching or semantic similarity (e.g., BERT embeddings) for Sci-Lit paper title matching to handle abbreviations and paraphrasing
• Explore caching and batch processing to reduce per-query cost and latency in production scenarios
• Extend to multimodal claims (images + text) relevant to deepfake and synthetic media detection
• Develop adversarial evaluation: design deliberately tricky claims that fool both LLM and tools, identify systematic blind spots
• Apply FacTool to real-time fact-checking scenarios (social media posts, breaking news) where speed and cost are critical constraints