SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models

Contributions

• First work to propose a practical zero-resource hallucination detection method for general LLM responses that requires only black-box access (text-in, text-out)
• Demonstrates that information consistency across stochastically sampled completions is a strong signal for factuality without requiring external resources
• Proposes five variants of SelfCheckGPT (BERTScore, QA, n-gram, NLI, Prompt-based) and shows prompt-based scoring achieves best performance
• Creates and releases a manually annotated WikiBio-derived dataset with sentence-level factuality labels (238 passages, 1908 sentences)
• Shows SelfCheckGPT substantially outperforms grey-box methods (using output probabilities) and black-box baselines on passage-level ranking

Method

SelfCheckGPT operates by:

1. Sampling: Given a user query, generate N stochastic responses from the LLM (using temperature 1.0 to maximize diversity)
2. Consistency Measurement: For each sentence in the main response, compare it against the N sampled responses using one of five scoring schemes: - BERTScore: Average maximum BERT similarity between the sentence and its closest match in samples - Question Answering (QA): Generate questions about the sentence and check if sampled responses answer consistently - n-gram: Find the least-likely token in an n-gram language model trained on the samples - Natural Language Inference (NLI): Use a classifier to check if samples entail/contradict the sentence - Prompt-based: Use an LLM to directly assess whether the sentence is supported by the samples
3. Scoring: Compute a consistency score S(i) ∈ [0.0, 1.0] where 0 indicates hallucination and 1.0 indicates factuality
4. Passage-level: Average sentence scores to obtain passage-level factuality rankings

The method handles the key challenge that hallucinated facts tend to appear inconsistently across samples (or contradict them), while factual statements are likely to appear consistently since they reflect knowledge the model learned during training.

Results

Sentence-level hallucination detection (WikiBio dataset, 1908 annotated sentences):

• SelfCheckGPT-Prompt (best variant): AUC-PR 93.42, Spearman correlation 0.78 with human judgments
• SelfCheckGPT-NLI: AUC-PR 92.50
• SelfCheckGPT-Unigram: AUC-PR 85.63
• Outperforms grey-box GPT-3 probability baseline: AUC-PR 72.96 → 83.21 (entropy)
• Substantially outperforms random baseline: AUC-PR 27.04

Passage-level ranking (Pearson correlation with human passage-level judgments):

• SelfCheckGPT-Prompt: 0.78 correlation
• Grey-box LLaMA baseline: 0.35 correlation
• Shows large peak in passage-level scores at "total hallucination" threshold where entire passages are fabricated

Ablations show:

• Prompt-based scoring is computationally expensive but achieves best performance
• NLI variant provides strong performance with better computational trade-off
• Performance generally improves with more samples (N=20 used in main experiments)
• Method works across different black-box LLMs (GPT-3, ChatGPT)

Notes

Strengths:

• Practical and immediately deployable: works with any black-box LLM via API without requiring model internals or external resources
• Comprehensive evaluation across five different consistency metrics, each revealing different aspects of hallucination detection
• Strong empirical results substantially above grey-box baselines despite using no privileged access to model probabilities
• Well-motivated: consistency-based reasoning directly exploits the property that hallucinated facts diverge stochastically while factual statements are stable

Limitations:

• Limited evaluation scope: only tested on WikiBio biographies; generalization to other domains (news, scientific claims, etc.) unclear
• Computationally expensive for the best-performing variant (Prompt), which requires multiple LLM calls
• Only evaluates sentence-level factuality; does not address fine-grained atomic fact verification or relation-level accuracy
• Evaluation dataset is relatively small (238 passages)
• Assumes the black-box LLM can be queried multiple times without cost or rate limits, which may not hold in all deployment scenarios

Open questions:

• How does performance scale to claims in other domains beyond biography (news, scientific, medical)?
• Can the method be made more computationally efficient while maintaining high accuracy?
• How robust is the approach to adversarial prompts designed to make hallucinated facts consistent across samples?