Contributions
- Demonstrates GPT-4 achieves better performance than prior approaches on both hard (binary) and soft (probabilistic) misinformation classification tasks in multiple languages
- Proposes uncertainty quantification methods that allow GPT-4 to refuse classification on examples it cannot reliably evaluate, improving overall performance
- Provides empirical analysis of generalization, finding GPT-4 and RoBERTa have substantially different failure modes
- Releases LIAR-New, a bilingual (English/French) misinformation dataset with novel "Possibility" labels and examples beyond GPT-4's training cutoff
- Explores practical factors: prompting strategies, temperature, model versioning, explainability, and web retrieval
Method
The paper evaluates GPT-4 using several prompting approaches:
Main prompting: Asks GPT-4 to rate a statement's truthfulness on a 0–100 scale without explanations. This "GPT-4 Score" output can be converted to binary classification by choosing a threshold (71 on validation data).
Uncertainty-enabled variant: GPT-4 can respond with 0.5 (uncertain), indicating it lacks sufficient context to judge the statement. Statements it marks 0.5 are excluded from evaluation, significantly improving F1 performance.
Comparison models: RoBERTa-large (6-way classification trained on LIAR), smaller language models, and other baselines. The authors also test different GPT-4 versions (0314, 0613, August 2023) to assess robustness to version changes.
Datasets: LIAR (6-point veracity labels, ~13K examples), CT-FAN-22 (4-way labels), and the new LIAR-New (scraped PolitiFact fact-checks from October 2021 onward, with binary Possible/False labels and Possibility categories).
Results
LIAR binary classification: GPT-4 Score Optimized achieves 68.2% accuracy and 68.1% F1, outperforming RoBERTa-L Binary (63.5% accuracy, 62.1% F1). When GPT-4 is allowed to refuse uncertain examples (LIAR-New, 49 and 51 predictions removed), performance improves to 81.1% F1 vs. 80.4% for RoBERTa.
Transfer to new data (CT-FAN-22): GPT-4 maintains 67.8% accuracy on English, outperforming SOTA. Performance is lower in German (57.6% vs. 42.7% SOTA) but still strong.
Multilingual results: French-translated LIAR-New shows lower performance (70.1% vs. 67.8% in English), but GPT-4 still outperforms RoBERTa. The authors show prompt engineering impacts which examples GPT-4 refuses.
Impossibility detection: The Uncertainty-Enabled variant successfully identifies over two-thirds of Impossible examples, enabling graceful failure and improved overall performance.
Error analysis: GPT-4 and RoBERTa make substantially different errors; embedding analysis shows statistically significant differences (p=0.0005) between cases where each model fails.
Notes
The paper makes a strong empirical case that GPT-4 improves on prior approaches, but explicitly notes that overall performance remains below what would be needed for many real-world applications. The focus on uncertainty quantification and graceful failure is practical and well-motivated. The LIAR-New dataset's inclusion of examples beyond the knowledge cutoff and multilingual versions is a valuable contribution, though the French performance gap suggests cultural or linguistic challenges worth deeper investigation. Web retrieval experiments are preliminary but promising for scaling beyond the training cutoff.