Calibrating Small Language Models for Claim Check-Worthiness Detection

Authors: Pratuat Amatya, Venktesh V, Vinay Setty

Venue: arXiv cs.CL, 2026

Links: arXiv

TL;DR

Claim check-worthiness detection is computationally expensive when using large language models, but essential for scaling fact-checking pipelines. This paper proposes NN-PPI (Nearest Neighbor Prediction-Powered Inference), a post-hoc calibration technique that improves small LM predictions using a small labeled calibration set and k-nearest neighbor residual correction, achieving 12–33.80% weighted F1 gains across model scales.

Contributions

• Proposes NN-PPI, a calibration method that extends Prediction-Powered Inference (PPI) to pointwise claim check-worthiness scoring.
• Demonstrates that small transformer-based LMs (270M–4B parameters) can match or exceed large LM performance on check-worthiness detection when calibrated.
• Shows residual calibration is complementary to supervised fine-tuning, enabling production-deployed models to maintain cost efficiency while recovering accuracy.
• Provides confidence intervals alongside calibrated predictions, quantifying uncertainty in model outputs.

Method

NN-PPI is a post-hoc calibration layer applied at inference time to any frozen LM without modifying model weights. For each unlabeled test claim:

1. Compute the LM's raw check-worthiness prediction score \(\hat{c}_i \in [0,1]\).
2. Retrieve the \(k\) labeled calibration examples most semantically similar to the test claim (via embedding-based k-nearest neighbors).
3. Compute residuals \(r_j = Y_j - \hat{c}_j\) for each neighbor, where \(Y_j\) is the true label.
4. Estimate a residual correction \(\bar{r} = \frac{1}{k}\sum_{j \in S_i} r_j\) and apply it: $\(\theta_i = \hat{c}_i + \bar{r}\)$

The final calibrated score \(\theta_i\) is thresholded at \(\epsilon=0.5\) to yield a binary decision. The method also produces a \((1-\alpha)\) confidence interval using residual variance.

Results

Claim worthiness detection F1 scores (weighted F1, across ClaimBuster and CLEF 2024 datasets):

• Small LMs (270M–4B): Largest gains. Gemma 3 270M baseline: 0.114 → calibrated 0.698–0.721 (12–33.80% gain on ClaimBuster). Gemma 3 4B: +33.80% on ClaimBuster, +20% on CLEF.
• Mid-scale LMs: Gemma 3 1B gains +15%; Claude Opus 4.6 and GPT-5.2 gain ~5% or less.
• Comparison: NN-PPI yields 11% gain over fine-tuned XLM-RoBERTa-Large on CLEF 2024 and 7.03% over plain KNN, while maintaining significantly lower inference cost.
• Neighbor size sensitivity: Optimal calibration set size is 1,500–2,000 labeled examples with \(k \in \{3,5\}\); larger \(k\) introduces noise.

Key qualitative finding: NN-PPI successfully corrects systematic over-triggered claims (false positives) and recovers missed high-priority claims (false negatives), including complex rhetorical claims and factual statements misclassified by base models.

Connections

• Related to The Perils & Promises of Fact-checking with Large Language Models via shared concern with LLM reliability in fact-checking.
• Uses techniques from FEVER: A Large-Scale Dataset for Fact Extraction and VERification and LIAR — standard fact-checking benchmarks.
• Builds on sentence embeddings (Sentence-BERT) for semantic similarity-based nearest-neighbor retrieval.
• Applies Prediction-Powered Inference (PPI) methodology to pointwise classification in information verification, extending approaches from population-level calibration to per-instance predictions.

Notes

Strengths: The work addresses a genuine production constraint — small model efficiency vs. accuracy — with a principled, post-hoc method that requires only a small labeled calibration set. Extensive ablations (neighbor size, temperature, calibration set size) and failure-mode analysis strengthen confidence in the approach. The residual correction is elegant and computationally lightweight.

Limitations: Semantic similarity via embeddings is a proxy for distributional similarity; the method may fail when test and calibration sets diverge in topic or claim type. Requires human-annotated calibration data, though reuse across topics is explored. Confidence intervals under-cover for Class 1 (check-worthy) claims in some settings.

Future work: Adaptive calibration sets that automatically update as claim worthiness definitions evolve; exploration of distributional similarity measures beyond embeddings; application to other classification tasks in information verification.