Contributions
- PubHealth dataset: The first large-scale dataset for explainable fact-checking in the public health domain, with 11,832 claims and journalist-written explanations covering health topics (diseases, vaccines, healthcare policy, health behaviors).
- Explanation generation framework: A joint extractive-abstractive summarization approach that generates veracity labels and coherent explanations simultaneously, outperforming extractive-only baselines.
- Formal explanation quality metrics: Three coherence properties for evaluating explanation quality—strong global coherence (all sentences entail the claim), weak global coherence (no sentences contradict it), and local coherence (no internal contradictions)—evaluated both automatically and through human annotation.
- Domain-specific improvements: Evidence that SciBERT and BioBERT significantly outperform standard BERT when fine-tuned on in-domain health data, supporting the use of specialized language models for domain-specific fact-checking.
Method
The paper constructs the PubHealth dataset by scraping 39,301 claims from five fact-checking websites and health news sources, standardizing veracity labels to four categories: true, false, mixture, and unproven. After filtering for health-related content using a curated lexicon of ~7,000 health and medical terms, the dataset contains 11,832 claims with accompanying journalist-written explanations.
Veracity Prediction: Uses Sentence-BERT (S-BERT) to rank evidence sentences by relevance to the claim, then feeds the top-k sentences into a BERT classifier to predict the veracity label. Four model variants are tested: standard BERT (uncased), SciBERT (pre-trained on scientific abstracts), and two versions of BioBERT (v1.0 and v1.1, pre-trained on biomedical literature).
Explanation Generation: Employs a joint extractive-abstractive summarization approach. Evidence sentences are selected and ranked by S-BERT, then fed to a BERT-based abstractive summarization model (fine-tuned on the Dailymail/CNN dataset) to generate concise, human-readable explanations. The ExplainerFC and ExplainerFC-Expert models compare extractive summaries to abstractive ones.
Explanation Quality Evaluation: Defines three coherence properties: (1) strong global coherence—all sentences in the explanation must entail the veracity label; (2) weak global coherence—no sentence should contradict it; (3) local coherence—sentences within the explanation must not contradict each other. Evaluates quality using ROUGE-1/2/L metrics and human annotation by five judges assessing agreement and relevance.
Results
Veracity Prediction (Table 4):
- SciBERT achieves the highest macro F1 (73.93%) and accuracy (65.18%), with precision 77.39% and recall 54.77%, outperforming standard BERT (66.02% F1) and BioBERT variants.
- SciBERT significantly improves over the random BERT baseline (38.97% F1), confirming the value of domain-specific pre-training.
Explanation Generation (Table 5):
- ExplainerFC-Expert achieves ROUGE-1 F1 of 32.30, significantly outperforming the ORACLE upper bound (14.89), suggesting the model generates plausible explanations beyond simple extraction.
- ExplainerFC achieves ROUGE-1 of 31.42, indicating that joint prediction and explanation generation improves over pipeline approaches.
Coherence Properties (Table 6, Human Evaluation):
- For gold explanations: strong global coherence 76.80%, weak global coherence 98.40%, local coherence 65.60% across human judges (κ=0.24 inter-annotator agreement for strong GC).
- For ExplainerFC-Expert-generated explanations: strong GC 60.4%, weak GC 76.80%, local coherence 59.30%, demonstrating that model-generated explanations are less coherent than gold standards but show reasonable quality.
- Computational evaluation using NLI models (MNLI) provides a reasonable proxy for human judgments on weak global coherence (0.87 correlation) but is weaker for strong global coherence.
Readability: Claims in PubHealth have a mean Flesch-Kincaid reading level of 59.1 (college level), significantly higher than political fact-checking datasets (LIAR: 63.9, FEVER: 8.9), reflecting the technical nature of health claims.
Notes
Strengths:
- Addresses a genuine gap: fact-checking in specialized domains (health) where general-purpose systems fail due to lack of domain expertise.
- Well-motivated coherence properties capture real qualities of good explanations (no contradictions, consistency with label).
- Thoroughly documented dataset construction: clear handling of duplicate claims, label standardization, and health-term filtering.
- Strong empirical validation that domain-specific models (SciBERT, BioBERT) outperform generic BERT, with principled evaluation of explanation quality via both automatic metrics and human judges.
- Readable writing with good examples (Table 1) showing the variety of health claims (unproven COVID-19 harms, false childhood vaccine claims, mixture claims with both true and false elements).
Weaknesses:
- Modest inter-annotator agreement on strong global coherence (κ=0.24) suggests the definition may be too strict or require clearer guidelines; weak global coherence (κ=0.39) is more reliable.
- Explanation generation model (ExplainerFC-Expert) fine-tuned on non-health data (Dailymail/CNN) and then evaluated on health claims—domain transfer not explored; retraining on health-specific summaries might improve coherence.
- Limited analysis of failure modes: what types of health claims does the model struggle with? Do biomedical vs. policy health claims differ in difficulty?
- PubHealth is smaller than political fact-checking benchmarks (LIAR: 12,791, FEVER: 145,449), limiting generalization studies.
Future directions:
- Extending to other specialized domains (law, finance, science) where expertise is similarly required.
- Exploring whether explanation coherence properties correlate with human trust in fact-checking verdicts.
- Cross-lingual health fact-checking, especially for low-resource languages affected by health misinformation.
- Integration with real-time health news to support journalists and fact-checkers during public health crises.