Beyond Fact Checking: Modelling Information Change in Scientific Communication

Isabelle Augenstein

In brief

Augenstein argues that automated fact-checking is often binary (true/false) but scientific communication involves nuanced information change: scientists publish findings, journalists translate for public understanding, and laypeople discuss on social media. Rather than detect lies, she proposes measuring how scientific claims change across communicators using the SPICED dataset, which evaluates information content similarity beyond surface-level textual similarity. Models trained on information-change patterns improve downstream fact-checking evidence retrieval, and analysis reveals that general news outlets distort scientific findings 0.4 points more than press releases.

Contributions

Information change beyond binary fact-checking: Moves fact-checking discourse from true/false labels to modeling degrees of change in claim accuracy, specificity, and strength as information flows through communication chains.
SPICED dataset: Curated 5,600+ news-paper and tweet-paper pairs annotated by domain experts (computer science, medicine, biology, psychology) with five-point information-matching scores (1 = completely different finding; 5 = completely identical).
Two-stage exaggeration detection: Develops semi-supervised method using pattern-exploiting training (PET) to detect both claim-strength changes (correlational → causal) and exaggeration in single texts and paired texts (abstract vs. press release).
Science communication analysis: Quantifies that general news outlets change information content ~0.4 points more than press releases; shows journalists tend to downplay uncertainty in abstracts but exaggerate strength of findings; demonstrates that limitations sections are most frequently exaggerated (suppressing caveats).
Evidence retrieval improvements: Shows models trained on information-change similarity outperform semantic-textual-similarity baselines on downstream scientific fact-checking datasets (CoVER, CoVER-Facts), improving evidence ranking from paraphrasing/NLI transfer.

Method

Exaggeration detection (stage 1): Given abstract and press release, predict claim-strength relationship: no relation → correlational → conditional/causal → direct causal. Uses 4 manually-crafted patterns with verbalizers (e.g., "scientists claim X, reporters claim Y, and reporters claim [MASK]"). Ensemble of RoBERTa-base models trained on 100–4,500 labeled examples with pattern-exploiting training (semi-supervised) transfers soft labels from large language models to unlabeled data via KL divergence distillation. Multi-task learning joint-trains claim-strength and exaggeration-detection tasks.

Information change modeling (stage 2): Proposes general 1–5 similarity score for scientific findings in two sentences, measured by domain experts. Stratified sampling from ~1M candidate pairs (news-paper, tweet-paper) using semantic-textual-similarity model as proxy; final dataset: 2,000–2,400 news-paper pairs + 1,200 tweet-paper pairs across 4 domains (computer science, medicine, biology, psychology).

Fine-tunes sentence transformers (MinilM, MPNet, SciBERT, Specter) on SPICED; evaluates zero-shot transfer from paraphrase detection and natural-language inference (both hurt performance, showing task novelty). Tests downstream scientific fact-checking on CoVER and CoVER-Facts datasets: SPICED-fine-tuned models substantially outperform BM25 lexical baseline and base semantic-textual-similarity models.

Results

Exaggeration detection: 10 F1 points improvement over supervised learning with small labeled data (100 examples) using PET; multi-task learning with claim-strength prediction adds further gains; multi-task with 200 samples reaches performance of supervised learning with 4,500 samples.
Information-change similarity: Semantic-textual-similarity models treat additional contextual information (author names, institution affiliation, etc.) as similarity-reducing; SPICED-trained models weight scientific content more heavily; average edit distance in SPICED training pairs ~twice as high as in STS pairs (19.8% vs 10.5%).
News vs. press release: Regression analysis on information-matching scores reveals general news outlets significantly differ from press releases and science/tech news (difference of 0.4 in information score), with general outlets exaggerating more.
Section analysis: Findings in abstracts are often downplayed (certainty markers weakened); limitations sections are exaggerated (caveats removed), which is problematic because limitations contain critical scope boundaries and replication requirements.
Downstream fact-checking: SPICED-fine-tuned models improve evidence retrieval on CoVER and CoVER-Facts scientific fact-checking benchmarks (Pearson correlation 0.62–0.68 range with 3–5% absolute improvements over baseline).

Notes

Augenstein emphasizes that most prior work on scientific fact-checking focuses on abstracts alone (easier to obtain than full papers) but misses critical limitations sections; demonstrates that analyzing abstract-only creates blind spot for key distortions. Proposes information-change framework as foundation for broader research agenda: selective reporting analysis (what drives journalist selection?), certainty modeling (degree of hedging/confidence change), and generative applications (faithful scientific summarization systems). Acknowledges manual pattern engineering for exaggeration detection is brittle and sensitive to formulation; proposes ensembling multiple patterns as stabilization approach. Speaker notes this is preliminary work; limitation is relatively small annotated dataset and domain-expert annotation cost; future direction includes cross-lingual generalization and multi-hop information change (finding → press release → news article → social media tweet chain).