A comprehensive dataset of quality ratings for 11,520 news domains, combining ratings from six different expert sources using an ensemble "wisdom of experts" approach with imputation and principal component analysis.
Overview
The dataset addresses a fundamental measurement problem in misinformation research: different media organizations and fact-checkers use different rating systems, making it unclear whether they agree on source quality. Lin et al. (2023) collected ratings from six diverse sources:
- NewsGuard: 8,178 rated domains on nine political criteria (credibility, transparency, etc.); 0–1 normalized scale
- Media Bias/Fact-Check (MBFC): 3,216 rated domains on factualness and bias dimensions
- Ad Fontes Media (AFM): 283 rated domains on reliability and bias dimensions
- Professional fact-checkers: 60 domains rated by eight professional fact-checkers from Poynter International Fact-Check Network
- Lässer et al. (2022): 4,767 domains compiled from multiple fact-checking organizations
- Iffy index: 1,388 unreliable/fake domains (binary ratings)
Methodology
The authors normalized all ratings to 0–1 scale and applied multiple imputation using LightGBM to handle missing values (since each source rated only a partially overlapping set of domains). They then performed principal component analysis (PCA) on the imputed ratings, finding that the first principal component (PC1) explained 68.21% of variance and correlated highly with all source ratings (r ≥ 0.43, min r = 0.67 excluding Iffy binary).
Key findings
Despite using different rating systems and evaluation criteria, the six expert sources correlated highly (Pearson r = 0.32–0.86; Spearman's ρ = 0.32–0.90). The PC1 aggregate rating serves as a domain quality measure spanning from 0 (lowest quality, e.g., Natural News) to 1 (highest quality, e.g., Reuters, New York Times).
Coverage
The dataset includes 11,520 domains, the most comprehensive coverage to date, spanning:
- High-quality outlets: Reuters, Associated Press, AP (PC1 ≈ 1.00)
- Medium-quality outlets: Daily Caller, MSNBC (PC1 ≈ 0.47–0.59)
- Low-quality outlets: Natural News, Infowars, Before It's News (PC1 ≈ 0.00–0.05)
Use cases
- Misinformation exposure research: Estimate population exposure to low-quality news by linking domain ratings to survey-based or platform-based content consumption
- Platform interventions: Prioritize fact-checking and content flagging for low-quality outlets
- Misinformation detection: Use domain quality as a feature in machine learning models
- Interventions research: Study whether domain-level source credibility labels affect sharing decisions
- Trend analysis: Track whether outlets' editorial standards change over time by recomputing aggregate ratings
Related datasets
- 2019-pennycook-crowdsourced-news-source-quality — Crowdsourced outlet trustworthiness ratings (small scale, laypeople ratings)
- 2022-lasser-misinformation-domains — Lässer et al. (2022) misinformation domains dataset (4,767 domains)
- newsguard-ratings — NewsGuard's proprietary database (8,178 domains with detailed credibility scores)
Access
The data is released publicly at https://github.com/hauselin/domain-quality-ratings with open-source code for reproduction and extension. Code for imputation and PCA is available; researchers can update ratings regularly by rerunning the pipeline with new data from the source organizations.