High level of concordance across different news domain quality rating sets

Contributions

• First direct comparison of agreement across six different expert rating sources (NewsGuard, Media Bias/Fact-Check, Ad Fontes Media, Lässer et al., independent professional fact-checkers, and Iffy index of unreliable sources)
• Demonstrates substantial concordance in relative quality judgments across different rating systems, validating the use of domain-level quality measures
• Proposes an ensemble methodology combining imputation and PCA to generate aggregate ratings that reduce bias from any single source
• Releases the most comprehensive domain quality rating dataset to date (11,520 domains, covering high-quality outlets like Reuters to low-quality outlets like Natural News)

Method

The authors collected quality ratings for news domains from six diverse expert sources using different scoring systems and evaluation criteria:

• NewsGuard: 8,178 rated domains on nine political criteria; ratings normalized to 0–1 scale with threshold 60 indicating basic credibility standards
• Media Bias/Fact-Check (MBFC): 3,216 rated domains on factualness and bias dimensions (accuracy/unbiasedness composite); two composite ratings created (mean and minimum)
• Ad Fontes Media (AFM): 283 rated domains on reliability and bias dimensions; two composites created (mean and minimum)
• Professional fact-checkers: 60 domains rated by eight professional fact-checkers from the Poynter International Fact-Check Network
• Lässer et al. (2022): 4,767 domains compiled from multiple fact-checking organizations; rated on accuracy, reliability, and transparency
• Iffy index: 1,388 unreliable/fake domains; converted to binary ratings (unreliable = 0, all other domains = 1)

These sources covered only partially overlapping sets of domains (60–4,767 per source; 12 example domains in common across all). To handle missing values and enable correlation analysis, the authors normalized all ratings to 0–1 scale and applied multiple imputation using gradient-boosted decision trees (LightGBM). They computed pairwise correlations for the imputed dataset and performed principal component analysis (PCA) on the imputed ratings from five sources (excluding NewsGuard per its data-sharing requirements). The first principal component (PC1) served as an aggregate rating capturing shared signal.

Results

Despite different rating dimensions and scoring systems, correlations across sources were substantial: Pearson correlations ranged from 0.32 to 0.86 (Spearman's ρ ranged 0.32–0.90). Three sources (MBFC, AFM, Lässer) that rated domains on multiple dimensions showed even stronger correlations on individual dimensions. The distribution of ratings varied considerably across sources (e.g., fact-checkers' ratings right-skewed toward low quality; AFM's left-skewed), yet pairwise correlations remained high.

The first principal component (PC1) from the imputed data explained 68.21% of variance and correlated highly with all imputed source ratings (r ≥ 0.43, min r = 0.67 excluding Iffy binary). PC1 ratings spanned 0 (lowest quality, e.g., Natural News) to 1 (highest quality, e.g., Reuters, New York Times), providing a nuanced alternative to binary classifications. The aggregate PC1 ratings were relatively robust to variations in the number of sources included (PC1 correlations remained high across different subsets).

The resulting 11,520-domain dataset is the most comprehensive domain rating coverage to date, far exceeding individual sources' coverage. The aggregate ratings enable finer-grained analysis: domain-level quality measures can identify content streams to flag for fact-checking, inform digital literacy interventions, and help researchers track whether domains are shifting their editorial standards over time.

Notes

A foundational contribution to measurement standardization in misinformation research. The paper directly addresses a critical measurement problem: different researchers using different rating sets may reach incompatible conclusions about misinformation prevalence. The ensemble approach and released dataset should become a standard reference. One limitation: domain-level ratings assume uniform content quality within a domain, though news outlets often publish both high-quality investigative reporting and opinion pieces. The paper acknowledges this and notes that individual content fact-checking remains necessary for precision despite the practical value of scalable domain measures.