Interpretive Variation and Discourse Structure in Detecting Enthymemes in Controversial Political Discourse

Authors: Martial Pastor, Nelleke Oostdijk Affiliation: Centre for Language Studies, Radboud University, Nijmegen Venue: arXiv cs.CL, 2026 — arXiv:2610.30987

TL;DR

Enthymemes (arguments with unstated premises or conclusions) are a persuasion technique in social media that readers may or may not recognize. This paper develops a DeBERTa-based detector on 1,482 tweets about vaccination and immigration, showing that macro-F1 reaches ~0.67 on hard labels but plateaus due to genuine human disagreement. The key contribution is an annotator-conditioned per-vote metric that treats disagreement as signal rather than noise, enabling models to be evaluated on how well they capture interpretive variation across readers.

Contributions

• An evaluation methodology for enthymeme detection under human label disagreement, combining work on learning with disagreement and perspectives that model multiple valid judgments.
• Betabin, a model that predicts a vote-count distribution rather than a binary label, treating each reader's vote as a sample from the annotators' collective belief.
• Evidence that macro-F1 plateaus around 0.67 regardless of data volume, feature engineering, or label regime—a property of the task, not the models.
• A discourse-structural account of where models succeed and fail: discourse features like argumentative satellites (CAUSE, EXPLANATION, CONDITION, EVALUATION, CONTRAST) and tree depth distinguish hard vs. easy cases.
• A per-vote consistency metric conditioned on annotator behavior, revealing that model errors concentrate on intrinsically contested tweets where disagreement reflects valid interpretive variation.

Method

The paper frames enthymeme detection as a binary task (enthymeme vs. none) on 1,482 English tweets on vaccination and immigration, drawn from the Flaccavento et al. (2025) trope dataset. Each tweet is annotated by five annotators; the task asks whether an implicit premise or conclusion exists that conveys a controversial stance.

Modeling pipeline:

The authors test three approaches: 1. Classical text-only methods (TF-IDF with logistic regression, linear SVM) 2. Transformer baselines (fine-tuned DeBERTa-v3) 3. Feature injection architectures that combine text with external features (RST discourse structure and trope annotations)

RST features encode discourse relations (ELABORATION, JOINT, CAUSE, etc.) and tree statistics (spine ratio, embedded nodes, satellite count). Trope features mark narrative or persuasion structures. Both are encoded as binary indicators or count-based ratios for a 24-dimensional feature space.

Handling disagreement: Hard training treats the majority vote as the gold label. Soft training uses a two-way focal cross-entropy loss on vote fraction, treating it as a probability. Betabin (Section 4.5) models the vote count itself via a Beta distribution over probabilities, allowing the model to output a confidence in both the leaning and the vote count, capturing both disagreement and interpretive variation.

Evaluation: Standard hard-label F1 reaches 0.67 with text-only soft training and plateaus there. The authors propose multiple soft metrics (Manhattan distance, soft cross-entropy, consensus separation, per-vote consistency) to measure how well model predictions align with the distribution of votes rather than a collapsed majority label.

Results

Hard-label performance: Macro-F1 plateaus at ~0.67 across all transformer configurations, regardless of training pool size (300–1,334 examples) or feature engineering. Classical methods achieve ~0.62 with combined features. Once the model learns to align with consensus items, adding more data doesn't improve the hard-label score.

Soft-label and distributional metrics: Betabin and soft-trained transformers reach 0.675±0.02 macro-F1, but more instructively, soft metrics reveal that: - The per-vote consistency metric (Spearman correlation of model surprise with annotator deviation) reaches 0.239 for Betabin, indicating that model errors concentrate where annotators disagree. - Calibration metrics (Manhattan distance to vote fractions, soft-ECE) show Betabin substantially improves over the text-only model (.282 vs. .172 Manhattan distance). - On consensus items (0/5 or 5/5 votes), all models agree; on contested items (2–4 votes), Betabin learns to sit near 0.5 rather than hedging everywhere.

Discourse analysis: RST dimensions distinguish easy from hard cases: - Easy nones (512 cases): flat trees, mostly nuclei in sequence, few argumentative satellites, low spine depth. - Easy enthymemes (244 cases): argumentative satellites (CAUSE, EXPLANATION, CONDITION) near the root, multi-level argument structure. - Contested enthymemes (15 cases, where annotators diverge 2/5 or 3/5): attribute argumentative relations to exterior material (bridging inferences, irony); high frequency of ATTRIBUTION-EVALUATION co-occurrence.

Injected RST features improve calibration but not hard-label F1, suggesting discourse structure is a stable signal about which texts are inherently ambiguous, not a tool to push classification boundaries.

Connections

• A Survey of Fake News: Fundamental Theories, Detection Methods, and Opportunities — surveys the field; implicit arguments and rhetorical framing are mechanisms of misinformation spread.
• Implicit argument detection — related task of recovering missing argument components; this work differs in focus on disagreement and social-media toxicity.
• Stance Detection — shares the NLP pipeline and political-discourse corpus structure; disagreement is treated similarly.
• Annotator disagreement — relates to the perspectivist turn in NLP; this paper contributes metrics tailored to argument-detection settings.
• Rhetoric — enthymemes are a classical persuasion device; the paper brings modern corpus linguistics and ML to an old rhetorical problem.

Notes

Strengths: - The paper takes annotator disagreement seriously as a genuine feature of the task, not noise to suppress. This is crucial for political-discourse understanding where people legitimately disagree on whether an implicit stance is present. - The per-vote consistency metric is a clever operationalization: it asks whether model errors cluster where readers divide, which is exactly what you'd hope for if you're building content-moderation classifiers. A model that is wrong uniformly is worse than one that is uncertain on genuinely contested items. - Discourse-structural analysis grounds the flat performance ceiling in interpretability: easy cases have clear tree structure; contested cases have high satellite counts and bridging inferences that no surface feature captures.

Weaknesses: - Dataset is small (1,482 tweets) and topically narrow (vaccination and immigration only). Generalization to other controversial topics and genres (news comments, forum posts) is unclear. - Only single-language (English) and single-parser (IsaNLP RST) evaluation. Robustness to parsing errors and other languages is unexplored. - The paper claims the 0.67 ceiling reflects task properties, but alternative architectures (larger models, multimodal, graph-based) are untested.

For practitioners: This paper is essential for anyone building classifiers on political or controversial discourse where disagreement is real, not error. It shows that standard "choose the majority label" evaluation erases the actual signal in the data. If you're moderating content or training detection systems, the per-vote consistency metric should be part of your evaluation suite. The discourse-structural analysis also suggests that rhetorical parsers can serve as diagnostic tools, even if they don't improve F1 directly.