Contributions
- TabFact dataset: 118k human-annotated statements paired with Wikipedia tables (16k tables); statements labeled as ENTAILED or REFUTED with high inter-annotator agreement (κ = 0.75).
- Two reasoning models: Table-BERT linearizes tables and uses pre-trained language models; LPA parses statements into executable programs and runs them on tables.
- Empirical analysis: Demonstrates the gap between automated systems (65.3% LPA) and human performance (92.1%); identifies consistent weaknesses in both models' handling of complex reasoning and structured operations.
- Dataset challenges: Reveals that table-based fact verification requires handling both linguistic inference (paraphrasing, entailment) and symbolic reasoning (arithmetic, aggregation, comparison).
Method
Table-BERT: Treats table verification as a two-sequence binary classification problem. The approach linearizes tables into sequences using two methods: (i) concatenation of [SEP]-separated cells with row/column position counters; (ii) natural language templating (e.g., "row one's game is 51..."). The linearized table is concatenated with the statement and a [CLS] token, then fed to a 12-layer BERT encoder to produce the final prediction.
Latent Program Algorithm (LPA): Frames verification as program synthesis. The method searches over plausible programs (API calls and operations: count, average, superlative, filter, etc.) that could be executed on tables. It uses a LSTM encoder-decoder with weak supervision to generate candidate programs, then a discriminator (transformer-based encoder) learns to rank programs by consistency with statement semantics. Top-ranked programs are executed against the table to obtain a binary verdict.
Results
| Model | Validation | Test | Test (simple) | Test (complex) | Small Test |
|---|---|---|---|---|---|
| BERT classifier (no table) | 50.9 | 50.5 | 51.0 | 50.1 | 50.4 |
| Table-BERT (horizontal) | 66.1 | 65.1 | 79.1 | 58.2 | 68.1 |
| LPA-Ranking w/ Discriminator | 65.2 | 65.3 | 78.4 | 58.5 | 68.9 |
| Human Performance | — | — | — | — | 92.1 |
Both Table-BERT and LPA achieve similar accuracy (~65%) on the test set. LPA-Ranking performs best overall (65.3%) but both methods remain significantly below human performance. Table-BERT struggles with complex reasoning and exhibits poor consistency on hard cases; LPA has issues with spurious programs and entity linking.
Notes
The large gap between human and model performance (92.1% vs. 65.3%) highlights fundamental challenges in combining linguistic and symbolic reasoning. The paper's error analysis reveals that models struggle with: (1) spurious programs in LPA (wrong program returns correct label by accident); (2) poor consistency in Table-BERT (simple cases misclassified, hard cases correct); (3) entity linking errors. The dataset and code release enable future work on bridging this gap, and the two complementary approaches (neural-linguistic vs. program-synthesis) suggest that hybrid methods may be necessary. The paper identifies an important and under-explored research direction: extending fact-checking beyond text to structured evidence ubiquitous in real-world applications (Wikipedia infoboxes, medical tables, financial records).