Mathematical reasoning encompasses language model performance on math problems—from simple arithmetic to multi-step word problems, algebraic manipulation, and abstract reasoning:
Challenge categories:
- Arithmetic: Basic calculations and numerical operations
- Multi-step reasoning: Decomposing complex problems into solvable components
- Symbolic manipulation: Algebraic and calculus operations
- Temporal reasoning: Tracking variable states across reasoning steps
Factuality challenges in math:
- Models can produce well-formatted solutions with incorrect intermediate calculations
- Floating-point precision differences between predicted and computed results
- Models sometimes apply wrong mathematical rules or operators
- Self-checking struggles due to models reasoning about results without verifying computations
Verification approaches:
- Execution-based: Convert mathematical reasoning into executable code and verify results
- Symbolic: Use formal math systems (Lean, Mathematica) for exact verification
- Numeric: Compare computed results against expected outputs
Key papers in this wiki
- 2023-chern-factool — Evaluates math problem solving by converting each arithmetic operation to executable Python code; identifies round-off errors and reasoning errors as main failure modes; achieves 98.24% claim-level F1 with tool-augmented verification