Code generation refers to automated synthesis of executable code, typically from natural language prompts. This spans:
Task variants:
- Completion: Predicting missing code given context
- Synthesis from specification: Generating code from natural language description of functionality
- Program repair: Fixing buggy code
- Transpilation: Converting between languages
Quality dimensions:
- Correctness: Code executes without errors and produces correct outputs
- Efficiency: Generated code runs fast and uses reasonable memory
- Readability: Code follows conventions and is maintainable
- Completeness: Code fully implements the specification
Factuality in code generation:
- Generated code often appears executable but contains logical errors
- Self-checking models cannot reliably audit their own code
- Execution-based verification against test cases provides objective correctness assessment
Key papers in this wiki
- 2023-chern-factool — Applies factuality detection to code generation; executes generated Python snippets against test cases, achieving 98.97% claim-level F1 with GPT-4 tool-augmented verification vs. 68.90% for unaided self-checking