Researchers are proposing a new methodology to measure how coding LLM perplexity scales with codebase size, using the Lean programming language to evaluate long-term software security and predictability.
Key Points
- The study aims to determine if formal languages like Lean possess superior scaling exponents compared to popular, less restrictive languages like Python or JavaScript.
- Researchers will measure perplexity across increasing context windows to identify which programming languages become easier for LLMs to understand and maintain as codebases grow.
- The methodology includes injecting artificial bugs to test "inverse scaling," where higher model surprise indicates a better grasp of the codebase's underlying logic.
- By analyzing the "predictability" of code, the project seeks to identify whether formal verification can mitigate the security risks inherent in AI-generated software.
- The study suggests that even if Lean currently lacks large training datasets, its potential for better scaling could justify massive investments in rewriting existing systems for improved correctness.