Google has launched Android Bench 2.0, an updated evaluation tool designed to test the performance of large language models on complex, multi-day Android software development tasks and projects.
Key Points
- Android Bench 2.0 introduces long-horizon tasks that simulate real-world engineering challenges like building apps from scratch or upgrading complex dependencies.
- The benchmark replaces binary pass-fail grading with continuous scoring to provide more nuanced data on how models handle incomplete development tasks.
- GPT-6 Astra currently leads the leaderboard with a 28% pass rate, significantly outperforming Gemini 3.8 Flash, which recorded an 8% pass rate.
- The testing suite includes various industry-leading models, such as Claude Fable 5.1, GPT-5.6 Sol, and Claude Opus 5.