If you've been believing the hype that AI is about to replace every developer by Diwali, Google has some sobering news. This week the Android team quietly shipped Android Bench 2.0, a new version of its benchmark for judging how well AI models handle real app development — and the results are a reality check. The best model on the board, OpenAI's GPT-6 Astra, fully completes just 28% of the tasks. That's the winner. Let that sink in.
01.What is Android Bench 2.0?
The first version of Android Bench, launched earlier this year, focused on small stuff — bug fixes, minor feature tweaks, incremental changes to an existing codebase. Useful, but not how professional developers actually spend their week. Version 2.0, announced on the official Android Developers Blog by Matthew McCullough, VP of Product Management for Android Developer, introduces what Google calls long-horizon tasks: jobs of genuine complexity that take a human engineer multiple days, or even a full week, to finish.
We're talking about upgrading dependencies across a whole project, adding major new features, building an Android app from scratch, or converting a cross-platform app (think Flutter or React Native) fully to native Android. Thirty of these tasks across four engineering streams, run inside containerized virtual device environments. This is the closest thing to a real dev sprint that an AI benchmark has attempted.
02.From 91% to 28% — what changed?
Here's the number that made me sit up. On the original Android Bench tasks, top models scored pass rates in the 90% range. On the new long-horizon tasks, the highest pass rate is around 28%. Nothing about the models got worse overnight — the exam just got honest. It's the difference between testing a student on one-line answers and asking them to write a full dissertation.
The current leaderboard tells an interesting story. OpenAI's GPT-6 Astra sits at the top with a 28% pass rate. Anthropic's Claude Fable 5.1 follows at 22.7%. Google's own Gemini 3.8 Flash managed just 8% — awkward, given Google built the test. To Google's credit, they published it anyway, and the leaderboard now also includes GPT-5.6 Sol, Claude Opus 5, Kimi K3, and Qwen 3.8 Max.
03.No more pass/fail — continuous scoring
Android Bench 2.0 also fixes something that always bugged me about benchmarks. Under binary pass/fail grading, an agent that refactored 40 screens, set up all the database tables, and met 90% of requirements — but failed one edge case — scored a flat zero. Google replaced that with a continuous completion rate from 0.0 to 1.0, judged on functionality, visual fidelity, and avoiding regressions, with penalties for deviating from instructions or structural constraints. It's far closer to how a tech lead would actually review a junior developer's pull request.
The verification itself is clever: an emulator drives a scripted UI walkthrough of whatever the AI built, then Gemini 3.5 Flash acts as a visual judge, comparing screen captures against reference images — with 100% consistency across 360 repeated calibration runs, according to Google's methodology page.
04.Where AI coding still falls apart
The pattern in the failures is fascinating. Models are genuinely good at deterministic transformations — converting a large codebase from Java to Kotlin or swapping Retrofit for Ktor, applying the same rule across 75 to 125 files without drifting. Building new features from a clean slate also works well.
Where they collapse: architectural refactoring, runtime validation, unreleased libraries, and porting cross-platform apps to Android. Not a single model has passed a full app conversion outright. The failure mode is almost human — models race to about 80% completion early, then flatline. In one replay Google shared, GPT-5.6 Sol hit 84% on a Flutter conversion in 94 steps, then burned another 121 steps to crawl to 86%... and never added the statistics screen or dark theme support. It's the AI equivalent of a developer who starts strong and gets lost in the weeds for three days.
And those wasted steps cost real money — every extra step is tokens burned going nowhere. It's exactly why the cost-optimization chapter in our GPT-6 Astra — Zero to Hero Guide matters so much for anyone running agents on real projects: knowing how to cache, structure, and cut off runaway sessions is what separates an affordable workflow from a shocking API bill.
05.Why this matters even if you don't code Android
Two reasons. First, Android Bench 2.0 now evaluates complete coding agents — Codex, Claude Code, Google's Antigravity SDK — not just raw models, and Google found that harness design (how the agent is wired up, its prompt caching, its tool windowing) measurably changes outcomes. The lesson transfers everywhere: how you set up your AI tool matters as much as which model you pick.
Second, this is the clearest public signal yet of where the frontier actually is. If the best AI in the world fully completes only 28% of week-long engineering tasks, the "AI will replace developers" discourse needs recalibrating. The realistic picture for Indian developers: AI is a brilliant junior colleague who handles the boring 80% — the Java-to-Kotlin migrations, the boilerplate, the first draft — while you own architecture, review, and the last 20% that actually ships. We covered what GPT-6 Astra can do in detail earlier, and how Google's Gemini models stack up in our Gemini 3.8 launch story, if you want the full background.
Related reads: if this story clicked for you, GPT-6 Astra Gets 'Critical' Cybersecurity Rating and GPT-6 Astra at Work: OpenAI's AI Now Runs Your Apps.
If benchmarks themselves feel broken to you, you're not alone — Vals just raised $40M to fix AI benchmarking.
Key Takeaways
- Android Bench 2.0 tests AI on multi-day, week-long real engineering tasks — and scores collapsed from ~91% to 28% at the top
- GPT-6 Astra leads (28%), Claude Fable 5.1 second (22.7%), Gemini 3.8 Flash at 8%
- Continuous completion rate replaces binary pass/fail — partial work now gets measured fairly
- AI excels at repetitive transformations; architecture-level refactors and app conversions remain unsolved
- Agent harness design matters — setup and cost control are becoming real developer skills
Frequently Asked Questions
What is Android Bench 2.0?
Android Bench 2.0 is a new Google benchmark that tests AI models on coding tasks that would take a human engineer days — or a full week — to finish.
How did GPT-6 Astra score on Android Bench 2.0?
GPT-6 Astra topped the leaderboard with a 28% pass rate, even though it is the best model on the test. No model has fully converted a cross-platform app to Android yet; the best reach about 80% completion.
Why are the scores so low compared to older benchmarks?
The old benchmark scored models in the 90s, but Android Bench 2.0 grades on a continuous completion rate over much longer, week-long tasks and tests coding agents like Codex rather than only raw models — so the bar is far higher.
What This Means for You
If you're a student or developer in India wondering whether to learn AI-assisted coding or stick to the old ways — this benchmark is your answer. The models are powerful enough to seriously multiply your output on the routine parts of your job, and nowhere close to replacing you on the hard parts. That gap is precisely where your career lives for the next few years. For freelancers and small studios, this is actually great news: a two-person team in Pune or Bengaluru with strong AI workflow skills can now chase contracts that previously needed eight people. The practical next step is learning to direct these systems properly — prompting, agent setup, and cost optimization — which is exactly what our GPT-6 Astra — Zero to Hero Guide walks you through, step by step, without assuming you're already an expert.