Test Coding Agents on Long Android Tasks

You are choosing a coding agent for Android work that outlives a demo: a build, a port, or a migration that touches screens, state, and dependencies. Run one long-horizon task from your backlog, score completion rate, and inspect where the agent stalls. Do not buy on a demo clip.

Long-horizon tasks expose the difference between a model that edits code and an agent that carries a build across screens, states, and regressions. Android Bench 2.0 evaluates long-horizon Android tasks that can require multiple days or a week of engineering work.

The best long-horizon pass rate reported for Android Bench 2.0 is about 28%, while the original benchmark tasks had a pass rate near 91%. OpenAI's GPT-6 Astra leads the Android Bench 2.0 leaderboard. The leaderboard is a starting point, not a shopping list.

Pick the task before you trust the agent

Choose a real Android task from your backlog, not a toy example. It needs enough surface area to force navigation across screens, state, and dependencies. Start with a repository, a target state, and a short description of the user-visible outcome.

Freeze the starting repository before the agent touches it. Capture the baseline build, known defects, and the screens you will compare later. A clean branch, a pinned dependency list, and a screenshot set for the affected flows make the run auditable.

Define the completion score before you run anything. Keep the checks visible: functionality, visual fidelity, and regressions. Give each check a column and a weighting note, so the score is not a mood.

Do not let the agent declare success. If it says done, stop and score it against your rubric. Treat the exercise as a test harness for AI tools, not a performance stage.

Keep the run log. Note where the agent asks for help, guesses, or gives up. A short failure note is worth more than a polished final screen.

Score completion, not a pass/fail badge

Run the agent on the full task, not a trimmed slice. Let it work through the build, edits, and verification steps. You need a final diff, a build log, and screenshots or recordings from the finished app.

Score functionality against the user-visible outcome. Walk the required flows, empty states, and failure states. Mark each required behavior as pass, partial pass, or fail.

Compare visual fidelity against the frozen screenshots. Check spacing, alignment, state changes, and layout drift. A side-by-side comparison should name what changed and why it matters.

Audit regressions against the baseline. Re-run the flows that worked before the agent started. List the screens or flows that broke, with severity and reproduction steps.

Combine the checks into a continuous completion rate. Android Bench 2.0 uses a continuous completion rate that combines functionality, visual fidelity, and regression avoidance. The score should be a comparable number across agents, not a yes-or-no badge.

Let the failure row decide the purchase

Map your work to a failure pattern: deterministic transformation, new feature, architectural migration, runtime validation, or cross-platform port. Attach that row label to every test result.

Mechanical refactors fit deterministic transformation. A new screen with new state fits new feature. If your work moves between architectures or platforms, choose the row that matches that move.

Compare agents on that row, not on the aggregate. Routine migration work can extend across more than 125 files and 8,000 lines, whereas tasks involving runtime validation, breaking framework changes, or unreleased libraries are harder for models. No model fully passes the cross-platform Android porting tasks, and the best frontier models top out at an 80% completion rate on those tasks.

Pair the model with the agent harness you would actually use. Android Bench 2.0 included agent evaluations paired with provider tooling, including GPT 5.6 Sol with Codex and Gemini 3.8 Flash with Google Antigravity. The result is useful when it reflects your tooling, not a model alone.

Repeat the harness with another task from the same row. The decision is defensible when you have paired completion rates, paired failure notes, and a reason to choose.