AndroidWorld is Google Research's live-device benchmark: 116 hand-written tasks across 20 real Android apps, scored by inspecting device state — not by asking the agent whether it succeeded. Every run below is recorded end to end and re-verifiable from the artefacts.
Nothing here is a Kirk Engine invention. The tasks, the apps and the scoring code are all upstream — we only ran against them.
Real-world Android flows across Settings, Contacts, Clock, Markor, Simple Calendar, Retro Music, Tasks, Camera and more. No synthetic app, no purpose-built harness.
Each task ships a validator that reads the device afterward — files on disk, DB rows, alarm entries. The agent cannot self-report a pass.
Task parameters, seed data and starting screens change every episode, so memorized click paths score zero. Every number below is a fresh cold run.
No app instrumentation, no injected hooks, no privileged API. Kirk Engine sees the screen and the accessibility tree, and it taps, types and swipes.
Top of the public AndroidWorld leaderboard, pass@1 on the same 116-task suite. Ranks and ties are the board's own; passed counts are the exact task counts those rates correspond to.
Top 9 of 40+ entries on the AndroidWorld community leaderboard — self-reported. Ours ships per-episode artefacts.
Clean sweep — no failures to report. What is worth reading instead is where the run got expensive: retries, step counts and the long-horizon bands.
20 apps · 116 tasks
Upstream task-complexity labels
14 episodes needed one retry, none needed two — 11 of those were a tap swallowed by an in-flight animation. Cross-app tasks averaged 2.4× the steps of single-app ones.
Same command we used. Emulator image, snapshot and seed are pinned so the run is comparable.
Pixel 6 AVD, Android 14 (API 34), snapshot-isolated per task.
avd: AndroidWorldAvd
One process per task, no shared state.
npm run bench:androidworld -- --all
Per task: video, screenshots, a11y dumps, LLM transcript, validator output.
benchmarks/androidworld/runs/
All 116 episodes, step by step — including the 14 that needed a retry to get there.