A Real-Device Benchmark for Mobile GUI Agents
Built from real user traffic, hand-rewritten for reproducibility, and executed on physical devices against live applications — not emulators. Every task is scored by fine-grained sub-goals with partial credit, and most span multiple apps.
High benchmark scores do not reliably predict performance on real devices. Account states, permission dialogs, payment authentication, and risk-control mechanisms continually reshape the state distribution a GUI agent actually encounters — opening a persistent gap between benchmark performance and real-world usability.
RealMobile closes that gap by evaluating agents where they will be deployed: on physical phones, driving live commercial Chinese applications, on tasks drawn from real user traffic. Instead of a single pass/fail signal, each task is decomposed into ordered sub-goals that award partial credit, so an agent that completes most of a long-horizon task is measured fairly.
Executed on physical phones against production applications — capturing permission dialogs, login walls, and risk control that emulators never surface.
Each task is broken into ordered sub-goals with partial credit, verified with XPath and code rules over the UI-hierarchy XML plus PaddleOCR text.
57% of tasks require coordinating two or more applications — cross-app search, information transfer, and comparison — reflecting how people use their phones.
Every query, sub-goal rule, and scoring script is open. Recorded trajectories are re-scored deterministically, so results are auditable end-to-end.
Success = fraction of tasks fully completed. Progress = mean fraction of sub-goals completed.
| # | Model | Type | Success | Progress |
|---|
| Model | Foundation | Safety & Reflection |
Memory & Knowledge | Complex Reasoning & Planning |
|---|
Browse the RealMobile tasks — each with its natural-language query, the applications involved, and the ordered sub-goal rubric used for scoring. Click any row for the full rubric.
| ID | Query | Apps | Sub-goals |
|---|
@misc{cao2026xiaomigui0technicalreport,
title={Xiaomi-GUI-0 Technical Report},
author={Wanxia Cao and Chengzhen Duan and Pei Fu and Pengzhi Gao and Niu Lian and Fazhan Liu and Hui Liu and Heng Qu and Qinzhuo Wu and Zhehao Yu and Tongbo Chen and Shiqi Cui and Anan Du and Shukai Jia and Yuanfa Li and Wei Liu and Yike Liu and Wenchao Lu and Zhenbo Luo and Haoyuan Sun and Jiatong Sun and Cheng Tan and Yajie Wang and Changqiao Wu and Tao Xiong and Jiahui Yang and Yuxuan Yuan and Ruoceng Zhang and Shaojie Zhang and Jian Zhu and Jian Luan and Cong Zou},
year={2026},
eprint={2606.31410},
archivePrefix={arXiv},
primaryClass={cs.AI},
url={https://arxiv.org/abs/2606.31410},
}