Repository files navigation

We proposeZeroGUI, a fully automated online reinforcement learning framework that enables GUI agents to train and adapt in interactive environments at zero human cost.

🔥 Updates

• 2025/07/05: Release the training scripts on OSWorld.

• 2025/06/15: Release the evaluation code and scripts on OSWorld.

• 2025/05/30: Release the task generation code on OSWorld.

• 2025/05/30: Release our paper and model checkpoints.

• 🚫Zero Human Cost: Requires no handcrafted task annotations or rule-based reward designs.

• 🧠VLM-based Automation: Both training tasks and rewards are generated by powerful VLMs.

• ♻️Online Learning: Agents continuously learn from interacting with GUI environments.

• 📈Significant Gains: +63% (Aguvis) and +14% (UI-TARS) relative improvements on OSWorld.

• 🧠Automatic Task Generation: Automatically proposes diverse, executable GUI tasks.

• ✅Automatic Reward Estimation: Assigns binary task rewards based on trajectory screenshots and employs a voting mechanism to avoid hallucinated success.

• ♻️Two-Stage Online RL: Combines training on generated tasks and test-time adaptation to continually improve agent's performance.

The prompts and parsing functions are provided inopenrlhf/agent/uitars.py. You can refer toUI-TARS for more details.

1. Setup python environment:pip install -r requirements.txt. Please usepython>=3.10.

2. Setup GUI environment:OSWorld.

Use the following command to evaluate the model on OSWorld:

bash scripts/eval/eval_osworld.sh OpenGVLab/ZeroGUI-OSWorld-7B<env-url><env-manager-port>

where<env-url> and<env-manager-port> are the URL and port of the API manager launchedhere.

scripts/train_osworld_test-time contains an example script for test-time training on OSWorld.

1. Modifytrain.sh according to your server setup.ENV_URL andENV_MANAGER_PORT are the URL and port of the environment API manager.API_BASE_URL andAPI_KEY are used for VLM-based reward estimation. You can deploy the VLM locally or use online APIs.

2. Modify the Slurm settings inslurm_launch.sh. Adjust the environment variables provided by your Slurm configuration inray_launch.sh

3. Run the following command:

bash scripts/train_osworld_test-time/srun_launch.sh

1. Run task generation:OSWorld. Organize the task metas likedata/osworld_test_all.jsonl.

2. Launch training similarly to test-time training.

If you find this work helpful in your research, please consider citing:

@article{yang2025zerogui,title={ZeroGUI: Automating Online GUI Learning at Zero Human Cost},author={Yang, Chenyu and Shiqian, Su and Liu, Shi and Dong, Xuan and Yu, Yue and Su, Weijie and Wang, Xuehui and Liu, Zhaoyang and Zhu, Jinguo and Li, Hao and Wang, Wenhai and Qiao, Yu and Zhu, Xizhou and Dai, Jifeng},journal={arXiv preprint arXiv:2505.23762},year={2025}}

Our code is built with reference to the code of the following projects:OpenRLHF,UI-TARS,AGUVIS,OSWorld, andAndroidLab.