Repository files navigation

📑Paper    |    🌐Project Page    |    💾 AGUVIS Data Collection   

AGUVIS is a unified pure vision-based framework for autonomous GUI agents that can operate across various platforms (web, desktop, mobile). Unlike previous approaches that rely on textual representations, AGUVIS leverages unified purely vision-based observations and a consistent action space to ensure better generalization across different platforms.

• 🔍Pure Vision Framework: First fully autonomous pure vision GUI agent capable of performing tasks independently without relying on closed-source models
• 🔄Cross-Platform Unification: Unified action space and plugin system that works consistently across different GUI environments
• 📊Comprehensive Dataset: Large-scale dataset of GUI agent trajectories with multimodal grounding and reasoning
• 🧠Two-Stage Training: Novel training pipeline focusing on GUI grounding followed by planning and reasoning
• 💭Inner Monologue: Explicit planning and reasoning capabilities integrated into the model training

Our framework demonstrates state-of-the-art performance in both offline and real-world online scenarios, offering a more efficient and generalizable approach to GUI automation.

1. Clone the repository:

git clone git@github.com:xlang-ai/aguvis.gitcd aguvis

2. Create and activate a conda environment:

conda create -n aguvis python=3.10conda activate aguvis

3. Install PyTorch and dependencies:

conda install pytorch torchvision torchaudio pytorch-cuda -c pytorch -c nvidiapip install -e.

1. Stage 1: Grounding

   • Download the dataset fromaguvis-stage1
   • Place the data according to the structure defined indata/stage1.yaml

2. Stage 2: Planning and Reasoning

   • Download the dataset fromaguvis-stage2
   • Place the data according to the structure defined indata/stage2.yaml

1. Configure your training settings:

   • Openscripts/train.sh
   • Set theSFT_TASK variable to specify your training stage

2. Start training:

bash scripts/train.sh

• Aguvis-7B-720P:Hugging Face
• Cooking... 🧑‍🍳

1. Configure your inference settings:

   • Openscripts/inference.sh
   • Set theMODEL_PATH variable to specify your model path
   • Set theIMAGE_PATH variable to specify your image path
   • Set theINSTRUCTION variable to specify your instruction
   • Set thePREVIOUS_ACTIONS variable to specify your previous actions or leave it empty
   • Set theLOW_LEVEL_INSTRUCTION variable to specify your low-level instruction or leave it empty

2. Start inference:

bash scripts/inference.sh

• Data
  • ✅ Stage 1: Grounding Dataset
  • ✅ Stage 2: Planning and Reasoning Trajectories
• Code
  • ✅ Training Pipeline
  • 🚧 Model Weights and Configurations
  • 🚧 Inference Scripts
  • 🚧 Evaluation Toolkit

If this work is helpful, please kindly cite as:

@article{xu2024aguvis,title={Aguvis: Unified Pure Vision Agents for Autonomous GUI Interaction},author={Yiheng Xu and Zekun Wang and Junli Wang and Dunjie Lu and Tianbao Xie and Amrita Saha and Doyen Sahoo and Tao Yu and Caiming Xiong},year={2024},url={https://arxiv.org/abs/2412.04454}}