Repository files navigation

A Gradio-based demonstration for the Microsoft Fara-7B model, designed as a computer use agent. Users upload UI screenshots (e.g., desktop or app interfaces), provide task instructions (e.g., "Click on the search bar"), and receive parsed actions (clicks, types) with visualized indicators (circles and labels) overlaid on the image. Supports JSON-formatted tool calls for precise coordinate-based interactions.

Demo:https://huggingface.co/spaces/prithivMLmods/CUA-GUI-Operator

• UI Image Processing: Upload screenshots; model analyzes and suggests actions like clicks or text input at specific coordinates.
• Task-Driven Inference: Natural language instructions generate structured JSON actions (e.g., {"action": "click", "coordinate": [400, 300]}).
• Action Visualization: Overlays red circles for clicks and blue for others, with labels (e.g., "Click" or "Type: 'Hello'") on the output image.
• Response Parsing: Extracts tool calls from model output using regex; handles multiple actions per task.
• Custom Theme: OrangeRedTheme with gradients for an intuitive interface.
• Examples Integration: Pre-loaded samples for quick testing (e.g., Windows start menu, search box).
• Queueing Support: Handles up to 50 concurrent inferences for efficient use.
• Error Resilience: Fallbacks for model loading failures or invalid inputs; console logging for debugging.

• Python 3.10 or higher.
• CUDA-compatible GPU (recommended for float16; falls back to CPU).
• Git for cloning dependencies.
• Hugging Face account (optional, for model caching viahuggingface_hub).

1. Clone the repository:

   git clone https://github.com/PRITHIVSAKTHIUR/Fara-7B-Action-Points-Demo.gitcd Fara-7B-Action-Points-Demo

2. Install dependencies:Create arequirements.txt file with the following content, then run:

   pip install -r requirements.txt

   requirements.txt content:

   transformers==4.57.1webdriver-managerhuggingface_hubpython-dotenvsentencepieceqwen-vl-utilsgradio_modaltorchvisionmatplotlibacceleratenum2wordspydanticrequestspillowopenaispaceseinopstorchpeft

3. Start the application:

   python app.py

   The demo launches athttp://localhost:7860 (or the provided URL if using Spaces).

1. Upload Image: Provide a UI screenshot (e.g., PNG of a desktop or app window).

2. Enter Task: Describe the action in the textbox (e.g., "Click on the start menu" or "Type 'Hello World' in the search box").

3. Execute: Click "Execute Agent" to run inference.

4. View Results:

   • Text: Raw model response with parsed JSON actions.
   • Image: Annotated screenshot showing action points (circles with labels).

• Upload a Windows desktop image.
• Task: "Click on the start menu."
• Output: Response with click action at coordinates; image with red circle labeled "Click" on the start button.

• Model Loading Errors: Ensure transformers 4.57.1; check CUDA withtorch.cuda.is_available(). Usetorch.float32 if float16 OOM occurs.
• No Actions Parsed: Verify task clarity; raw output logged in console. Adjust max_new_tokens if truncated.
• Visualization Issues: PIL font errors fallback to default; ensure images are RGB.
• Queue Full: Increasemax_size indemo.queue() for higher traffic.
• Vision Utils: Installqwen-vl-utils for image processing; test with examples.
• UI Rendering: Setssr_mode=True if gradients fail; check CSS for custom styles.

Contributions encouraged! Fork, create a feature branch (e.g., for multi-step tasks), and submit PRs with tests. Focus areas:

• Support for video inputs or real-time GUI control.
• Additional action types (e.g., scroll, drag).
• Integration with browser automation.

Repository:https://github.com/PRITHIVSAKTHIUR/Fara-7B-Action-Points-Demo.git

Apache License 2.0. SeeLICENSE for details.

Built by Prithiv Sakthi. Report issues via the repository.