Repository files navigation

Get started building aComputer Using Agent (CUA) with the OpenAI API.

Set up python env and install dependencies.

python3 -m venv .venvsource .venv/bin/activatepip install -r requirements.txt

Run CLI to let CUA use a local browser window, usingplaywright. (Stop with CTRL+C)

python cli.py --computer local-playwright

Other included samplecomputer environments:

• Docker (containerized desktop)
• Browserbase (remote browser, requires account)
• Scrapybara (remote browser or computer, requires account)
• ...or implement your ownComputer!

The computer use tool and model are available via theResponses API. At a high level, CUA will look at a screenshot of the computer interface and recommend actions. Specifically, it sendscomputer_call(s) withactions likeclick(x,y) ortype(text) that you have to execute on your environment, and then expects screenshots of the outcomes.

You can learn more about this tool in theComputer use guide.

This repository defines two lightweight abstractions to make interacting with CUA agents more ergonomic. Everything works without them, but they provide a convenient separation of concerns.

The CLI (cli.py) is the easiest way to get started with CUA. It accepts the following arguments:

• --computer: The computer environment to use. See theComputer Environments section below for options. By default, the CLI will use thelocal-playwright environment.
• --input: The initial input to the agent (optional: the CLI will prompt you for input if not provided)
• --debug: Enable debug mode.
• --show: Show images (screenshots) during the execution.
• --start-url: Start the browsing session with a specific URL (only for browser environments). By default, the CLI will start the browsing session withhttps://bing.com.

Theexamples folder contains more examples of how to use CUA.

python -m examples.weather_example

For reference, the filesimple_cua_loop.py implements the basics of the CUA loop.

You can run it with:

python simple_cua_loop.py

CUA can work with anyComputer environment that can handle theCUA actions (plus a few extra):

This sample app provides a set of implementedComputer examples, but feel free to add your own!

Using the CLI, you can run the sample app with different computer environments using the options listed above:

python cli.py --show --computer<computer-option>

For example, to run the sample app with theDocker computer environment, you can run:

python cli.py --show --computer docker

If you want to run the sample app with theDocker computer environment, you need to build and run a local Docker container.

Open a new shell to build and run the Docker image. The first time you do this, it may take a few minutes, but subsequent runs should be much faster. Once the logs stop, proceed to the next setup step. To stop the container, press CTRL+C on the terminal where you ran the command below.

docker build -t cua-sample-app.docker run --rm -it --name cua-sample-app -p 5900:5900 --dns=1.1.1.3 -e DISPLAY=:99 cua-sample-app

docker: Error response from daemon: Conflict. The container name "/cua-sample-app" is already in use by container "e72fcb962b548e06a9dcdf6a99bc4b49642df2265440da7544330eb420b51d87"

docker rm -f cua-sample-app

This repository contains example implementations of third-party hosted environments.To use these, you will need to set up an account with the service by following the links aboveand add your API key to the.env file.

TheAgent class accepts regular function schemas intools – it will return a hard-coded value for any invocations.

However, if you pass in anytools that are also defined in yourComputer methods, in addition to the requiredComputer methods, they will be routed to yourComputer to be handled when called.This is useful for cases where screenshots often don't capture the search bar or back arrow, so CUA may get stuck. So instead, you can provide aback() orgoto(url) functions. Seeexamples/playwright_with_custom_functions.py for an example.

This repository provides example implementations with basic safety measures in place.

We recommend reviewing the best practices outlined in ourguide, and making sure you understand the risks involved with using this tool.

To contribute a new computer, you'll need to implement it, test it, and submit a PR. Please follow the steps below:

You will create or modify the following files (and only these files):

Create a new file incomputers/contrib/[your_computer_name].py and define your computer class. Make sure to implement the methods defined in theComputer class – use the existing implementations as a reference.

classYourComputerName:def__init__(self):passdefscreenshot(self):# TODO: implementpassdefclick(self,x,y):# TODO: implementpass# ... add other methods as needed

Import your new computer in thecomputers/contrib/__init__.py:

# ... existing computer importsfrom .your_computer_nameimportYourComputerName

And add your new computer to thecomputers_config dictionary incomputers/config.py:

# ... existing computers_config"your_computer_name":YourComputerName,

Feel free to add your new computer to the "Contributed Computers" section of the README.md file. Clearly indicate any auth / signup requirements.

Test your new computer (with the CLI). Make sure:

• Basic search / navigation works.
• Any setup / teardown is handled correctly.
• Test e2e with a few different tasks.

Potential gotchas (Seedefault computers for reference):

• Scrolling, dragging, and control/command keys.
• Resource allocation and teardown.
• Auth / signup requirements.

Your PR should clearly define the following:

• Title:[contrib] Add computer: <your_computer_name>
• Description:

# Add computer: <your_computer_name>#### AffiliationsWhat organization / company / institution are you affiliated with?#### Computer Description- Computer type (e.g. browser, linux)#### Testing Plan- Signup steps.- Auth steps.- Sample queries.

Thank you for your contribution! Please follow all of the above guidelines. Failure to do so may result in your PR being rejected.