OpenAI Options

Theweb_search() tool can use OpenAI’s built-in search capability when running on a limited number of OpenAI models (currently “gpt-4o”, “gpt-4o-mini”, and “gpt-4.1”). This provider does not require any API keys beyond what’s needed for the model itself.

For more details on OpenAI’s web search parameters, seeOpenAI Web Search Documentation.

Note that when using the “openai” provider, you should also specify a fallback external provider (like “tavily”, “exa”, or “google”) if you are also running the evaluation with non-OpenAI model.

Anthropic Options

Theweb_search() tool can use Anthropic’s built-in search capability when running on a limited number of Anthropic models (currently “claude-opus-4-20250514”, “claude-sonnet-4-20250514”, “claude-3-7-sonnet-20250219”, “claude-3-5-sonnet-latest”, “claude-3-5-haiku-latest”). This provider does not require any API keys beyond what’s needed for the model itself.

For more details on Anthropic’s web search parameters, seeAnthropic Web Search Documentation.

Note that when using the “anthropic” provider, you should also specify a fallback external provider (like “tavily”, “exa”, or “google”) if you are also running the evaluation with non-Anthropic model.

Gemini Options

Theweb_search() tool can use Google’s built-in search capability (called grounding) when running on Gemini 2.0 models and later. This provider does not require any API keys beyond what’s needed for the model itself.

This is distinct from the “google” provider (described below), which uses Google’s external Programmable Search Engine service and requires separate API keys.

For more details, seeGrounding with Google Search.

Note that when using the “gemini” provider, you should also specify a fallback external provider (like “tavily”, “exa”, or “google”) if you are also running the evaluation with non-Gemini models.

Perplexity Options

Theweb_search() tool can use Perplexity’s built-in search capability when running on Perplexity models. This provider does not require any API keys beyond what’s needed for the model itself. Search parameters can be passed using theperplexity provider options and will be forwarded to the model API.

For more details, seePerplexity API Documentation.

Note that when using the “perplexity” provider, you should also specify a fallback external provider (like “tavily”, “exa”, or “google”) if you are also running the evaluation with non-Perplexity models.

Tavily Options

Theweb_search() tool can useTavily’s Research API. To use it you will need to set up your own Tavily account. Then, ensure that the following environment variable is defined:

• TAVILY_API_KEY — Tavily Research API key

Tavily supports the following options:

For more options, see theTavily API Documentation.

Exa Options

Theweb_search() tool can useExa’s Answer API. To use it you will need to set up your own Exa account. Then, ensure that the following environment variable is defined:

• EXA_API_KEY — Exa API key

Exa supports the following options:

For more details, see theExa API Documentation.

Google Options

Theweb_search() tool can useGoogle Programmable Search Engine as an external provider. This is different from the “gemini” provider (described above), which uses Google’s built-in search capability for Gemini models.

To use the “google” provider you will need to set up your own Google Programmable Search Engine and also enable theProgrammable Search Element Paid API. Then, ensure that the following environment variables are defined:

• GOOGLE_CSE_ID — Google Custom Search Engine ID
• GOOGLE_CSE_API_KEY — Google API key used to enable the Search API

Google supports the following options:

Configuration

Bash sessions require the use of aSandbox Environment for the execution of untrusted code. In addition, you’ll need some dependencies installed in the sandbox container. Please seeSandbox Dependencies below for additional instructions.

RUNapt-get update&&apt-get install-y pipx&&\apt-get clean&&rm-rf /var/lib/apt/lists/*&&\pipx ensurepathENV PATH="$PATH:/root/.local/bin"RUNpipx install inspect-tool-support&&inspect-tool-support post-install

RUNinspect-tool-support post-install--no-web-browser

compose.yaml

services:default:image: aisiuk/inspect-tool-supportinit:true

Task Setup

A task configured to use the bash session tool might look like this:

from inspect_aiimport Task, taskfrom inspect_ai.scorerimport includesfrom inspect_ai.solverimport generate, system_message, use_toolsfrom inspect_ai.toolimport bash_session@taskdef intercode_ctf():return Task(        dataset=read_dataset(),        solver=[            system_message("system.txt"),            use_tools([bash_session(timeout=180)]),            generate(),        ],        scorer=includes(),        sandbox=("docker","compose.yaml")    )

Note that we provide atimeout for bash session commands (this is a best practice to guard against extremely long running commands).

Configuration

The text editor tools requires the use of aSandbox Environment. In addition, you’ll need some dependencies installed in the sandbox container. Please seeSandbox Dependencies below for additional instructions.

RUNapt-get update&&apt-get install-y pipx&&\apt-get clean&&rm-rf /var/lib/apt/lists/*&&\pipx ensurepathENV PATH="$PATH:/root/.local/bin"RUNpipx install inspect-tool-support&&inspect-tool-support post-install

RUNinspect-tool-support post-install--no-web-browser

compose.yaml

services:default:image: aisiuk/inspect-tool-supportinit:true

Task Setup

A task configured to use the text editor tool might look like this (note that this task is also configured to use thebash_session() tool):

from inspect_aiimport Task, taskfrom inspect_ai.scorerimport includesfrom inspect_ai.solverimport generate, system_message, use_toolsfrom inspect_ai.toolimport bash_session, text_editor@taskdef intercode_ctf():return Task(        dataset=read_dataset(),        solver=[            system_message("system.txt"),            use_tools([                bash_session(timeout=180),                text_editor(timeout=180)            ]),            generate(),        ],        scorer=includes(),        sandbox=("docker","compose.yaml")    )

Note that we provide atimeout for the bash session and text editor tools (this is a best practice to guard against extremely long running commands).

Tool Binding

The schema for thetext_editor() tool is based on the standard Anthropictext editor tool type. Thetext_editor() works with all models that support tool calling, but when using Claude, the text editor tool will automatically bind to the native Claude tool definition.

Configuration

Under the hood, the web browser is an instance ofChromium orchestrated byPlaywright, and runs in aSandbox Environment. In addition, you’ll need some dependencies installed in the sandbox container. Please seeSandbox Dependencies below for additional instructions.

Note that Playwright (used for theweb_browser() tool) does not support some versions of Linux (e.g. Kali Linux).

RUNapt-get update&&apt-get install-y pipx&&\apt-get clean&&rm-rf /var/lib/apt/lists/*&&\pipx ensurepathENV PATH="$PATH:/root/.local/bin"RUNpipx install inspect-tool-support&&inspect-tool-support post-install

compose.yaml

services:default:image: aisiuk/inspect-tool-supportinit:true

Task Setup

A task configured to use the web browser tools might look like this:

from inspect_aiimport Task, taskfrom inspect_ai.scorerimport matchfrom inspect_ai.solverimport generate, use_toolsfrom inspect_ai.toolimport bash, python, web_browser@taskdef browser_task():return Task(        dataset=read_dataset(),        solver=[            use_tools([bash(), python()]+ web_browser()),            generate(),        ],        scorer=match(),        sandbox=("docker","compose.yaml"),    )

Unlike some other tool functions likebash(), theweb_browser() function returns a list of tools. Therefore, we concatenate it with a list of the other tools we are using in the call touse_tools().

Note that a separate web browser process is created within the sandbox for each instance of the web browser tool. See theweb_browser() reference docs for details on customizing this behavior.

Browsing

If you review the transcripts of a sample with access to the web browser tool, you’ll notice that there are several distinct tools made available for control of the web browser. These tools include:

The return value of each of these tools is aweb accessibility tree for the page, which provides a clean view of the content, links, and form fields available on the page (you can look at the accessibility tree for any web page usingChrome Developer Tools).

Disabling Interactions

You can use the web browser tools with page interactions disabled by specifyinginteractive=False, for example:

use_tools(web_browser(interactive=False))

In this mode, the interactive tools (web_browser_click(),web_browser_type(), andweb_browser_type_submit()) are not made available to the model.

Configuration

Thecomputer() tool runs within a Docker container. To use it with a task you need to reference theaisiuk/inspect-computer-tool image in your Docker compose file. For example:

compose.yaml

services:default:image: aisiuk/inspect-computer-tool

You can configure the container to not have Internet access as follows:

compose.yaml

services:default:image: aisiuk/inspect-computer-toolnetwork_mode: none

Note that if you’d like to be able to view the model’s interactions with the computer desktop in realtime, you will need to also do some port mapping to enable a VNC connection with the container. See theVNC Client section below for details on how to do this.

Theaisiuk/inspect-computer-tool image is based on theubuntu:22.04 image and includes the following additional applications pre-installed:

• Firefox
• VS Code
• Xpdf
• Xpaint
• galculator

Task Setup

A task configured to use the computer tool might look like this:

from inspect_aiimport Task, taskfrom inspect_ai.scorerimport matchfrom inspect_ai.solverimport generate, use_toolsfrom inspect_ai.toolimport computer@taskdef computer_task():return Task(        dataset=read_dataset(),        solver=[            use_tools([computer()]),            generate(),        ],        scorer=match(),        sandbox=("docker","compose.yaml"),    )

To evaluate the task with models tuned for computer use:

inspect eval computer.py--model anthropic/claude-3-7-sonnet-latestinspect eval computer.py--model openai/computer-use-preview

solver=[    use_tools([computer(max_screenshots=2, timeout=300)]),    generate()]

VNC Client

You can use aVNC connection to the container to watch computer use in real-time. This requires some additional port-mapping in the Docker compose file. You can define dynamic port ranges for VNC (5900) and a browser based noVNC client (6080) with the followingports entries:

compose.yaml

services:default:image: aisiuk/inspect-computer-toolports:-"5900"-"6080"

To connect to the container for a given sample, locate the sample in theRunning Samples UI and expand the sample info panel at the top:

Click on the link for the noVNC browser client, or use a native VNC client to connect to the VNC port. Note that the VNC server will take a few seconds to start up so you should give it some time and attempt to reconnect as required if the first connection fails.

The browser based client provides a view-only interface. If you use a native VNC client you should also set it to “view only” so as to not interfere with the model’s use of the computer. For example, for Real VNC Viewer:

Approval

If the container you are using is connected to the Internet, you may want to configure human approval for a subset of computer tool actions. Here are the possible actions (specified using theaction parameter to thecomputer tool):

• key: Press a key or key-combination on the keyboard.
• type: Type a string of text on the keyboard.
• cursor_position: Get the current (x, y) pixel coordinate of the cursor on the screen.
• mouse_move: Move the cursor to a specified (x, y) pixel coordinate on the screen.
• Example: execute(action=“mouse_move”, coordinate=(100, 200))
• left_click: Click the left mouse button.
• left_click_drag: Click and drag the cursor to a specified (x, y) pixel coordinate on the screen.
• right_click: Click the right mouse button.
• middle_click: Click the middle mouse button.
• double_click: Double-click the left mouse button.
• screenshot: Take a screenshot.

Here is an approval policy that requires approval for key combos (e.g. Enter or a shortcut) and mouse clicks:

approval.yaml

approvers:-name: humantools:- computer(action='key'- computer(action='left_click'- computer(action='middle_click'- computer(action='double_click'-name: autotools:"*"

Note that since this is a prefix match and there could be other arguments, we don’t end the tool match pattern with a parentheses.

You can apply this policy using the--approval command line option:

inspect eval computer.py--approval approval.yaml

Tool Binding

The computer tool’s schema is a superset of the standardAnthropic andOpen AI computer tool schemas. When using models tuned for computer use (currentlyanthropic/claude-3-7-sonnet-latest andopenai/computer-use-preview) the computer tool will automatically bind to the native computer tool definitions (as this presumably provides improved performance).

If you want to experiment with bypassing the native computer tool types and just register the computer tool as a normal function based tool then specify the--no-internal-tools generation option as follows:

inspect eval computer.py--no-internal-tools

Usage

You should read the originalthink tool article in its entirely to understand where and where not to use the think tool. In summary, good contexts for the think tool include:

1. Tool output analysis. When models need to carefully process the output of previous tool calls before acting and might need to backtrack in its approach;
2. Policy-heavy environments. When models need to follow detailed guidelines and verify compliance; and
3. Sequential decision making. When each action builds on previous ones and mistakes are costly (often found in multi-step domains).

Use thethink() tool alongside other tools like this:

from inspect_aiimport Task, taskfrom inspect_ai.scorerimport includesfrom inspect_ai.solverimport generate, system_message, use_toolsfrom inspect_ai.toolimport bash_session, text_editor, think@taskdef intercode_ctf():return Task(        dataset=read_dataset(),        solver=[            system_message("system.txt"),            use_tools([                bash_session(timeout=180),                text_editor(timeout=180),                think()            ]),            generate(),        ],        scorer=includes(),        sandbox=("docker","compose.yaml")    )

Tool Description

In the originalthink tool article (which was based on experimenting with Claude) they found that providing clear instructions on when and how to use thethink() tool for the particular problem domain it is being used within could sometimes be helpful. For example, here’s the prompt they used with SWE-Bench:

from textwrapimport dedentfrom inspect_aiimport Task, taskfrom inspect_ai.scorerimport includesfrom inspect_ai.solverimport generate, system_message, use_toolsfrom inspect_ai.toolimport bash_session, text_editor, think@taskdef swe_bench():    tools= [        bash_session(timeout=180),        text_editor(timeout=180),        think(dedent("""            Use the think tool to think about something. It will not obtain            new information or make any changes to the repository, but just            log the thought. Use it when complex reasoning or brainstorming            is needed. For example, if you explore the repo and discover            the source of a bug, call this tool to brainstorm several unique            ways of fixing the bug, and assess which change(s) are likely to            be simplest and most effective. Alternatively, if you receive            some test results, call this tool to brainstorm ways to fix the            failing tests.        """))    ])return Task(        dataset=read_dataset(),        solver=[            system_message("system.txt"),            use_tools(tools),            generate(),        ),        scorer=includes(),        sandbox=("docker","compose.yaml")    )

System Prompt

In the article they also found that when tool instructions are long and/or complex, including instructions about thethink() tool in the system prompt can be more effective than placing them in the tool description itself.

Here’s an example of moving the customthink() prompt into the system prompt (note that this wasnot done in the article’s SWE-Bench experiment, this is merely an example):

from textwrapimport dedentfrom inspect_aiimport Task, taskfrom inspect_ai.scorerimport includesfrom inspect_ai.solverimport generate, system_message, use_toolsfrom inspect_ai.toolimport bash_session, text_editor, think@taskdef swe_bench():    think_system_message= system_message(dedent("""        Use the think tool to think about something. It will not obtain        new information or make any changes to the repository, but just        log the thought. Use it when complex reasoning or brainstorming        is needed. For example, if you explore the repo and discover        the source of a bug, call this tool to brainstorm several unique        ways of fixing the bug, and assess which change(s) are likely to        be simplest and most effective. Alternatively, if you receive        some test results, call this tool to brainstorm ways to fix the        failing tests.    """))return Task(        dataset=read_dataset(),        solver=[            system_message("system.txt"),            think_system_message,            use_tools([                bash_session(timeout=180),                text_editor(timeout=180),                think(),            ]),            generate(),        ],        scorer=includes(),        sandbox=("docker","compose.yaml")    )

Note that the effectivess of using the system prompt will vary considerably across tasks, tools, and models, so should definitely be the subject of experimentation.