Repository files navigation

LLM Gym is a unified environment interface for developing and benchmarking LLM applications that learn from feedback. Thinkgym for LLM agents.

As the space of benchmarks rapidly grows, fair and comprehensive comparisons are getting trickier, so we aim to make that easier for you. The vision is an intuitive interface for a suite of environments you can seamlessly swap out for research and development purposes.

• BabyAI - Text-based versions ofBabyAI grid world environments for instruction following
• Multi-Hop - Multi-hop question answering with iterative search and note-taking
• NER - Named Entity Recognition tasks
• Tau Bench -Customer service environments for airline and retail domains
• Terminal Bench - Docker-basedterminal environments for solving programming and system administration tasks
• Twenty-One Questions - The classic guessing game where agents ask yes/no questions to identify a secret

importllmgymfromllmgym.logsimportget_loggerfromllmgym.agentsimportOpenAIAgentenv=llmgym.make("21_questions_v0")agent=llmgym.agents.OpenAIAgent(model_name="gpt-4o-mini",function_configs=env.functions,tool_configs=env.tools,)# Get default horizonmax_steps=env.horizon# Reset the environmentreset_data=awaitenv.reset()obs=reset_data.observation# Run the episodefor_stepinrange(max_steps):# Get action from agentaction=awaitagent.act(obs)# Step the environmentstep_data=awaitenv.step(action)obs=step_data.observation# Check if the episode is donedone=step_data.terminatedorstep_data.truncatedifdone:breakenv.close()

This can also be run in theQuickstart Notebook.

Follow these steps to set up the development environment for LLM Gym using uv for virtual environment management and Hatch (with Hatchling) for building and packaging.

• Python 3.12 (or a compatible version, e.g., >=3.12, <4.0)
• uv – an extremely fast Python package manager and virtual environment tool

Clone the repository to your local machine:

git clone git@github.com:tensorzero/gym-scratchpad.gitcd llmgym

Use uv to create a virtual environment. This command will create a new environment (by default in the .venv directory) using Python 3.12:

uv venv --python 3.12

Activate the virtual environment:

source .venv/bin/activate

Install the project in editable mode along with its development dependencies:

uv pip install -e.

To ensure everything is set up correctly, you can run the tests or simply import the package in Python.

Run tests:

uv run pytest

Import the package in Python:

python>>> import llmgym>>> llmgym.__version__'0.0.0'

To set theOPENAI_API_KEY environment variable, run the following command:

export OPENAI_API_KEY="your_openai_api_key"

We recommend usingdirenv and creating a local.envrc file to manage environment variables. For example, the.envrc file might look like this:

export OPENAI_API_KEY="your_openai_api_key"

and then rundirenv allow to load the environment variables.

For a full tutorial, see theTutorial Notebook.

To see how to run multiple episodes concurrently, see theTau Bench or21 Questions notebooks.

For a supervised finetuning example, see theSupervised Finetuning Notebook.