Add structured output with tools support for models with limited structured output

Summary

Adds an opt-in structured output with tools feature to enable using tools and structured outputs simultaneously on models that don't natively support both (specifically Google Gemini via LiteLLM).

Problem: Models like Gemini returnBadRequestError: Function calling with a response mime type 'application/json' is unsupported when trying to use tools with structured outputs (viaresponse_schema).

Solution: Newenable_structured_output_with_tools parameter on theAgent class that:

• Injects JSON formatting instructions into the system prompt
• Disables nativeresponse_format to avoid API errors
• Parses the model's JSON response into the specified Pydantic model
• Defaults toFalse for backward compatibility
• Only actively used byLitellmModel (OpenAI models ignore it as they have native support)

Test plan

Unit tests (13 tests added):

• tests/utils/test_prompts.py - Tests for prompt generation and injection logic
  • Validates JSON schema extraction from Pydantic models
  • Tests conditional injection based on tools/output_schema presence
  • Verifies opt-in behavior (only injects when explicitly enabled)

Integration test:

• tests/test_gemini_local.py - Manual test script for Gemini (requires API key)
  • Tests actual Gemini integration with tools + structured output
  • Verifies structured output with tools works end-to-end

Verification performed:

make lint# ✅ All checks passedmake format# ✅ 359 files formatted correctlymake mypy# ✅ No type errorsmake tests# ✅ 995 passed, 3 skipped, 20 failed*

*20 failures are pre-existing Windows-specific SQLite file locking issues in temporary directory cleanup (not related to this PR). All 13 new tests pass.

Test coverage:

• Updated all existing test mock models to acceptenable_structured_output_with_tools parameter
• Verified backward compatibility (parameter is optional with defaultFalse)
• Tested with and without the feature enabled
• Validated structured output parsing with nested Pydantic models

Issue number

#2032

Checks

• I've added new tests (if relevant)
• I've added/updated the relevant documentation
• I've runmake lint andmake format
• I've made sure tests pass

Documentation

Added comprehensive documentation:

• docs/models/structured_output_with_tools.md - Complete guide with examples
  • Problem explanation with error messages
  • Step-by-step solution with code examples
  • "When to Use" table for different model providers
  • Debugging tips and best practices
• docs/agents.md - Added section on using structured outputs with tools
• docs/models/litellm.md - Added example for Gemini integration
• mkdocs.yml - Added new doc page to navigation

Files Changed

Source code:

• src/agents/agent.py - Addedenable_structured_output_with_tools parameter
• src/agents/util/_prompts.py - New utility for JSON prompt generation
• src/agents/models/interface.py - Added parameter toModel interface
• src/agents/models/openai_chatcompletions.py - Pass-through (ignores parameter)
• src/agents/models/openai_responses.py - Pass-through (ignores parameter)
• src/agents/extensions/models/litellm_model.py - Active implementation
• src/agents/run.py - Passes parameter to model calls

Tests:

• tests/utils/test_prompts.py - 13 new unit tests
• tests/test_gemini_local.py - Integration test
• Updated 5 test mock models to match new interface

Documentation:

• docs/models/structured_output_with_tools.md - New comprehensive guide
• docs/agents.md - Added usage section
• docs/models/litellm.md - Added Gemini example
• mkdocs.yml - Updated navigation

Implementation Details

Design decisions:

1. Opt-in by default -enable_structured_output_with_tools=False ensures no impact on existing code
2. LiteLLM-focused - OnlyLitellmModel actively uses this; OpenAI models ignore it
3. Interface consistency - All models accept the parameter for uniform signatures
4. User control - Users explicitly enable based on their model's requirements

How it works:

1. User setsenable_structured_output_with_tools=True on Agent
2. LitellmModel checks if both tools and output_schema are present
3. If yes, generates JSON instructions from Pydantic model schema
4. Injects instructions into system prompt
5. Disablesresponse_format to avoid API errors
6. Model returns JSON in text response
7. SDK parses and validates against Pydantic model

Example Usage

fromagentsimportAgent,function_toolfromagents.extensions.models.litellm_modelimportLitellmModelfrompydanticimportBaseModel,FieldclassWeatherReport(BaseModel):city:str=Field(description="City name")temperature:float=Field(description="Temperature in Celsius")@function_tooldefget_weather(city:str)->dict:return {"city":city,"temperature":22.5}# This now works with Gemini!agent=Agent(name="WeatherBot",model=LitellmModel("gemini/gemini-1.5-flash"),tools=[get_weather],output_type=WeatherReport,enable_structured_output_with_tools=True,# Required for Gemini)

Backward Compatibility

✅Fully backward compatible - All changes are additive:

• New parameter has default value (False)
• Existing code works without modification
• OpenAI models ignore the parameter (use native support)
• Only affects users who explicitly enable it