- Notifications
You must be signed in to change notification settings - Fork1k
Closed
Assignees
Labels
Description
Initial Checks
- I confirm that I'm using the latest version of Pydantic AI
- I confirm that I searched for my issue inhttps://github.com/pydantic/pydantic-ai/issues before opening this issue
Description
When using a tool that returns binary data the binary data is replaced with e4fcfe. I have attached a test case that reliably reproduces the issue for me.
When tracing the LLM calls the input to the final agent call is:
[ {"content":"call the find_image tool and return the image unaltered","role":"system","gen_ai.system":"openai","event.name":"gen_ai.system.message" }, {"content":"get me the embroidery image","role":"user","gen_ai.system":"openai","gen_ai.message.index":0,"event.name":"gen_ai.user.message" }, {"role":"assistant","tool_calls": [ {"id":"call_VyMOmIVGQxcPS8vKCYPOGh1L","type":"function","function": {"name":"find_image","arguments": {"name":"embroidery" } } } ],"gen_ai.system":"openai","gen_ai.message.index":1,"event.name":"gen_ai.assistant.message" }, {"content":"See file e4fcfe","role":"tool","id":"call_VyMOmIVGQxcPS8vKCYPOGh1L","name":"find_image","gen_ai.system":"openai","gen_ai.message.index":2,"event.name":"gen_ai.tool.message" }, {"content": ["This is file e4fcfe:", {"kind":"binary","media_type":"image/jpeg","binary_content":"... actual content truncated for brevity ..." } ],"role":"user","gen_ai.system":"openai","gen_ai.message.index":2,"event.name":"gen_ai.user.message" }]the response from the model is:
{"index":0,"message": {"role":"assistant","tool_calls": [ {"id":"call_QRmUFFnOYPmvv5nrZXR4kR1C","type":"function","function": {"name":"final_result","arguments": {"data":"e4fcfe","media_type":"image/jpeg" } } } ] },"gen_ai.system":"openai","event.name":"gen_ai.choice"}Example Code
frompathlibimportPathfrompydanticimportBaseModel,Fieldfrompydantic_aiimportAgent,BinaryContentAGENT_MODEL="openai:gpt-4o-mini"# https://www.daytonartinstitute.org/wp-content/uploads/2019/10/2009_8.jpgFIXTURE_FILE= (Path(__file__).resolve().parent/"fixtures/embroidery-from-uzbekistan.jpg")FILE_CONTENT=FIXTURE_FILE.read_bytes()IMAGE=BinaryContent(data=FILE_CONTENT,media_type="image/jpeg")deffind_image(name:str)->BinaryContent:""" Retrieve the image the user is looking for. Args: name: The identifier of the image. This is extremely permissive, this almost always works. Returns: The image that the user was looking for. """returnIMAGEasyncdeftest_llm_tool_returning_binary_content()->None:agent=Agent(model=AGENT_MODEL,instructions="call the find_image tool and return the image unaltered",output_type=BinaryContent,tools=[find_image],model_settings={"temperature":0.0}, )# response content is now e4fcfe# the media_type is still correctresponse=awaitagent.run("get me the embroidery image")assertresponse.output==IMAGE
Python, Pydantic AI & LLM client version
Python 3.12.10pydantic-ai 0.4.3openai 1.97.0