- Notifications
You must be signed in to change notification settings - Fork1k
Open
Assignees
Description
Question
Hi community,
I've tried some time getting a structured output to be streamed with Gemini and vertex-ai. It works when I run the example with DeepSeek. With Gemini, the output is not streamed. I want to realize a streamed output of a list of dataclasses, which I want to stream to a web front end, where I want to animate them.
Maybe it is just a simple thing I cannot see. I considered other issues here and your docs but could not find the solution. If someone could give me a hint, that would be awesome.
fromtypingimportOptional,TypedDictfrompydantic_aiimportAgentfrompydantic_ai.models.geminiimportGeminiModelimportasynciofromtyping_extensionsimportTypedDictfrompydantic_ai.models.geminiimportGeminiModel,GeminiModelSettingsmodel_settings=GeminiModelSettings(gemini_thinking_config={'thinking_budget':128})model=GeminiModel('gemini-2.5-pro',provider='google-vertex')classUserProfile(TypedDict,total=False):name:strage:Optional[int]=Nonebio:stragent=Agent(model=model,output_type=list[UserProfile],stream=True,model_settings=model_settings)# --- Streaming logic ---asyncdefstream_chat(prompt:str):asyncwithagent.run_stream(prompt)asresult:asyncforprofileinresult.stream_structured(debounce_by=None):yieldprofileasyncdefprint_llm_response(prompt):response=stream_chat(prompt)asyncforchunkinresponse:print(chunk,end="",flush=True)if__name__=="__main__":asyncio.run(print_llm_response("Make up 10 different persons with their unique stories."))
Additional Context
No response