- Notifications
You must be signed in to change notification settings - Fork746
Add provider Azure Anthropic#5151
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
base:main
Are you sure you want to change the base?
Conversation
github-actionsbot commentedDec 12, 2025 • edited
Loading Uh oh!
There was an error while loading.Please reload this page.
edited
Uh oh!
There was an error while loading.Please reload this page.
All contributors have signed the CLA ✍️ ✅ |
hugentobler commentedDec 12, 2025
I have read the Contributor License Agreement (CLA) and hereby sign the CLA. |
hugentobler commentedDec 12, 2025
recheck |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Pull request overview
This PR adds a new Azure Anthropic provider that enables the TensorZero Gateway to interact with Anthropic models hosted on Azure. The implementation follows the pattern established by the GCP Vertex Anthropic provider and uses Azure's/anthropic/messages endpoint, which wraps the standard Anthropic API.
Key Changes:
- New provider implementation supporting Azure's Anthropic Claude models (claude-haiku-4-5, claude-opus-4-1, claude-sonnet-4-5, claude-opus-4-5)
- Support for core features including streaming, tools, thinking blocks, and JSON mode
- Integration with existing credential and endpoint management systems
Reviewed changes
Copilot reviewed 5 out of 6 changed files in this pull request and generated 10 comments.
Show a summary per file
| File | Description |
|---|---|
| tensorzero-core/src/providers/azure_anthropic.rs | New provider implementation with request/response handling, streaming support, and basic tests |
| tensorzero-core/src/providers/mod.rs | Adds azure_anthropic module to the providers list |
| tensorzero-core/src/model_table.rs | Registers AzureAnthropic provider type and implements credential management |
| tensorzero-core/src/model.rs | Integrates AzureAnthropicProvider into the model configuration and inference routing |
| tensorzero-core/src/config/provider_types.rs | Adds configuration structure with default AZURE_ANTHROPIC_API_KEY environment variable |
| .gitignore | Adds dev-config/ directory to ignored paths |
| // TODO: Implement test_from_tool | ||
| // This test should verify that AnthropicTool is correctly created from a FunctionToolConfig | ||
| // See gcp_vertex_anthropic.rs for reference implementation | ||
| #[tokio::test] | ||
| async fn test_from_tool() { | ||
| // TODO: Test tool conversion from FunctionToolConfig to AnthropicTool | ||
| // Should test both dynamic and static tool configs | ||
| } |
CopilotAIDec 12, 2025
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
This TODO test for tool conversion is critical for verifying that tools work correctly with the Azure Anthropic provider. Given that other similar providers in the codebase have this test implemented, this should be completed before merging to ensure feature parity and prevent potential issues with tool usage.
| // TODO: Implement test_initialize_anthropic_request_body | ||
| // This test should verify request body construction with various configurations | ||
| // See gcp_vertex_anthropic.rs for reference implementation | ||
| #[tokio::test] | ||
| async fn test_initialize_anthropic_request_body() { | ||
| // TODO: Test case 1: Empty message list (should error) | ||
| // TODO: Test case 2: Messages with system message | ||
| // TODO: Test case 3: Messages with temperature, top_p, max_tokens | ||
| // TODO: Test case 4: Tool use & choice | ||
| } |
CopilotAIDec 12, 2025
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
This TODO test for request body initialization should verify critical functionality including system messages, temperature, top_p, max_tokens, and tool use. These are core features that need testing before production use.
| // TODO: Implement test_prefill_json_message | ||
| // This test should verify JSON prefill functionality | ||
| #[test] | ||
| fn test_prefill_json_message() { | ||
| // TODO: Test that prefill_json_message adds an assistant message with "{" | ||
| } |
CopilotAIDec 12, 2025
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
This TODO test for JSON prefill functionality should verify that the JSON mode works correctly by adding the appropriate prefix. This is particularly important given the inconsistency with other Anthropic providers noted elsewhere.
| text: String, | ||
| }, | ||
| Thinking { | ||
| thinking: String, |
CopilotAIDec 12, 2025
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
The Thinking content block is missing the signature field that is present in the standard Anthropic provider. According to the Anthropic API, thinking blocks should include both the thinking text and a signature field. This omission could lead to incomplete data being captured from Azure Anthropic responses. The struct should match the standard AnthropicContentBlock which has both thinking and signature fields.
| thinking:String, | |
| thinking:String, | |
| signature:String, |
| AzureAnthropicContentBlock::Thinking { thinking } => { | ||
| Ok(ContentBlockOutput::Thought(Thought { | ||
| text: Some(thinking), | ||
| signature: None, | ||
| summary: None, | ||
| provider_type: Some(PROVIDER_TYPE.to_string()), | ||
| })) | ||
| } |
CopilotAIDec 12, 2025
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
The conversion of Thinking blocks sets signature to None, but the standard Anthropic provider expects a signature field. This inconsistency with the Anthropic API spec could result in data loss when Azure Anthropic returns thinking blocks with signatures.
| id: String, | ||
| name: String, | ||
| input: Value, | ||
| }, |
CopilotAIDec 12, 2025
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
The AzureAnthropicContentBlock enum is missing the RedactedThinking variant that exists in both the standard Anthropic and GCP Vertex Anthropic providers. If Azure Anthropic can return redacted thinking blocks (where the thinking content is hidden but metadata is provided), this provider will fail to deserialize such responses. Consider adding the RedactedThinking variant for API completeness.
| }, | |
| }, | |
| RedactedThinking{ | |
| reason:Option<String>, | |
| }, |
| fn prefill_json_message(mut messages: Vec<AnthropicMessage>) -> Vec<AnthropicMessage> { | ||
| messages.push(AnthropicMessage { | ||
| role: AnthropicRole::Assistant, | ||
| content: vec![FlattenUnknown::Normal(AnthropicMessageContent::Text { | ||
| text: "{", | ||
| })], | ||
| }); | ||
| messages | ||
| } |
CopilotAIDec 12, 2025
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
The JSON prefill message implementation differs from the standard Anthropic and GCP Vertex Anthropic providers. The standard prefill adds "Here is the JSON requested:\n{" while this implementation only adds "{". This inconsistency could lead to different behavior between providers when using JSON mode. Consider using the same prefill text as the other Anthropic-based providers for consistency.
| // TODO: Implement test_try_from_content_block | ||
| // This test should verify content block conversions (text, tool calls, tool results) | ||
| // See gcp_vertex_anthropic.rs for reference implementation | ||
| #[tokio::test] | ||
| async fn test_try_from_content_block() { | ||
| // TODO: Test text content block conversion | ||
| // TODO: Test tool call content block conversion | ||
| // TODO: Test tool result content block conversion | ||
| } |
CopilotAIDec 12, 2025
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
This TODO test for content block conversion is critical for ensuring that text content, tool calls, and tool results are correctly handled. These conversions are essential for proper functioning of the provider, especially when using tools or processing structured outputs.
| // TODO: Implement test_azure_anthropic_response_conversion | ||
| // This test should verify full response conversion from AzureAnthropicResponse | ||
| // See gcp_vertex_anthropic.rs for reference implementation | ||
| #[test] | ||
| fn test_azure_anthropic_response_conversion() { | ||
| // TODO: Test case 1: Text response | ||
| // TODO: Test case 2: Tool call response | ||
| // TODO: Test case 3: Thinking block response | ||
| // TODO: Test case 4: Mixed content response | ||
| } |
CopilotAIDec 12, 2025
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
This TODO test for response conversion should verify correct handling of text responses, tool calls, thinking blocks, and mixed content. This is important for ensuring that various response types from Azure Anthropic are properly converted to the internal format.
| // TODO: Implement test_azure_anthropic_apply_inference_params | ||
| // This test should verify inference params are applied correctly | ||
| #[test] | ||
| fn test_azure_anthropic_apply_inference_params() { | ||
| // TODO: Test thinking_budget_tokens is applied | ||
| // TODO: Test that reasoning_effort and service_tier emit warnings | ||
| } |
CopilotAIDec 12, 2025
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
This TODO test for inference params application should verify that thinking_budget_tokens is correctly applied and that unsupported parameters like reasoning_effort and service_tier emit appropriate warnings.
virajmehta commentedDec 12, 2025
Hi@hugentobler, thanks for the PR. I haven't reviewed thoroughly yet but just based on what you said I support the current approach. In my opinion, the less magical & stateful the integration the better. |
hugentobler commentedDec 12, 2025
Thanks@virajmehta so to confirm todos before I resume:
Do I need to generate any sdk type bindings? |
GabrielBianconi commentedDec 12, 2025
@hugentobler Does Azure Anthropic require a different API key vs. regular Azure? If it's re-usable, let's just re-use the same env var. If it's different, the naming convention above is fine. You'll likely need to generate bindings for TypeScript: This will force you to edit some assorted files like |
hugentobler commentedDec 13, 2025
@GabrielBianconi Probably it is using the same azure key. The reason I suggested another key is coz the current default azure key is |
GabrielBianconi commentedDec 13, 2025
@hugentobler Good point, I forgot that. Azure is actually deprecating "OpenAI Service", so might as well deprecate that in favor of You can use |
hugentobler commentedDec 13, 2025 • edited
Loading Uh oh!
There was an error while loading.Please reload this page.
edited
Uh oh!
There was an error while loading.Please reload this page.
@GabrielBianconi OK good I will catch up to#5178 and follow that style |
Uh oh!
There was an error while loading.Please reload this page.
Hi I started off on a new azure-anthropic provider, using a new azure /anthropic/messages endpoint. Borrowed heavily from gcp-vertex-anthropic provider example. Not yet added all the tests yet. Want to see if you guys agree with the direction first.
The solution I implemented is using the /anthropic/messages endpoint on Azure. It's basically a wrapper around vanilla Anthropic API request and response.
Azure also has a separate direction that supports a unified response format for both OpenAI and Anthropic models - using the Responses API.Here is an illustration of what that looks like.
What do you guys think. Look good to you? Then I will add all the tests.
Important
Add Azure Anthropic provider for handling API requests with support for non-streaming and streaming inference.
AzureAnthropicProviderinazure_anthropic.rsfor handling Azure Anthropic API requests./anthropic/messagesendpoint with Azure-specific configurations.AzureAnthropicProviderTypeConfigtoprovider_types.rs.AzureAnthropicCredentialsfor handling API keys with support for static, dynamic, and fallback credentials.ProviderConfiginmodel.rsto includeAzureAnthropic.AzureAnthropicKindinmodel_table.rsfor credential management.AzureAnthropicRequestBodyandget_default_max_tokens()inazure_anthropic.rs.This description was created by
forb313e34. You cancustomize this summary. It will automatically update as commits are pushed.