- Notifications
You must be signed in to change notification settings - Fork746
Validate endpoint response OpenAI openapi spec#5194
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
base:main
Are you sure you want to change the base?
Validate endpoint response OpenAI openapi spec#5194
Uh oh!
There was an error while loading.Please reload this page.
Conversation
…validate-endpoint-response-openai-openapi-spec
…validate-endpoint-response-openai-openapi-spec
…validate-endpoint-response-openai-openapi-spec
…validate-endpoint-response-openai-openapi-spec
…s://github.com/BrianLi23/tensorzero into validate-endpoint-response-openai-openapi-spec
…validate-endpoint-response-openai-openapi-spec
…s://github.com/BrianLi23/tensorzero into validate-endpoint-response-openai-openapi-spec
…validate-endpoint-response-openai-openapi-spec
…validate-endpoint-response-openai-openapi-spec
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Pull request overview
This PR adds comprehensive OpenAPI specification compliance validation for TensorZero's OpenAI-compatible inference endpoints. It downloads the official OpenAI OpenAPI spec and validates that TensorZero's responses conform to the schema definitions.
Key changes:
- Implements spec downloading, parsing, and sanitization logic to handle OpenAI spec quirks
- Adds validation tests for chat completions endpoint (
/v1/chat/completions) covering basic requests, tool calls, and various finish reasons - Adds validation tests for embeddings endpoint (
/v1/embeddings) with single and array inputs
Reviewed changes
Copilot reviewed 3 out of 4 changed files in this pull request and generated 21 comments.
| File | Description |
|---|---|
tensorzero-core/tests/e2e/openai_spec_compliance.rs | New test module with 411 lines implementing OpenAPI spec validation framework and compliance tests for chat completions and embeddings endpoints |
tensorzero-core/tests/e2e/tests.rs | Adds module declaration to include the new compliance test suite |
tensorzero-core/Cargo.toml | Addsserde_yml dev dependency for YAML parsing |
Cargo.lock | Updates lock file with newserde_yml andlibyml dependencies |
Uh oh!
There was an error while loading.Please reload this page.
Uh oh!
There was an error while loading.Please reload this page.
Uh oh!
There was an error while loading.Please reload this page.
Uh oh!
There was an error while loading.Please reload this page.
Uh oh!
There was an error while loading.Please reload this page.
Uh oh!
There was an error while loading.Please reload this page.
Uh oh!
There was an error while loading.Please reload this page.
Uh oh!
There was an error while loading.Please reload this page.
Uh oh!
There was an error while loading.Please reload this page.
Uh oh!
There was an error while loading.Please reload this page.
| /// Spec yaml publishes `seed.minimum` with a value smaller than i64::MIN. | ||
| /// We clamp the literal so serde_yml can parse the document | ||
| fn fix_yaml_parse_issues(raw: &str) -> Cow<'_, str> { | ||
| const OPENAI_SEED_MIN: &str = "-9223372036854776000"; | ||
| if raw.contains(OPENAI_SEED_MIN) { | ||
| Cow::Owned(raw.replace(OPENAI_SEED_MIN, "-9223372036854775808")) | ||
| } else { | ||
| Cow::Borrowed(raw) | ||
| } | ||
| } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Would e.g.i128 work?
Overview:
The endpoints to check on:
These tests will be wired to CI for additional validation in ensuring all related calls are in compliance (For now this will be done as non-blocking)
Resolves#2387