- Notifications
You must be signed in to change notification settings - Fork715
feat: upgrade datafusion to 51#9202
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
Uh oh!
There was an error while loading.Please reload this page.
Conversation
Failed to generate code suggestions for PR |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Pull request overview
This pull request upgrades the DataFusion query engine from version 50.3.0 to 51.0.0, along with related dependencies including Arrow (56→57), Parquet (56→57), and various OpenTelemetry/Tonic libraries. The upgrade adapts to breaking API changes in the DataFusion ecosystem, particularly around physical plan construction, codec interfaces, and configuration.
Key Changes:
- Updated DataFusion APIs for physical plan codecs (SessionContext → TaskContext)
- Adapted to fallible
UnionExec::try_new()constructor - Updated protobuf/tonic stack from 0.13 to 0.14
- Fixed return value handling for
span.set_parent()
Reviewed changes
Copilot reviewed 34 out of 37 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| Cargo.toml, Cargo.lock | Upgraded DataFusion, Arrow, Parquet, Tonic, and OpenTelemetry dependencies |
| src/service/search/datafusion/exec.rs | Updated SQL dialect configuration to use enum instead of string |
| src/service/search/datafusion/table_provider/*.rs | Updated physical plan APIs and ParquetFileReaderFactory signature |
| src/service/search/datafusion/distributed_plan/codec/*.rs | Migrated from FunctionRegistry to TaskContext in codec interfaces |
| src/service/search/grpc/flight.rs | Updated physical plan deserialization to use TaskContext |
| src/flight/src/*.rs | Updated IPC encoding to use CompressionContext |
| src/proto/build.rs | Switched from tonic::codec to tonic_prost for protobuf encoding |
| src/handler/grpc/**/*.rs | Fixed span.set_parent() return value handling |
| src/infra/src/storage/mod.rs | Updated parquet metadata API to use FooterTail |
💡Add Copilot custom instructions for smarter, more guided reviews.Learn how to get started.
9190a9d intomainUh oh!
There was an error while loading.Please reload this page.
Uh oh!
There was an error while loading.Please reload this page.