Repository files navigation

Website |API Docs |Chat

DataFusion is an extensible query engine written inRust thatusesApache Arrow as its in-memory format.

This crate provides libraries and binaries for developers building fast andfeature rich database and analytic systems, customized to particular workloads.Seeuse cases for examples. The following related subprojects target end users:

• DataFusion Python offers a Python interface for SQL and DataFramequeries.
• DataFusion Ray provides a distributed version of DataFusion that scalesout on Ray clusters.
• DataFusion Comet is an accelerator for Apache Spark based onDataFusion.

"Out of the box,"DataFusion offers [SQL] and [Dataframe] APIs, excellentperformance,built-in support for CSV, Parquet, JSON, and Avro, extensive customization, anda great community.

DataFusion features a full query planner, a columnar, streaming, multi-threaded,vectorized execution engine, and partitioned data sources. You cancustomize DataFusion at almost all points including additional data sources,query languages, functions, custom operators and more.See theArchitecture section for more details.

Here are links to some important information

• Project Site
• Installation
• Rust Getting Started
• Rust DataFrame API
• Rust API docs
• Rust Examples
• Python DataFrame API
• Architecture

DataFusion is great for building projects such as domain specific query engines, new database platforms and data pipelines, query languages and more.It lets you start quickly from a fully working engine, and then customize those features specific to your use.Click Here to see a list known users.

Please see thecontributor guide andcommunication pages for more information.

This crate has severalfeatures which can be specified in yourCargo.toml.

Default features:

• nested_expressions: functions for working with nested type function such asarray_to_string
• compression: reading files compressed withxz2,bzip2,flate2, andzstd
• crypto_expressions: cryptographic functions such asmd5 andsha256
• datetime_expressions: date and time functions such asto_timestamp
• encoding_expressions:encode anddecode functions
• parquet: support for reading theApache Parquet format
• parquet_encryption: support for usingParquet Modular Encryption
• regex_expressions: regular expression functions, such asregexp_match
• unicode_expressions: Include unicode aware functions such ascharacter_length
• unparser: enables support to reverse LogicalPlans back into SQL
• recursive_protection: usesrecursive for stack overflow protection.

Optional features:

• avro: support for reading theApache Avro format
• backtrace: include backtrace information in error messages
• pyarrow: conversions between PyArrow and DataFusion types
• serde: enable arrow-schema'sserde feature

Public methods in Apache DataFusion evolve over time: while we try to maintain astable API, we also improve the API over time. As a result, we typicallydeprecate methods before removing them, according to thedeprecation guidelines.

Following theguidance on committingCargo.lock files, this project commitsitsCargo.lock file.

CI uses the committedCargo.lock file, and dependencies are updated regularlyusingDependabot PRs.