Repository files navigation

Welcome to theRust implementation ofApache Arrow, the popular in-memory columnar format.

This repository contains the following crates:

The current development version the API documentation in this repo can be foundhere.

Note: previously theobject_store crate was also part of this repository,but it has been moved to thearrow-rs-object-store repository

The Arrow Rust project releases approximately monthly and followsSemanticVersioning.

Due to available maintainer and testing bandwidth,arrow crates (arrow,arrow-flight, etc.) are released on the same schedule with the same versionsas theparquet and [parquet-derive] crates.

This crate releases every month. We release new major versions (with potentiallybreaking API changes) at most once a quarter, and release incremental minorversions in the intervening months. Seeticket #5368 for more details.

To keep our maintenance burden down, we do regularly scheduled releases (majorand minor) from themain branch. How we handle PRs with breaking API changesis described in thecontributing guide.

Planned Release Schedule

arrow-rs and parquet are built and tested with stable Rust, and will keep a rolling MSRV (minimum supported Rust version) that can only be updated in major releases on a need by basis (e.g. project dependencies bump their MSRV or a particular Rust feature is useful for us etc.). The new MSRV if selected will be at least 6 months old. The minor releases are guaranteed to have the same MSRV.

Note: If a Rust hotfix is released for the current MSRV, the MSRV will be updated to the specific minor version that includes all applicable hotfixes preceding other policies.

In general, use panics for bad states that are unreachable, unrecoverable or harmful.For those caused by invalid user input, however, we prefer to report that invaliditygracefully as an error result instead of panicking. In general, invalid input should resultin anError as soon as possible. Itis ok for code paths after validation to assumevalidation has already occurred and panic if not. Seeticket #6737 for more nuances.

Minor releases may deprecate, but not remove APIs. Deprecating APIs allowsdownstream Rust programs to still compile, but generate compiler warnings. Thisgives downstream crates time to migrate prior to API removal.

To deprecate an API:

• Mark the API as deprecated using#[deprecated] and specify the exact arrow-rs version in which it was deprecated
• Concisely describe the preferred API to help the user transition

The deprecated version is the next version which will be released (pleaseconsult the list above). To mark the API as deprecated, use the#[deprecated(since = "...", note = "...")] attribute.

Foe example

#[deprecated(since ="51.0.0", note ="Use `date_part` instead")]

In general, deprecated APIs will remain in the codebase for at least two major releases afterthey were deprecated (typically between 6 - 9 months later). For example, an APIdeprecated in51.3.0 can be removed in54.0.0 (or later). Deprecated APIsmay be removed earlier or later than these guidelines at the discretion of themaintainers.

There are several related crates in different repositories

Collectively, these crates support a wider array of functionality for analytic computations in Rust.

For example, you can write SQL queries or aDataFrame (using thedatafusion crate) to read a parquet file (using theparquet crate),evaluate it in-memory using Arrow's columnar format (using thearrow crate),and send to another process (using thearrow-flight crate).

Generally speaking, thearrow crate offers functionality for using Arrowarrays, anddatafusion offers most operations typically found in SQL,includingjoins and window functions.

You can find more details about each crate in their respective READMEs.

Thedev@arrow.apache.org mailing list serves as the core communication channel for the Arrow community. Instructions for signing up and links to the archives can be found on theArrow Community page. All major announcements and communications happen there.

The Rust Arrow community also uses the officialASF Slack for informal discussions and coordination. This isa great place to meet other contributors and get guidance on where to contribute. Join us in the#arrow-rust channel and feel free to ask for an invite via:

1. thedev@arrow.apache.org mailing list
2. theGitHub Discussions
3. theDiscord channel

The Rust implementation usesGitHub issues as the system of record for new features and bug fixes andthis plays a critical role in the release process.

For design discussions we generally use GitHub issues.

There is more information in thecontributing guide.