Repository files navigation

Join the community on Slack! |Documentation |Performance Benchmarks

Vortex is a next-generation columnar file format and toolkit designed for high-performance data processing.It is the fastest and most extensible format for building data systems backed by object storage. It provides:

• Blazing Fast Performance

  • 100x faster random access reads (vs. modern Apache Parquet)
  • 10-20x faster scans
  • 5x faster writes
  • Similar compression ratios
  • Efficient support for wide tables with zero-copy/zero-parse metadata

• Extensible Architecture

  • Modeled after Apache DataFusion's extensible approach
  • Pluggable encoding system, type system, compression strategy, & layout strategy
  • Zero-copy compatibility with Apache Arrow

• Open Source, Neutral Governance

  • A Linux Foundation (LF AI & Data) Project
  • Apache-2.0 Licensed

• Integrations

  • Arrow, DataFusion, DuckDB, Spark, Pandas, Polars, & more
  • Apache Iceberg (coming soon)

🟢Development Status: Library APIs may change from version to version, but we now considerthe file formatstable. From release 0.36.0, all future releases of Vortex shouldmaintain backwards compatibility of the file format (i.e., be able to read files written byany earlier version >= 0.36.0).

• Logical Types - Clean separation between logical schema and physical layout
• Zero-Copy Arrow Integration - Seamless conversion to/from Apache Arrow arrays
• Extensible Encodings - Pluggable physical layouts with built-in optimizations
• Cascading Compression - Support for nested encoding schemes
• High-Performance Computing - Optimized compute kernels for encoded data
• Rich Statistics - Lazy-loaded summary statistics for optimization

Vortex strictly separates logical and physical concerns:

• Logical Layer: Defines data types and schema
• Physical Layer: Handles encoding and storage implementation
• Built-in Encodings: Compatible with Apache Arrow's memory format
• Extension Encodings: Optimized compression schemes (RLE, dictionary, etc.)

All features are exported through the mainvortex crate.

cargo add vortex

uv add vortex-data

For browsing the structure of Vortex files, you can use thevx command-line tool.

# Install latest releasecargo install vortex-tui --locked# Or build from sourcecargo install --path vortex-tui --locked# Usagevx browse<file>

# Optional but recommended dependenciesbrew install flatbuffers protobuf# For .fbs and .proto filesbrew install duckdb# For benchmarks# Install Rust toolchaincurl --proto'=https' --tlsv1.2 -sSf https://sh.rustup.rs| sh# orbrew install rustup# Initialize submodulesgit submodule update --init --recursive# Setup dependencies with uvuv sync --all-packages

For optimal performance, we suggest usingMiMalloc:

#[global_allocator]staticGLOBAL_ALLOC:MiMalloc =MiMalloc;

Licensed under the Apache License, Version 2.0.

Vortex is an independent open-source project and not controlled by any single company. The Vortex Project is asub-project of the Linux Foundation Projects. The governance model is documented inCONTRIBUTING.md and is subject to the terms oftheTechnical Charter.

SeeCONTRIBUTING.md for guidelines.

If you discover a security vulnerability, please emailvuln-report@vortex.dev.

Copyright © Vortex a Series of LF Projects, LLC.For terms of use, trademark policy, and other project policies please seehttps://lfprojects.org

The Vortex project benefits enormously from groundbreaking work from the academic & open-source communities.

• BtrBlocks - Efficient columnar compression
• FastLanes &FastLanes on GPU - High-performance integer compression
• FSST - Fast random access string compression
• ALP &G-ALP - Adaptive lossless floating-point compression
• Procella - YouTube's unified data system
• Anyblob - High-performance access to object storage
• ClickHouse - Fast analytics for everyone
• MonetDB/X100 - Hyper-Pipelining Query Execution
• Morsel-Driven Parallelism: A NUMA-Aware Query Evaluation Format for the Many-Core Age
• The FastLanes File Format - Expression Operators

• Anyblox - A Framework for Self-Decoding Datasets
• F3 - Open-Source Data File Format for the Future

• Apache Arrow
• Apache DataFusion
• parquet2 by Jorge Leitao
• DuckDB
• Velox &Nimble