Out-of-Core hybrid Apache Arrow/NumPy DataFrame for Python, ML, visualization and exploration of big tabular data at a billion rows per second 🚀

• UpdatedOct 8, 2024
• Python

the portable Python dataframe library

• UpdatedMar 18, 2025
• Python

Petastorm library enables single machine or distributed training and evaluation of deep learning models from datasets in Apache Parquet format. It supports ML frameworks such as Tensorflow, Pytorch, and PySpark and can be used from pure Python code.

• UpdatedDec 2, 2023
• Python

Lightweight and extensible compatibility layer between dataframe libraries!

• UpdatedMar 17, 2025
• Python

Work with bioinformatic files using Arrow, Polars, and/or DuckDB

• UpdatedMar 10, 2025
• Rust

Command-line interface to quickly generate fake CSV and JSON data

• UpdatedJul 11, 2024
• Python

Exploring Chicago crimes dataset with Jupyter notebooks, DuckDB, Malloy and new Panel/PyScript data and dashboard tools.

• UpdatedJan 29, 2023
• Jupyter Notebook

An open-source tool for reading OvertureMaps data with multiprocessing and additional Quality-of-Life features

• UpdatedMar 12, 2025
• Python

Type annotations for pyarrow

• UpdatedMar 17, 2025
• Python

Converts a whole subdirectory with a big (or small) volume of PDF documents to a dataset (pandas DataFrame) with error tracking and choice of features

• UpdatedJan 9, 2025
• Python

db2ixf is a python package with a CLI that simplifies the parsing and processing of IBM Integration eXchange Format (IXF) files.

• UpdatedMar 16, 2024
• Python

(PoC) A very memory-efficient way to read data from PostgreSQL

• UpdatedOct 28, 2022
• Rust

A web application for viewing Apache Parquet files . This is a Python + Flask application

• UpdatedApr 17, 2018
• HTML

Reading both XLSX and XLSB files, fast and memory-safe, with Python, into PyArrow

• UpdatedFeb 6, 2024
• Jupyter Notebook

Poor mans simple python api for creating a local or remote datalake based on several (pyarrow) datasets using duckdb

• UpdatedJul 14, 2023
• Python

Seamlessly switch Pandas DataFrame backend to PyArrow.

• UpdatedMar 17, 2025
• Python

poor man´s data lake - Simple api to efficiently query your parquet datasets using Duckdb or polars

• UpdatedFeb 20, 2025
• Python

Converts AsyncApi and JsonSchema to PyArrow schema

• UpdatedFeb 11, 2025
• Python

SQL2Arrow, short for 'SQL to Arrow,' is a Python library that provides convenient and high-performance methods to parse INSERT SQL statements into Arrow arrays. It is particularly useful for analyzing data dumped by mysqldump or other tools.

• UpdatedDec 31, 2024
• Rust

Python scripts to process, and analyze log files using PySpark.

• UpdatedJul 13, 2024
• Python