Feather File Format#

Feather is a portable file format for storing Arrow tables or data frames (fromlanguages like Python or R) that utilizes theArrow IPC formatinternally. Feather was created early in the Arrow project as a proof ofconcept for fast, language-agnostic data frame storage for Python (pandas) andR. There are two file format versions for Feather:

  • Version 2 (V2), the default version, which is exactly represented as theArrow IPC file format on disk. V2 files support storing all Arrow data typesas well as compression with LZ4 or ZSTD. V2 was first made available inApache Arrow 0.17.0.

  • Version 1 (V1), a legacy version available starting in 2016, replaced byV2. V1 files are distinct from Arrow IPC files and lack many features, suchas the ability to store all Arrow data types. V1 files also lack compressionsupport. We intend to maintain read support for V1 for the foreseeablefuture.

Thepyarrow.feather module contains the read and write functions for theformat.write_feather() accepts either aTable orpandas.DataFrame object:

importpyarrow.featherasfeatherfeather.write_feather(df,'/path/to/file')

read_feather() reads a Feather file as apandas.DataFrame.read_table() reads a Feather fileas aTable. Internally,read_feather()simply callsread_table() and the result is converted topandas:

# Result is pandas.DataFrameread_df=feather.read_feather('/path/to/file')# Result is pyarrow.Tableread_arrow=feather.read_table('/path/to/file')

These functions can read and write with file-paths or file-like objects. Forexample:

withopen('/path/to/file','wb')asf:feather.write_feather(df,f)withopen('/path/to/file','rb')asf:read_df=feather.read_feather(f)

A file input toread_feather must support seeking.

Using Compression#

As of Apache Arrow version 0.17.0, Feather V2 files (the default version)support two fast compression libraries, LZ4 (using the frame format) andZSTD. LZ4 is used by default if it is available (which it should be if youobtained pyarrow through a normal package manager):

# Uses LZ4 by defaultfeather.write_feather(df,file_path)# Use LZ4 explicitlyfeather.write_feather(df,file_path,compression='lz4')# Use ZSTDfeather.write_feather(df,file_path,compression='zstd')# Do not compressfeather.write_feather(df,file_path,compression='uncompressed')

Note that the default LZ4 compression generally yields much smaller fileswithout sacrificing much read or write performance. In some instances,LZ4-compressed files may be faster to read and write than uncompressed due toreduced disk IO requirements.

Writing Version 1 (V1) Files#

For compatibility with libraries without support for Version 2 files, you canwrite the version 1 format by passingversion=1 towrite_feather. Weintend to maintain read support for V1 for the foreseeable future.