Building PyArrow#

This page provides source build instructions for PyArrow for all platforms.

System Requirements#

On macOS, any modern XCode or Xcode Command Line Tools (xcode-select--install)are sufficient.

On Linux, for this guide, we require a minimum of gcc or clang 9.You can check your version by running

$gcc--version

If the system compiler is older than gcc 9, it can be set to a newer versionusing the$CC and$CXX environment variables:

$exportCC=gcc-9$exportCXX=g++-9

Building on Windows requires one of the following compilers to beinstalled:

During the setup of Build Tools, ensure at least one Windows SDKis selected.

Environment setup#

First, start from a fresh clone of Apache Arrow:

$gitclonehttps://github.com/apache/arrow.git

There are two supported ways to set up the build environment for PyArrow: usingConda to manage the dependencies or usingpip with manual dependencymanagement.

Both methods are shown bellow for Linux and macOS. For Windows, only theConda-based setup is currently documented, skipping some of theLinux/macOS-only packages.

Note that in case you are not using conda on a Windows platform, Arrow C++libraries need to be bundled withpyarrow. For extra information see theWindows tab under theBuild PyArrow section.

Pull in the test data and setup the environment variables:

$pushdarrow$gitsubmoduleupdate--init$exportPARQUET_TEST_DATA="${PWD}/cpp/submodules/parquet-testing/data"$exportARROW_TEST_DATA="${PWD}/testing/data"$popd

Theconda package manager allows installing build-timedependencies for Arrow C++ and PyArrow as pre-built binaries, which can makeArrow development easier and faster.

Let’s create a conda environment with all the C++ build and Python dependenciesfrom conda-forge, targeting development for Python 3.13:

$condacreate-y-npyarrow-dev-cconda-forge\--filearrow/ci/conda_env_unix.txt\--filearrow/ci/conda_env_cpp.txt\--filearrow/ci/conda_env_python.txt\--filearrow/ci/conda_env_gandiva.txt\compilers\python=3.13\pandas

As of January 2019, thecompilers package is needed on many Linuxdistributions to use packages from conda-forge.

With this out of the way, you can now activate the conda environment

$condaactivatepyarrow-dev

We need to set some environment variables to let Arrow’s build system knowabout our build toolchain:

$exportARROW_HOME=$CONDA_PREFIX

Warning

If you installed Python using the Anaconda distribution orMiniconda, you cannot currently use apip-based virtual environment. Please follow the conda-based developmentinstructions instead.

Pull in the test data and setup the environment variables:

$pushdarrow$gitsubmoduleupdate--init$exportPARQUET_TEST_DATA="${PWD}/cpp/submodules/parquet-testing/data"$exportARROW_TEST_DATA="${PWD}/testing/data"$popd

Using system and bundled dependencies

If not using conda, you must arrange for your system to provide the requiredbuild tools and dependencies. Note that if some dependencies are absent,the Arrow C++ build chain may still be able to download and compile themon the fly, but this will take a longer time than with pre-installed binaries.

On macOS, use Homebrew to install all dependencies required forbuilding Arrow C++:

$brewupdate&&brewbundle--file=arrow/cpp/Brewfile

Seehere for a list of dependencies youmay need.

On Debian/Ubuntu, you need the following minimal set of dependencies:

$sudoapt-getinstallbuild-essentialninja-buildcmakepython3-dev

Now, let’s create a Python virtual environment with all Python dependenciesin the same folder as the repositories, and a target installation folder:

$python3-mvenvpyarrow-dev$source./pyarrow-dev/bin/activate$pipinstall-rarrow/python/requirements-build.txt$# This is the folder where we will install the Arrow libraries during$# development$mkdirdist

If your CMake version is too old on Linux, you could get a newer one viapipinstallcmake.

We need to set some environment variables to let Arrow’s build system knowabout our build toolchain:

$exportARROW_HOME=$(pwd)/dist$exportLD_LIBRARY_PATH=$(pwd)/dist/lib:$LD_LIBRARY_PATH$exportCMAKE_PREFIX_PATH=$ARROW_HOME:$CMAKE_PREFIX_PATH

Let’s create a conda environment with all the C++ build and Python dependenciesfrom conda-forge, targeting development for Python 3.13:

$condacreate-y-npyarrow-dev-cconda-forge^      --file arrow\ci\conda_env_cpp.txt ^      --file arrow\ci\conda_env_python.txt ^      --file arrow\ci\conda_env_gandiva.txt ^      python=3.13$condaactivatepyarrow-dev

Now, we can build and install Arrow C++ libraries.

We set the path of the installation directory of the Arrow C++libraries asARROW_HOME. When using a conda environment,Arrow C++ is installed in the environment directory, which pathis saved in theCONDA_PREFIXenvironment variable.

$setARROW_HOME=%CONDA_PREFIX%\Library

Build#

First we need to configure, build and install the Arrow C++ libraries.Then we can build PyArrow.

Build C++#

Now build the Arrow C++ libraries and install them into the directory wecreated above (stored in$ARROW_HOME):

$cmake-Sarrow/cpp-Barrow/cpp/build\-DCMAKE_INSTALL_PREFIX=$ARROW_HOME\--presetninja-release-python$cmake--buildarrow/cpp/build--targetinstall

About presets

ninja-release-python is not the only preset available - if you would like abuild with more features like CUDA, Flight and Gandiva support you may opt fortheninja-release-python-maximal preset. If you wanted less features, (i.e.removing ORC and dataset support) you could opt forninja-release-python-minimal. Changing the wordrelease todebugwith any of the aforementioned presets will generate a debug build of Arrow.

Individual components

The presets are provided as a convenience, but you may instead opt tospecify the individual components:

$cmake-Sarrow/cpp-Barrow/cpp/build\-DCMAKE_INSTALL_PREFIX=$ARROW_HOME\-DCMAKE_BUILD_TYPE=Debug\-DARROW_BUILD_TESTS=ON\-DARROW_COMPUTE=ON\-DARROW_CSV=ON\-DARROW_DATASET=ON\-DARROW_FILESYSTEM=ON\-DARROW_HDFS=ON\-DARROW_JSON=ON\-DARROW_PARQUET=ON\-DARROW_WITH_BROTLI=ON\-DARROW_WITH_BZ2=ON\-DARROW_WITH_LZ4=ON\-DARROW_WITH_SNAPPY=ON\-DARROW_WITH_ZLIB=ON\-DARROW_WITH_ZSTD=ON\-DPARQUET_REQUIRE_ENCRYPTION=ON$cmake--buildarrow/cpp/build--targetinstall-j4

If multiple versions of Python are installed in your environment, you may haveto pass additional parameters to CMake so that it can find the rightexecutable, headers and libraries. For example, specifying-DPython3_EXECUTABLE=<path/to/bin/python> lets CMake choose thePython executable which you are using.

Note

On Linux systems with support for building on multiple architectures,make may install libraries in thelib64 directory by default. Forthis reason we recommend passing-DCMAKE_INSTALL_LIBDIR=lib because thePython build scripts assume the library directory islib

Note

If you have conda installed but are not using it to manage dependencies,and you have trouble building the C++ library, you may need to set-DARROW_DEPENDENCY_SOURCE=AUTO or some other value (describedhere)to explicitly tell CMake not to use conda.

There are presets provided as a convenience for building C++ (see Linux and macOStab). Here we will instead specify the individual components:

$mkdirarrow\cpp\build$pushdarrow\cpp\build$cmake-G"Ninja"^      -DCMAKE_INSTALL_PREFIX=%ARROW_HOME% ^      -DCMAKE_UNITY_BUILD=ON ^      -DARROW_COMPUTE=ON ^      -DARROW_CSV=ON ^      -DARROW_CXXFLAGS="/WX /MP" ^      -DARROW_DATASET=ON ^      -DARROW_FILESYSTEM=ON ^      -DARROW_HDFS=ON ^      -DARROW_JSON=ON ^      -DARROW_PARQUET=ON ^      -DARROW_WITH_LZ4=ON ^      -DARROW_WITH_SNAPPY=ON ^      -DARROW_WITH_ZLIB=ON ^      -DARROW_WITH_ZSTD=ON ^      ..$cmake--build.--targetinstall--configRelease$popd

Optional build components#

There are several optional components that can be enabled or disabled by settingspecific flags toON orOFF, respectively. See the list ofRelevant components and environment variables below.

You may choose between different kinds of C++ build types:

  • -DCMAKE_BUILD_TYPE=Release (the default) produces a build with optimizationsenabled and debugging information disabled;

  • -DCMAKE_BUILD_TYPE=Debug produces a build with optimizationsdisabled and debugging information enabled;

  • -DCMAKE_BUILD_TYPE=RelWithDebInfo produces a build with both optimizationsand debugging information enabled.

See also

Building Arrow C++.

For any other C++ build challenges, seeC++ Development.

In case you may need to rebuild the C++ part due to errors in the process it isadvisable to delete the build folder, seeRelevant components and environment variables.If the build has passed successfully and you need to rebuild due to latest pullfrom git main, then this step is not needed.

Build PyArrow#

If you did build one of the optional components in C++, the equivalent componentswill be enabled by default for building pyarrow. This default can be overriddenby setting the correspondingPYARROW_WITH_$COMPONENT environment variableto 0 or 1, seeRelevant components and environment variables below.

To build PyArrow run:

$pushdarrow/python$pythonsetup.pybuild_ext--inplace$popd
$pushdarrow\python$pythonsetup.pybuild_ext--inplace$popd

Note

If you are using Conda with Python 3.9 or earlier, you mustsetCONDA_DLL_SEARCH_MODIFICATION_ENABLE=1.

Note

With the above instructions the Arrow C++ libraries are not bundled withthe Python extension. This is recommended for development as it allows theC++ libraries to be re-built separately.

If you are using the conda package manager then conda will ensure the Arrow C++libraries are found.In case you are NOT using conda then you have to:

  • add the path of installed DLL libraries toPATH every time beforeimportingpyarrow, or

  • bundle the Arrow C++ libraries withpyarrow.

Bundle Arrow C++ and PyArrow

If you want to bundle the Arrow C++ libraries withpyarrow, set thePYARROW_BUNDLE_ARROW_CPP environment variable before buildingpyarrow:

$setPYARROW_BUNDLE_ARROW_CPP=1$pythonsetup.pybuild_ext--inplace

Note that bundled Arrow C++ libraries will not be automaticallyupdated when rebuilding Arrow C++.

To set the number of threads used to compile PyArrow’s C++/Cython components,set thePYARROW_PARALLEL environment variable.

If you build PyArrow but then make changes to the Arrow C++ or PyArrow code,you can end up with stale build artifacts. This can lead tounexpected behavior or errors. To avoid this, you can clean the buildartifacts before rebuilding. SeeRelevant components and environment variables.

By default, PyArrow will be built in release mode even if Arrow C++ has beenbuilt in debug mode. To create a debug build of PyArrow, runexportPYARROW_BUILD_TYPE=debug prior to runningpythonsetup.pybuild_ext--inplace above. Arelwithdebinfo build can be createdsimilarly.

Self-Contained Wheel#

If you’re preparing a PyArrow wheel for distribution (e.g., for PyPI), you’llneed to build a self-contained wheel (including the Arrow and Parquet C++libraries). This ensures that all necessary native libraries are bundled insidethe wheel, so users can install it without needing to have Arrow or Parquetinstalled separately on their system.

To do this, pass the--bundle-arrow-cpp option to the build command:

$pipinstallwheel# if not installed$pythonsetup.pybuild_ext--build-type=$ARROW_BUILD_TYPE\--bundle-arrow-cppbdist_wheel

This option is typically only needed for releases or distribution scenarios,not for local development.

Editable install#

To install an editable PyArrow build, run the following command from thearrow/python directory:

pip install -e . --no-build-isolation``

This creates aneditable install, meaning changes to the Python source codewill be reflected immediately without needing to reinstall the package.The--no-build-isolation flag ensures that the build uses your currentenvironment’s dependencies instead of creating an isolated one. This isespecially useful during development and debugging.

Deleting stale build artifacts#

When there have been changes to the structure of the Arrow C++ library or PyArrow,a thorough cleaning is recommended as a first attempt to fixing build errors.

Note

It is not necessarily intuitive from the error itself that the problem is due to stale artifacts.Example of a build error from stale artifacts isUnknownCMakecommand"arrow_keep_backward_compatibility".

To delete stale Arrow C++ build artifacts:

$rm-rfarrow/cpp/build

To delete stale PyArrow build artifacts:

$gitclean-Xfdpython

If using a Conda environment, there are some build artifacts that get installed in$ARROW_HOME (aka$CONDA_PREFIX). For example,$ARROW_HOME/lib/cmake/Arrow*,$ARROW_HOME/include/arrow,$ARROW_HOME/lib/libarrow*, etc.

These files can be manually deleted. If unsure which files to erase, one approachis to recreate the Conda environment.

Either delete the current one, and start fresh:

$condadeactivate$condaremove-npyarrow-dev

Or, less destructively, create a different environment with a different name.

Docker examples#

If you are having difficulty building the Python library from source, take alook at thepython/examples/minimal_builddirectory which illustrates a complete build and test from source both withthe conda- and pip-based build methods.

Test#

Now you are ready to install test dependencies and runUnit Testing, asdescribed in development section.

Relevant components and environment variables#

List of relevant environment variables that can be used to buildPyArrow are:

PyArrow environment variable

Description

Default value

PYARROW_BUILD_TYPE

Build type for PyArrow (release, debug or relwithdebinfo), setsCMAKE_BUILD_TYPE

release

PYARROW_CMAKE_GENERATOR

Example:'VisualStudio172022Win64'

''

PYARROW_CMAKE_OPTIONS

Extra CMake and Arrow options (ex."-DARROW_SIMD_LEVEL=NONE-DCMAKE_OSX_ARCHITECTURES=x86_64;arm64")

''

PYARROW_CXXFLAGS

Extra C++ compiler flags

''

PYARROW_GENERATE_COVERAGE

SettingXlinetrace flag to true for the Cython compiler

false

PYARROW_BUNDLE_ARROW_CPP

Bundle the Arrow C++ libraries

0 (OFF)

PYARROW_BUNDLE_CYTHON_CPP

Bundle the C++ files generated by Cython

0 (OFF)

PYARROW_BUILD_VERBOSE

Enable verbose output from Makefile builds

0 (OFF)

PYARROW_PARALLEL

Number of processes used to compile PyArrow’s C++/Cython components

''

The components being disabled or enabled when building PyArrow is by defaultbased on how Arrow C++ is build (i.e. it follows theARROW_$COMPONENT flags).However, thePYARROW_WITH_$COMPONENT environment variables can still be usedto override this when building PyArrow (e.g. to disable components, or to enforcecertain components to be built):

Arrow flags/options

Corresponding environment variables for PyArrow

ARROW_GCS

PYARROW_WITH_GCS

ARROW_S3

PYARROW_WITH_S3

ARROW_AZURE

PYARROW_WITH_AZURE

ARROW_HDFS

PYARROW_WITH_HDFS

ARROW_CUDA

PYARROW_WITH_CUDA

ARROW_SUBSTRAIT

PYARROW_WITH_SUBSTRAIT

ARROW_FLIGHT

PYARROW_WITH_FLIGHT

ARROW_ACERO

PYARROW_WITH_ACERO

ARROW_DATASET

PYARROW_WITH_DATASET

ARROW_PARQUET

PYARROW_WITH_PARQUET

PARQUET_REQUIRE_ENCRYPTION

PYARROW_WITH_PARQUET_ENCRYPTION

ARROW_ORC

PYARROW_WITH_ORC

ARROW_GANDIVA

PYARROW_WITH_GANDIVA

Installing Nightly Packages#

Warning

These packages are not official releases. Use them at your own risk.

PyArrow has nightly wheels for testing purposes hosted atscientific-python-nightly-wheels.

These may be suitable for downstream libraries in their continuous integrationsetup to maintain compatibility with the upcoming PyArrow features,deprecations, and/or feature removals.

To install the most recent nightly version of PyArrow, run:

pipinstall\-ihttps://pypi.anaconda.org/scientific-python-nightly-wheels/simple\pyarrow