Building PyArrow#
This page provides source build instructions for PyArrow for all platforms.
System Requirements#
On macOS, any modern XCode or Xcode Command Line Tools (xcode-select--install)are sufficient.
On Linux, for this guide, we require a minimum of gcc or clang 9.You can check your version by running
$gcc--versionIf the system compiler is older than gcc 9, it can be set to a newer versionusing the$CC and$CXX environment variables:
$exportCC=gcc-9$exportCXX=g++-9
Building on Windows requires one of the following compilers to beinstalled:
Visual Studio 2022
During the setup of Build Tools, ensure at least one Windows SDKis selected.
Environment setup#
First, start from a fresh clone of Apache Arrow:
$gitclonehttps://github.com/apache/arrow.gitThere are two supported ways to set up the build environment for PyArrow: usingConda to manage the dependencies or usingpip with manual dependencymanagement.
Both methods are shown bellow for Linux and macOS. For Windows, only theConda-based setup is currently documented, skipping some of theLinux/macOS-only packages.
Note that in case you are not using conda on a Windows platform, Arrow C++libraries need to be bundled withpyarrow. For extra information see theWindows tab under theBuild PyArrow section.
Pull in the test data and setup the environment variables:
$pushdarrow$gitsubmoduleupdate--init$exportPARQUET_TEST_DATA="${PWD}/cpp/submodules/parquet-testing/data"$exportARROW_TEST_DATA="${PWD}/testing/data"$popd
Theconda package manager allows installing build-timedependencies for Arrow C++ and PyArrow as pre-built binaries, which can makeArrow development easier and faster.
Let’s create a conda environment with all the C++ build and Python dependenciesfrom conda-forge, targeting development for Python 3.13:
$condacreate-y-npyarrow-dev-cconda-forge\--filearrow/ci/conda_env_unix.txt\--filearrow/ci/conda_env_cpp.txt\--filearrow/ci/conda_env_python.txt\--filearrow/ci/conda_env_gandiva.txt\compilers\python=3.13\pandas
As of January 2019, thecompilers package is needed on many Linuxdistributions to use packages from conda-forge.
With this out of the way, you can now activate the conda environment
$condaactivatepyarrow-devWe need to set some environment variables to let Arrow’s build system knowabout our build toolchain:
$exportARROW_HOME=$CONDA_PREFIX
Warning
If you installed Python using the Anaconda distribution orMiniconda, you cannot currently use apip-based virtual environment. Please follow the conda-based developmentinstructions instead.
Pull in the test data and setup the environment variables:
$pushdarrow$gitsubmoduleupdate--init$exportPARQUET_TEST_DATA="${PWD}/cpp/submodules/parquet-testing/data"$exportARROW_TEST_DATA="${PWD}/testing/data"$popd
Using system and bundled dependencies
If not using conda, you must arrange for your system to provide the requiredbuild tools and dependencies. Note that if some dependencies are absent,the Arrow C++ build chain may still be able to download and compile themon the fly, but this will take a longer time than with pre-installed binaries.
On macOS, use Homebrew to install all dependencies required forbuilding Arrow C++:
$brewupdate&&brewbundle--file=arrow/cpp/Brewfile
Seehere for a list of dependencies youmay need.
On Debian/Ubuntu, you need the following minimal set of dependencies:
$sudoapt-getinstallbuild-essentialninja-buildcmakepython3-devNow, let’s create a Python virtual environment with all Python dependenciesin the same folder as the repositories, and a target installation folder:
$python3-mvenvpyarrow-dev$source./pyarrow-dev/bin/activate$pipinstall-rarrow/python/requirements-build.txt$# This is the folder where we will install the Arrow libraries during$# development$mkdirdist
If your CMake version is too old on Linux, you could get a newer one viapipinstallcmake.
We need to set some environment variables to let Arrow’s build system knowabout our build toolchain:
$exportARROW_HOME=$(pwd)/dist$exportLD_LIBRARY_PATH=$(pwd)/dist/lib:$LD_LIBRARY_PATH$exportCMAKE_PREFIX_PATH=$ARROW_HOME:$CMAKE_PREFIX_PATH
Let’s create a conda environment with all the C++ build and Python dependenciesfrom conda-forge, targeting development for Python 3.13:
$condacreate-y-npyarrow-dev-cconda-forge^ --file arrow\ci\conda_env_cpp.txt ^ --file arrow\ci\conda_env_python.txt ^ --file arrow\ci\conda_env_gandiva.txt ^ python=3.13$condaactivatepyarrow-dev
Now, we can build and install Arrow C++ libraries.
We set the path of the installation directory of the Arrow C++libraries asARROW_HOME. When using a conda environment,Arrow C++ is installed in the environment directory, which pathis saved in theCONDA_PREFIXenvironment variable.
$setARROW_HOME=%CONDA_PREFIX%\Library
Build#
First we need to configure, build and install the Arrow C++ libraries.Then we can build PyArrow.
Build C++#
Now build the Arrow C++ libraries and install them into the directory wecreated above (stored in$ARROW_HOME):
$cmake-Sarrow/cpp-Barrow/cpp/build\-DCMAKE_INSTALL_PREFIX=$ARROW_HOME\--presetninja-release-python$cmake--buildarrow/cpp/build--targetinstall
About presets
ninja-release-python is not the only preset available - if you would like abuild with more features like CUDA, Flight and Gandiva support you may opt fortheninja-release-python-maximal preset. If you wanted less features, (i.e.removing ORC and dataset support) you could opt forninja-release-python-minimal. Changing the wordrelease todebugwith any of the aforementioned presets will generate a debug build of Arrow.
Individual components
The presets are provided as a convenience, but you may instead opt tospecify the individual components:
$cmake-Sarrow/cpp-Barrow/cpp/build\-DCMAKE_INSTALL_PREFIX=$ARROW_HOME\-DCMAKE_BUILD_TYPE=Debug\-DARROW_BUILD_TESTS=ON\-DARROW_COMPUTE=ON\-DARROW_CSV=ON\-DARROW_DATASET=ON\-DARROW_FILESYSTEM=ON\-DARROW_HDFS=ON\-DARROW_JSON=ON\-DARROW_PARQUET=ON\-DARROW_WITH_BROTLI=ON\-DARROW_WITH_BZ2=ON\-DARROW_WITH_LZ4=ON\-DARROW_WITH_SNAPPY=ON\-DARROW_WITH_ZLIB=ON\-DARROW_WITH_ZSTD=ON\-DPARQUET_REQUIRE_ENCRYPTION=ON$cmake--buildarrow/cpp/build--targetinstall-j4
If multiple versions of Python are installed in your environment, you may haveto pass additional parameters to CMake so that it can find the rightexecutable, headers and libraries. For example, specifying-DPython3_EXECUTABLE=<path/to/bin/python> lets CMake choose thePython executable which you are using.
Note
On Linux systems with support for building on multiple architectures,make may install libraries in thelib64 directory by default. Forthis reason we recommend passing-DCMAKE_INSTALL_LIBDIR=lib because thePython build scripts assume the library directory islib
Note
If you have conda installed but are not using it to manage dependencies,and you have trouble building the C++ library, you may need to set-DARROW_DEPENDENCY_SOURCE=AUTO or some other value (describedhere)to explicitly tell CMake not to use conda.
There are presets provided as a convenience for building C++ (see Linux and macOStab). Here we will instead specify the individual components:
$mkdirarrow\cpp\build$pushdarrow\cpp\build$cmake-G"Ninja"^ -DCMAKE_INSTALL_PREFIX=%ARROW_HOME% ^ -DCMAKE_UNITY_BUILD=ON ^ -DARROW_COMPUTE=ON ^ -DARROW_CSV=ON ^ -DARROW_CXXFLAGS="/WX /MP" ^ -DARROW_DATASET=ON ^ -DARROW_FILESYSTEM=ON ^ -DARROW_HDFS=ON ^ -DARROW_JSON=ON ^ -DARROW_PARQUET=ON ^ -DARROW_WITH_LZ4=ON ^ -DARROW_WITH_SNAPPY=ON ^ -DARROW_WITH_ZLIB=ON ^ -DARROW_WITH_ZSTD=ON ^ ..$cmake--build.--targetinstall--configRelease$popd
Optional build components#
There are several optional components that can be enabled or disabled by settingspecific flags toON orOFF, respectively. See the list ofRelevant components and environment variables below.
You may choose between different kinds of C++ build types:
-DCMAKE_BUILD_TYPE=Release(the default) produces a build with optimizationsenabled and debugging information disabled;-DCMAKE_BUILD_TYPE=Debugproduces a build with optimizationsdisabled and debugging information enabled;-DCMAKE_BUILD_TYPE=RelWithDebInfoproduces a build with both optimizationsand debugging information enabled.
In case you may need to rebuild the C++ part due to errors in the process it isadvisable to delete the build folder, seeRelevant components and environment variables.If the build has passed successfully and you need to rebuild due to latest pullfrom git main, then this step is not needed.
Build PyArrow#
If you did build one of the optional components in C++, the equivalent componentswill be enabled by default for building pyarrow. This default can be overriddenby setting the correspondingPYARROW_WITH_$COMPONENT environment variableto 0 or 1, seeRelevant components and environment variables below.
To build PyArrow run:
$pushdarrow/python$pythonsetup.pybuild_ext--inplace$popd
$pushdarrow\python$pythonsetup.pybuild_ext--inplace$popd
Note
If you are using Conda with Python 3.9 or earlier, you mustsetCONDA_DLL_SEARCH_MODIFICATION_ENABLE=1.
Note
With the above instructions the Arrow C++ libraries are not bundled withthe Python extension. This is recommended for development as it allows theC++ libraries to be re-built separately.
If you are using the conda package manager then conda will ensure the Arrow C++libraries are found.In case you are NOT using conda then you have to:
add the path of installed DLL libraries to
PATHevery time beforeimportingpyarrow, orbundle the Arrow C++ libraries with
pyarrow.
Bundle Arrow C++ and PyArrow
If you want to bundle the Arrow C++ libraries withpyarrow, set thePYARROW_BUNDLE_ARROW_CPP environment variable before buildingpyarrow:
$setPYARROW_BUNDLE_ARROW_CPP=1$pythonsetup.pybuild_ext--inplace
Note that bundled Arrow C++ libraries will not be automaticallyupdated when rebuilding Arrow C++.
To set the number of threads used to compile PyArrow’s C++/Cython components,set thePYARROW_PARALLEL environment variable.
If you build PyArrow but then make changes to the Arrow C++ or PyArrow code,you can end up with stale build artifacts. This can lead tounexpected behavior or errors. To avoid this, you can clean the buildartifacts before rebuilding. SeeRelevant components and environment variables.
By default, PyArrow will be built in release mode even if Arrow C++ has beenbuilt in debug mode. To create a debug build of PyArrow, runexportPYARROW_BUILD_TYPE=debug prior to runningpythonsetup.pybuild_ext--inplace above. Arelwithdebinfo build can be createdsimilarly.
Self-Contained Wheel#
If you’re preparing a PyArrow wheel for distribution (e.g., for PyPI), you’llneed to build a self-contained wheel (including the Arrow and Parquet C++libraries). This ensures that all necessary native libraries are bundled insidethe wheel, so users can install it without needing to have Arrow or Parquetinstalled separately on their system.
To do this, pass the--bundle-arrow-cpp option to the build command:
$pipinstallwheel# if not installed$pythonsetup.pybuild_ext--build-type=$ARROW_BUILD_TYPE\--bundle-arrow-cppbdist_wheel
This option is typically only needed for releases or distribution scenarios,not for local development.
Editable install#
To install an editable PyArrow build, run the following command from thearrow/python directory:
pip install -e . --no-build-isolation``This creates aneditable install, meaning changes to the Python source codewill be reflected immediately without needing to reinstall the package.The--no-build-isolation flag ensures that the build uses your currentenvironment’s dependencies instead of creating an isolated one. This isespecially useful during development and debugging.
Deleting stale build artifacts#
When there have been changes to the structure of the Arrow C++ library or PyArrow,a thorough cleaning is recommended as a first attempt to fixing build errors.
Note
It is not necessarily intuitive from the error itself that the problem is due to stale artifacts.Example of a build error from stale artifacts isUnknownCMakecommand"arrow_keep_backward_compatibility".
To delete stale Arrow C++ build artifacts:
$rm-rfarrow/cpp/buildTo delete stale PyArrow build artifacts:
$gitclean-XfdpythonIf using a Conda environment, there are some build artifacts that get installed in$ARROW_HOME (aka$CONDA_PREFIX). For example,$ARROW_HOME/lib/cmake/Arrow*,$ARROW_HOME/include/arrow,$ARROW_HOME/lib/libarrow*, etc.
These files can be manually deleted. If unsure which files to erase, one approachis to recreate the Conda environment.
Either delete the current one, and start fresh:
$condadeactivate$condaremove-npyarrow-dev
Or, less destructively, create a different environment with a different name.
Docker examples#
If you are having difficulty building the Python library from source, take alook at thepython/examples/minimal_builddirectory which illustrates a complete build and test from source both withthe conda- and pip-based build methods.
Test#
Now you are ready to install test dependencies and runUnit Testing, asdescribed in development section.
Relevant components and environment variables#
List of relevant environment variables that can be used to buildPyArrow are:
PyArrow environment variable | Description | Default value |
|---|---|---|
| Build type for PyArrow (release, debug or relwithdebinfo), sets |
|
| Example: |
|
| Extra CMake and Arrow options (ex. |
|
| Extra C++ compiler flags |
|
| Setting |
|
| Bundle the Arrow C++ libraries |
|
| Bundle the C++ files generated by Cython |
|
| Enable verbose output from Makefile builds |
|
| Number of processes used to compile PyArrow’s C++/Cython components |
|
The components being disabled or enabled when building PyArrow is by defaultbased on how Arrow C++ is build (i.e. it follows theARROW_$COMPONENT flags).However, thePYARROW_WITH_$COMPONENT environment variables can still be usedto override this when building PyArrow (e.g. to disable components, or to enforcecertain components to be built):
Arrow flags/options | Corresponding environment variables for PyArrow |
|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Installing Nightly Packages#
Warning
These packages are not official releases. Use them at your own risk.
PyArrow has nightly wheels for testing purposes hosted atscientific-python-nightly-wheels.
These may be suitable for downstream libraries in their continuous integrationsetup to maintain compatibility with the upcoming PyArrow features,deprecations, and/or feature removals.
To install the most recent nightly version of PyArrow, run:
pipinstall\-ihttps://pypi.anaconda.org/scientific-python-nightly-wheels/simple\pyarrow

