Building the Arrow libraries 🏋🏿♀️#
The Arrow project contains a number of libraries that enablework in many languages. Most libraries (C++, C#, Go, Java,JavaScript, Julia, and Rust) already contain distinct implementationsof Arrow.
This is different for C (Glib), MATLAB, Python, R, and Ruby as theyare built on top of the C++ library. In this section of the guidewe will try to make a friendly introduction to the build,dealing with some of these libraries as well has how they work withthe C++ library.
If you decide to contribute to Arrow you might need to compile theC++ source code. This is done using a tool called CMake, which youmay or may not have experience with. If not, this section of theguide will help you better understand CMake and the processof building Arrow’s C++ code.
This content is intended to help explain the concepts related toand tools required for building Arrow’s C++ library from source.If you are looking for the specific required steps, or already feel comfortablewith compiling Arrow’s C++ library, then feel free to proceedto theC++,PyArrow orR package build section.
Building Arrow C++#
Why build Arrow C++ from source?#
For Arrow implementations which are built on top of the C++ implementation(e.g. Python and R), wrappers and interfaces have been written to theunderlying C++ functions. If you want to work on PyArrow or the R package,you may need to edit the source code of the C++ library too.
Detailed instructions on building C++ library from source canbe foundhere.
About CMake#
CMake is a cross-platform build system generator and it defersto another program such asmake orninja for the actual build.If you are running into errors with the build process, the first thing todo is to look at the error message thoroughly and check the buildingdocumentation for any similar error advice. Also changing the CMake flagsfor compiling Arrow could be useful.
CMake presets#
You could also try to build with CMake presets which are a collection ofbuild and test recipes for Arrow’s CMake. They are a very usefulstarting points.
More detailed information about CMake presets can be found intheCMake presets section.
Optional flags and environment variables#
Flags used in the CMake build are used to include additional componentsand to handle third-party dependencies.The build for C++ library can be minimal with no use of flags or canbe changed with adding optional components from thelist.
See also
Full list of optional flags:Optional Components
R and Python have specific lists of flags in their respective buildsthat need to be included. You can find the links at the endof this section.
In general on Python side, the options are set with CMake flags andpaths with environment variables. In R the environment variables are usedfor all things connected to the build, also for setting CMake flags.
Building other Arrow libraries#
After building the Arrow C++ library, you need to build PyArrow on topof it also. The reason is the same; so you can edit the code and runtests on the edited code you have locally.
Why do we have to do builds separately?
As mentioned at the beginning of this page, the Python part of the Arrowproject is built on top of the C++ library. In order to make changes inthe Python part of Arrow as well as the C++ part of Arrow, you need tobuild them separately.
We hope this introduction was enough to help you start with the buildingprocess.
When you will make change to the code, you may need to recompilePyArrow or Arrow C++:
Recompiling Cython
If you only make changes to.py files, you do not need torecompile PyArrow. However, you should recompile it if you makechanges in.pyx or.pxd files.
To do that run this command again:
$pythonsetup.pybuild_ext--inplaceRecompiling C++
Similarly, you will need to recompile the C++ code if you havemade changes to any C++ files. In this case,re-run the build commands again.
When working on code in the R package, depending on your OS and plannedchanges, you may or may not need to build the Arrow C++ library (oftenreferred to in the R documentation as ‘libarrow’) from source.
More information on this and full instructions on setting up the Arrow C++library and Arrow R package can be found in theR developer docs.
Reinstalling R package and running ‘make clean’
If you make changes to the Arrow C++ part of the code, alsocalled libarrow, you will need to:
reinstall libarrow,
run
makeclean,reinstall the R package.
Themakeclean function is defined inr/Makefile and willremove any cached object code in ther/src/ directory, ensuringyou have a clean reinstall. TheMakefile also includes functionslikemaketest,makedoc, etc. and was added to help withcommon tasks from the command line.
See more in theTroubleshootingsection of the R Developer environment setup article.
Building from source vs. using binaries
Using binaries is a fast and simple way of working with the last releaseof Arrow. However, if you use these it means that you will be unable tomake changes to the Arrow C++ library.
Note
Every language has its own way of dealing with binaries.To get more information navigate to the section of the language you areinterested to find more information.