Installing Java Modules#
System Compatibility#
Java modules are regularly built and tested on macOS and Linux distributions.
Java Compatibility#
Java modules are compatible with JDK 11 and above. Currently, JDK versions11, 17, 21, and latest are tested in CI.
Note that some JDK internals must be exposed byadding--add-opens=java.base/java.nio=org.apache.arrow.memory.core,ALL-UNNAMED to thejava command:
# Directly on the command line$java--add-opens=java.base/java.nio=org.apache.arrow.memory.core,ALL-UNNAMED-jar...# Indirectly via environment variables$envJDK_JAVA_OPTIONS="--add-opens=java.base/java.nio=org.apache.arrow.memory.core,ALL-UNNAMED"java-jar...
Otherwise, you may see errors likemodulejava.basedoesnot"opensjava.nio"tounnamedmodule ormodulejava.basedoesnot"opensjava.nio"toorg.apache.arrow.memory.core
Note that the command has changed from Arrow 15 and earlier. If you are still using the flags from that version(--add-opens=java.base/java.nio=org.apache.arrow.memory.core,ALL-UNNAMED) you will see themodulejava.basedoesnot"opensjava.nio"toorg.apache.arrow.memory.core error.
If you are using flight-core or dependent modules, you will need to mark that flight-core can read unnamed modules.Modifying the command above for Flight:
# Directly on the command line$java--add-opens=java.base/java.nio=org.apache.arrow.memory.core,ALL-UNNAMED-jar...# Indirectly via environment variables$envJDK_JAVA_OPTIONS="--add-reads=org.apache.arrow.flight.core=ALL-UNNAMED --add-opens=java.base/java.nio=org.apache.arrow.memory.core,ALL-UNNAMED"java-jar...
Otherwise, you may see errors likejava.lang.IllegalAccessError:superclassaccesscheckfailed:classorg.apache.arrow.flight.ArrowMessage$ArrowBufRetainingCompositeByteBuf(inmoduleorg.apache.arrow.flight.core)cannotaccessclassio.netty.buffer.CompositeByteBuf(inunnamedmodule...)becausemoduleorg.apache.arrow.flight.coredoesnotreadunnamedmodule...
Finally, if you are using arrow-dataset, you’ll also need to report that JDK internals need to be exposed.Modifying the command above for arrow-memory:
# Directly on the command line$java--add-opens=java.base/java.nio=org.apache.arrow.memory.core,ALL-UNNAMED-jar...# Indirectly via environment variables$envJDK_JAVA_OPTIONS="--add-opens=java.base/java.nio=org.apache.arrow.dataset,org.apache.arrow.memory.core,ALL-UNNAMED"java-jar...
Otherwise you may see errors such asjava.lang.RuntimeException:java.lang.reflect.InaccessibleObjectException:Unabletomakestaticvoidjava.nio.Bits.reserveMemory(long,long)accessible:modulejava.basedoesnot"opensjava.nio"tomoduleorg.apache.arrow.dataset
If using Maven and Surefire for unit testing,this argument mustbe added to Surefire as well.
Installing from Maven#
By default, Maven will download from the central repository:https://repo.maven.apache.org/maven2/org/apache/arrow/
Configure your pom.xml with the Java modules needed, for example:arrow-vector, and arrow-memory-netty.
<?xml version="1.0" encoding="UTF-8"?><projectxmlns="http://maven.apache.org/POM/4.0.0"xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"><modelVersion>4.0.0</modelVersion><groupId>org.example</groupId><artifactId>demo</artifactId><version>1.0-SNAPSHOT</version><properties><arrow.version>9.0.0</arrow.version></properties><dependencies><dependency><groupId>org.apache.arrow</groupId><artifactId>arrow-vector</artifactId><version>${arrow.version}</version></dependency><dependency><groupId>org.apache.arrow</groupId><artifactId>arrow-memory-netty</artifactId><version>${arrow.version}</version></dependency></dependencies></project>
A bill of materials (BOM) module has been provided to simplify addingArrow modules. This eliminates the need to specify the version forevery module. An alternative to the above would be:
<?xml version="1.0" encoding="UTF-8"?><projectxmlns="http://maven.apache.org/POM/4.0.0"xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"><modelVersion>4.0.0</modelVersion><groupId>org.example</groupId><artifactId>demo</artifactId><version>1.0-SNAPSHOT</version><properties><arrow.version>15.0.0</arrow.version></properties><dependencies><dependency><groupId>org.apache.arrow</groupId><artifactId>arrow-vector</artifactId></dependency><dependency><groupId>org.apache.arrow</groupId><artifactId>arrow-memory-netty</artifactId></dependency></dependencies><dependencyManagement><dependencies><dependency><groupId>org.apache.arrow</groupId><artifactId>arrow-bom</artifactId><version>${arrow.version}</version><type>pom</type><scope>import</scope></dependency></dependencies></dependencyManagement></project>
To use the Arrow Flight dependencies, also add theos-maven-pluginplugin. This plugin generates useful platform-dependent propertiessuch asos.detected.name andos.detected.arch needed to resolvetransitive dependencies of Flight.
<?xml version="1.0" encoding="UTF-8"?><projectxmlns="http://maven.apache.org/POM/4.0.0"xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"><modelVersion>4.0.0</modelVersion><groupId>org.example</groupId><artifactId>demo</artifactId><version>1.0-SNAPSHOT</version><properties><arrow.version>9.0.0</arrow.version></properties><dependencies><dependency><groupId>org.apache.arrow</groupId><artifactId>flight-core</artifactId><version>${arrow.version}</version></dependency></dependencies><build><extensions><extension><groupId>kr.motd.maven</groupId><artifactId>os-maven-plugin</artifactId><version>1.7.0</version></extension></extensions></build></project>
The--add-opens flag must be added when running unit tests through Maven:
<build><plugins><plugin><groupId>org.apache.maven.plugins</groupId><artifactId>maven-surefire-plugin</artifactId><version>3.0.0-M6</version><configuration><argLine>--add-opens=java.base/java.nio=ALL-UNNAMED</argLine></configuration></plugin></plugins></build>
Or they can be added via environment variable, for example when executing your code:
JDK_JAVA_OPTIONS="--add-opens=java.base/java.nio=ALL-UNNAMED"mvnexec:java-Dexec.mainClass="YourMainCode"
Installing from Source#
SeeJava Development.
IDE Configuration#
Generally, no additional configuration should be needed. However,ensure your Maven or other build configuration has the--add-opensflag as described above, so that the IDE picks it up and runs testswith that flag as well.