Movatterモバイル変換

[0]ホーム
URL:

Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

 Sign up
Appearance settings

Commit288bb09

Browse files
authored
Zig cross-compilation of native libraries (lmdbjava#217)
This new approach provides a number of benefits:1. The LmdbJava Native project can be archived. This project requiredcoordination with LmdbJava module releases and added extra artifactsto Maven Central simply to wrap native libraries. The native projectalso required a great deal of Maven configuration and it was alwaystedious to support additional platforms. Using Zig eliminates the needto run QEMU emulators in builds to support unusual platforms etc.2. Zig supports numerous cross-compilation targets straight out of thebox. A full list is available via zig targets | jq -r '.libc[]' andthis presently lists 64 on my machine. This is likely to accommodatemost LmdbJava platform support requests we are likely to receive.3. Platform naming conventions have now been standardised based on theZig target name. Support for individual build chains or specificprocessors is now simplified and much more transparent.4. The GitHub Action has been amended to perform cross-compilationunder Linux and upload the resulting native artifacts for later buildsteps. The later build steps run the Verifier on a native VM whereavailable (eg Windows, Mac OS) and this therefore tests thecross-compiled libraries.5. Target name resolution logic has been refactored and externalisedin its own class with corresponding unit tests to ensure corner casesare duly considered and any bugs more easily reproduced andpermanently rectified.This change is backwards compatible with users who used (and maycontinue to use) the lmdbjava.native.lib system property.
1 parent9cf97a5 commit288bb09

File tree

9 files changed

+310
-94
lines changed

9 files changed

+310
-94
lines changed

‎.github/workflows/maven.yml‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,14 +22,29 @@ jobs:
2222
java-version:17
2323
cache:maven
2424

25+
-name:Install Zig
26+
uses:goto-bus-stop/setup-zig@v2
27+
28+
-name:Cross compile using Zig
29+
run:./cross-compile.sh
30+
2531
-name:Build with Maven
2632
run:mvn -B verify
2733

34+
-name:Store built native libraries for later jobs
35+
uses:actions/upload-artifact@v3
36+
with:
37+
name:native-libraries
38+
path:|
39+
src/main/resources/org/lmdbjava/*.so
40+
src/main/resources/org/lmdbjava/*.dll
41+
2842
-name:Upload code coverage to Codecov
2943
uses:codecov/codecov-action@v3
3044

3145
compatibility-checks:
3246
name:Java ${{ matrix.java }} on ${{ matrix.os }} Compatibility
47+
needs:[build]
3348
runs-on:${{ matrix.os }}
3449

3550
strategy:
@@ -48,6 +63,12 @@ jobs:
4863
java-version:${{ matrix.java }}
4964
cache:maven
5065

66+
-name:Fetch built native libraries
67+
uses:actions/download-artifact@v3
68+
with:
69+
name:native-libraries
70+
path:src/main/resources/org/lmdbjava
71+
5172
-name:Execute verifier
5273
run:mvn -B test -Dtest=VerifierTest -DverificationSeconds=10
5374

@@ -81,6 +102,12 @@ jobs:
81102
gpg-private-key:${{ secrets.gpg_private_key }}
82103
gpg-passphrase:MAVEN_GPG_PASSPHRASE
83104

105+
-name:Install Zig
106+
uses:goto-bus-stop/setup-zig@v2
107+
108+
-name:Cross compile using Zig
109+
run:./cross-compile.sh
110+
84111
-name:Publish Maven package
85112
run:mvn -B -Possrh-deploy deploy -DskipTests
86113
env:

‎.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,3 +16,4 @@ dependency-reduced-pom.xml
1616
gpg-sign.json
1717
mvn-sync.json
1818
secrets.tar
19+
lmdb

‎README.md‎

Lines changed: 19 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,7 @@
2727
* Modern, idiomatic Java API (including iterators, key ranges, enums, exceptions etc)
2828
* Nothing to install (the JAR embeds the latest LMDB libraries for Linux, OS X and Windows)
2929
* Buffer agnostic (Java`ByteBuffer`, Agrona`DirectBuffer`, Netty`ByteBuf`, your own buffer)
30-
* 100% stock-standard, officially-released, widely-tested LMDB C code ([no extra](https://github.com/lmdbjava/native) C/JNI code)
30+
* 100% stock-standard, officially-released, widely-tested LMDB C code (no extra C/JNI code)
3131
* Low latency design (allocation-free; buffer pools; optional checks can be easily disabled in production etc)
3232
* Mature code (commenced in 2016) and used for heavy production workloads (eg > 500 TB of HFT data)
3333
* Actively maintained and with a "Zero Bug Policy" before every release (see[issues](https://github.com/lmdbjava/lmdbjava/issues))
@@ -55,6 +55,24 @@ We're happy to help you use LmdbJava. Simply
5555
[open a GitHub issue](https://github.com/lmdbjava/lmdbjava/issues) if you have
5656
any questions.
5757

58+
###Building
59+
60+
This project uses[Zig](https://ziglang.org/) to cross-compile the LMDB native
61+
library for all supported architectures. To locally build LmdbJava you must
62+
firstly install a recent version of Zig and then execute the project's
63+
[cross-compile.sh](https://github.com/lmdbjava/lmdbjava/tree/master/cross-compile.sh)
64+
script. This only needs to be repeated when the`cross-compile.sh` script is
65+
updated (eg following a new official release of the upstream LMDB library).
66+
67+
If you do not wish to install Zig and/or use an operating system which cannot
68+
easily execute the`cross-compile.sh` script, you can download the compiled
69+
LMDB native library for your platform from a location of your choice and set the
70+
`lmdbjava.native.lib` system property to the resulting file system system
71+
location. Possible sources of a compiled LMDB native library include operating
72+
system package managers, running`cross-compile.sh` on a supported system, or
73+
copying it from the`org/lmdbjava` directory of any recent, officially released
74+
LmdbJava JAR.
75+
5876
###Contributing
5977

6078
Contributions are welcome! Please see the[Contributing Guidelines](CONTRIBUTING.md).

‎cross-compile.sh‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
#!/bin/bash
2+
3+
set -o errexit
4+
5+
rm -rf lmdb
6+
git clone --depth 1 --branch LMDB_0.9.29 https://github.com/LMDB/lmdb.git
7+
pushd lmdb/libraries/liblmdb
8+
trappopd SIGINT
9+
10+
# zig targets | jq -r '.libc[]'
11+
fortargetin aarch64-linux-gnu \
12+
aarch64-macos-none \
13+
x86_64-linux-gnu \
14+
x86_64-macos-none \
15+
x86_64-windows-gnu
16+
do
17+
echo"##### Building$target ####"
18+
make -e clean liblmdb.so CC="zig cc -target$target" AR="zig ar"
19+
if [["$target"==*-windows-* ]];then
20+
extension="dll"
21+
else
22+
extension="so"
23+
fi
24+
cp -v liblmdb.so ../../../src/main/resources/org/lmdbjava/$target.$extension
25+
done
26+
27+
ls -l ../../../src/main/resources/org/lmdbjava

‎pom.xml‎

Lines changed: 0 additions & 45 deletions
Original file line numberDiff line numberDiff line change
@@ -70,24 +70,6 @@
7070
<groupId>org.hamcrest</groupId>
7171
<artifactId>hamcrest</artifactId>
7272
</dependency>
73-
<dependency>
74-
<groupId>org.lmdbjava</groupId>
75-
<artifactId>lmdbjava-native-linux-x86_64</artifactId>
76-
<version>0.9.29-1</version>
77-
<optional>true</optional>
78-
</dependency>
79-
<dependency>
80-
<groupId>org.lmdbjava</groupId>
81-
<artifactId>lmdbjava-native-osx-x86_64</artifactId>
82-
<version>0.9.29-1</version>
83-
<optional>true</optional>
84-
</dependency>
85-
<dependency>
86-
<groupId>org.lmdbjava</groupId>
87-
<artifactId>lmdbjava-native-windows-x86_64</artifactId>
88-
<version>0.9.29-1</version>
89-
<optional>true</optional>
90-
</dependency>
9173
</dependencies>
9274
<build>
9375
<plugins>
@@ -103,11 +85,6 @@
10385
<groupId>org.apache.maven.plugins</groupId>
10486
<artifactId>maven-dependency-plugin</artifactId>
10587
<configuration>
106-
<usedDependencies>
107-
<usedDependency>org.lmdbjava:lmdbjava-native-linux-x86_64</usedDependency>
108-
<usedDependency>org.lmdbjava:lmdbjava-native-windows-x86_64</usedDependency>
109-
<usedDependency>org.lmdbjava:lmdbjava-native-osx-x86_64</usedDependency>
110-
</usedDependencies>
11188
<ignoredDependencies>
11289
<ignoredDependency>com.github.jnr:jffi</ignoredDependency>
11390
</ignoredDependencies>
@@ -132,28 +109,6 @@
132109
<groupId>org.apache.maven.plugins</groupId>
133110
<artifactId>maven-pmd-plugin</artifactId>
134111
</plugin>
135-
<plugin>
136-
<groupId>org.apache.maven.plugins</groupId>
137-
<artifactId>maven-shade-plugin</artifactId>
138-
<executions>
139-
<execution>
140-
<id>lmdbjava-shade</id>
141-
<goals>
142-
<goal>shade</goal>
143-
</goals>
144-
<phase>package</phase>
145-
<configuration>
146-
<artifactSet>
147-
<includes>
148-
<include>org.lmdbjava:lmdbjava-native-linux-x86_64</include>
149-
<include>org.lmdbjava:lmdbjava-native-windows-x86_64</include>
150-
<include>org.lmdbjava:lmdbjava-native-osx-x86_64</include>
151-
</includes>
152-
</artifactSet>
153-
</configuration>
154-
</execution>
155-
</executions>
156-
</plugin>
157112
<plugin>
158113
<groupId>org.apache.maven.plugins</groupId>
159114
<artifactId>maven-surefire-plugin</artifactId>

‎src/main/java/org/lmdbjava/Library.java‎

Lines changed: 4 additions & 48 deletions
Original file line numberDiff line numberDiff line change
@@ -21,11 +21,8 @@
2121
packageorg.lmdbjava;
2222

2323
importstaticjava.io.File.createTempFile;
24-
importstaticjava.lang.Boolean.getBoolean;
2524
importstaticjava.lang.System.getProperty;
2625
importstaticjava.lang.Thread.currentThread;
27-
importstaticjava.util.Locale.ENGLISH;
28-
importstaticjava.util.Objects.nonNull;
2926
importstaticjava.util.Objects.requireNonNull;
3027
importstaticjnr.ffi.LibraryLoader.create;
3128
importstaticjnr.ffi.Runtime.getRuntime;
@@ -55,71 +52,30 @@
5552
*/
5653
finalclassLibrary {
5754

58-
/**
59-
* Java system property name that can be set to disable automatic extraction
60-
* of the LMDB system library from the LmdbJava JAR. This may be desirable if
61-
* an operating system-provided LMDB system library is preferred (eg operating
62-
* system package management, vendor support, special compiler flags, security
63-
* auditing, profile guided optimization builds, faster startup time by
64-
* avoiding the library copy etc).
65-
*/
66-
publicstaticfinalStringDISABLE_EXTRACT_PROP ="lmdbjava.disable.extract";
6755
/**
6856
* Java system property name that can be set to the path of an existing
6957
* directory into which the LMDB system library will be extracted from the
7058
* LmdbJava JAR. If unspecified the LMDB system library is extracted to the
7159
* <code>java.io.tmpdir</code>. Ignored if the LMDB system library is not
7260
* being extracted from the LmdbJava JAR (as would be the case if other
73-
* system properties defined in <code>Library</code> have been set).
61+
* system properties defined in <code>TargetName</code> have been set).
7462
*/
7563
publicstaticfinalStringLMDB_EXTRACT_DIR_PROP ="lmdbjava.extract.dir";
76-
/**
77-
* Java system property name that can be set to provide a custom path to a
78-
* external LMDB system library. If set, the system property
79-
* DISABLE_EXTRACT_PROP will be overridden.
80-
*/
81-
publicstaticfinalStringLMDB_NATIVE_LIB_PROP ="lmdbjava.native.lib";
82-
/**
83-
* Indicates whether automatic extraction of the LMDB system library is
84-
* permitted.
85-
*/
86-
publicstaticfinalbooleanSHOULD_EXTRACT = !getBoolean(DISABLE_EXTRACT_PROP);
8764
/**
8865
* Indicates the directory where the LMDB system library will be extracted.
8966
*/
9067
staticfinalStringEXTRACT_DIR =getProperty(LMDB_EXTRACT_DIR_PROP,
9168
getProperty("java.io.tmpdir"));
9269
staticfinalLmdbLIB;
9370
staticfinaljnr.ffi.RuntimeRUNTIME;
94-
/**
95-
* Indicates whether external LMDB system library is provided.
96-
*/
97-
staticfinalbooleanSHOULD_USE_LIB =nonNull(
98-
getProperty(LMDB_NATIVE_LIB_PROP));
99-
privatestaticfinalStringLIB_NAME ="lmdb";
10071

10172
static {
10273
finalStringlibToLoad;
10374

104-
finalStringarch =getProperty("os.arch");
105-
finalbooleanarch64 ="x64".equals(arch) ||"amd64".equals(arch)
106-
||"x86_64".equals(arch);
107-
108-
finalStringos =getProperty("os.name");
109-
finalbooleanlinux =os.toLowerCase(ENGLISH).startsWith("linux");
110-
finalbooleanosx =os.startsWith("Mac OS X");
111-
finalbooleanwindows =os.startsWith("Windows");
112-
113-
if (SHOULD_USE_LIB) {
114-
libToLoad =getProperty(LMDB_NATIVE_LIB_PROP);
115-
}elseif (SHOULD_EXTRACT &&arch64 &&linux) {
116-
libToLoad =extract("org/lmdbjava/lmdbjava-native-linux-x86_64.so");
117-
}elseif (SHOULD_EXTRACT &&arch64 &&osx) {
118-
libToLoad =extract("org/lmdbjava/lmdbjava-native-osx-x86_64.dylib");
119-
}elseif (SHOULD_EXTRACT &&arch64 &&windows) {
120-
libToLoad =extract("org/lmdbjava/lmdbjava-native-windows-x86_64.dll");
75+
if (TargetName.IS_EXTERNAL) {
76+
libToLoad =TargetName.RESOLVED_FILENAME;
12177
}else {
122-
libToLoad =LIB_NAME;
78+
libToLoad =extract(TargetName.RESOLVED_FILENAME);
12379
}
12480

12581
LIB =create(Lmdb.class).load(libToLoad);

0 commit comments

Comments
 (0)
[8]ページ先頭
©2009-2025 Movatter.jp