kherud/java-llama.cppPublic

NotificationsYou must be signed in to change notification settings
Fork53
Star395

Java Bindings for llama.cpp - A Port of Facebook's LLaMA model in C/C++

License

MIT license

395 stars 53 forks Branches Tags Activity

Star

Notifications

You must be signed in to change notification settings

Branches Tags

Folders and files

Name		Name	Last commit message	Last commit date
Latest commit History 495 Commits
.github		.github
models		models
src		src
.clang-format		.clang-format
.clang-tidy		.clang-tidy
.gitignore		.gitignore
CMakeLists.txt		CMakeLists.txt
LICENSE.md		LICENSE.md
README.md		README.md
pom.xml		pom.xml

Repository files navigation

Java Bindings forllama.cpp

Inference of Meta's LLaMA model (and others) in pure C/C++.

You are welcome to contribute

Quick Start
1.1No Setup required
1.2Setup required
Documentation
2.1Example
2.2Inference
2.3Infilling
Android

Note

Now with support for Gemma 3

Quick Start

Access this library via Maven:

<dependency>    <groupId>de.kherud</groupId>    <artifactId>llama</artifactId>    <version>4.1.0</version></dependency>

There are multipleexamples.

No Setup required

We support CPU inference for the following platforms out of the box:

Linux x86-64, aarch64
MacOS x86-64, aarch64 (M-series)
Windows x86-64, x64

If any of these match your platform, you can include the Maven dependency and get started.

Setup required

If none of the above listed platforms matches yours, currently you have to compile the library yourself (also if youwant GPU acceleration).

This consists of two steps: 1) Compiling the libraries and 2) putting them in the right location.

Library Compilation

First, have a look atllama.cpp to know which build arguments to use (e.g. for CUDA support).Any build option of llama.cpp works equivalently for this project.You then have to run the following commands in the directory of this repository (java-llama.cpp):

mvn compile# don't forget this linecmake -B build# add any other arguments for your backend, e.g. -DGGML_CUDA=ONcmake --build build --config Release

Tip

Use-DLLAMA_CURL=ON to download models via Java code usingModelParameters#setModelUrl(String).

All compiled libraries will be put in a resources directory matching your platform, which will appear in the cmake output. For example something like:

--  Installing files to /java-llama.cpp/src/main/resources/de/kherud/llama/Linux/x86_64

Library Location

This project has to load a single shared libraryjllama.

Note, that the file name varies between operating systems, e.g.,jllama.dll on Windows,jllama.so on Linux, andjllama.dylib on macOS.

The application will search in the following order in the following locations:

Inde.kherud.llama.lib.path: Use this option if you want a custom location for your shared libraries, i.e., set VM option-Dde.kherud.llama.lib.path=/path/to/directory.
Injava.library.path: These are predefined locations for each OS, e.g.,/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib on Linux.You can find out the locations usingSystem.out.println(System.getProperty("java.library.path")).Use this option if you want to install the shared libraries as system libraries.
From theJAR: If any of the libraries weren't found yet, the application will try to use a prebuilt shared library.This of course only works for thesupported platforms .

Documentation

Example

This is a short example on how to use this library:

publicclassExample {publicstaticvoidmain(String...args)throwsIOException {ModelParametersmodelParams =newModelParameters()                .setModel("models/mistral-7b-instruct-v0.2.Q2_K.gguf")                .setGpuLayers(43);Stringsystem ="This is a conversation between User and Llama, a friendly chatbot.\n" +"Llama is helpful, kind, honest, good at writing, and never fails to answer any " +"requests immediately and with precision.\n";BufferedReaderreader =newBufferedReader(newInputStreamReader(System.in,StandardCharsets.UTF_8));try (LlamaModelmodel =newLlamaModel(modelParams)) {System.out.print(system);Stringprompt =system;while (true) {prompt +="\nUser: ";System.out.print("\nUser: ");Stringinput =reader.readLine();prompt +=input;System.out.print("Llama: ");prompt +="\nLlama: ";InferenceParametersinferParams =newInferenceParameters(prompt)                        .setTemperature(0.7f)                        .setPenalizeNl(true)                        .setMiroStat(MiroStat.V2)                        .setStopStrings("User:");for (LlamaOutputoutput :model.generate(inferParams)) {System.out.print(output);prompt +=output;                }            }        }    }}

Also have a look at the otherexamples.

Inference

There are multiple inference tasks. In general,LlamaModel is stateless, i.e., you have to append the output of themodel to your prompt in order to extend the context. If there is repeated content, however, the library will internallycache this, to improve performance.

ModelParametersmodelParams =newModelParameters().setModel("/path/to/model.gguf");InferenceParametersinferParams =newInferenceParameters("Tell me a joke.");try (LlamaModelmodel =newLlamaModel(modelParams)) {// Stream a response and access more information about each output.for (LlamaOutputoutput :model.generate(inferParams)) {System.out.print(output);    }// Calculate a whole response before returning it.Stringresponse =model.complete(inferParams);// Returns the hidden representation of the context + prompt.float[]embedding =model.embed("Embed this");}

Note

Since llama.cpp allocates memory that can't be garbage collected by the JVM,LlamaModel is implemented as anAutoClosable. If you use the objects withtry-with blocks like the examples, the memory will be automaticallyfreed when the model is no longer needed. This isn't strictly required, but avoids memory leaks if you use differentmodels throughout the lifecycle of your application.

Infilling

You can simply setInferenceParameters#setInputPrefix(String) andInferenceParameters#setInputSuffix(String).

Model/Inference Configuration

There are two sets of parameters you can configure,ModelParameters andInferenceParameters. Both provide builderclasses to ease configuration.ModelParameters are once needed for loading a model,InferenceParameters are neededfor every inference task. All non-specified options have sensible defaults.

ModelParametersmodelParams =newModelParameters()        .setModel("/path/to/model.gguf")        .addLoraAdapter("/path/to/lora/adapter");Stringgrammar ="""root  ::= (expr "=" term "\\n")+expr  ::= term ([-+*/] term)*term  ::= [0-9]""";InferenceParametersinferParams =newInferenceParameters("")        .setGrammar(grammar)        .setTemperature(0.8);try (LlamaModelmodel =newLlamaModel(modelParams)) {model.generate(inferParams);}

Logging

Per default, logs are written to stdout.This can be intercepted via the static methodLlamaModel.setLogger(LogFormat, BiConsumer<LogLevel, String>).There is text- and JSON-based logging. The default is JSON.Note, that text-based logging will include additional output of the GGML backend, while JSON-based loggingonly provides request logs (while still writing GGML messages to stdout).To only change the log format while still writing to stdout,null can be passed for the callback.Logging can be disabled by passing an empty callback.

// Re-direct log messages however you like (e.g. to a logging library)LlamaModel.setLogger(LogFormat.TEXT, (level,message) ->System.out.println(level.name() +": " +message));// Log to stdout, but change the formatLlamaModel.setLogger(LogFormat.TEXT,null);// Disable logging by passing a no-opLlamaModel.setLogger(null, (level,message) -> {});

Importing in Android

You can use this library in Android project.

Add java-llama.cpp as a submodule in your an droidapp project directory

git submodule add https://github.com/kherud/java-llama.cpp

Declare the library as a source in your build.gradle

android {    val jllamaLib= file("java-llama.cpp")// Execute "mvn compile" if folder target/ doesn't exist at ./java-llama.cpp/if (!file("$jllamaLib/target").exists()) {        exec {            commandLine= listOf("mvn","compile")            workingDir= file("java-llama.cpp/")        }    }...    defaultConfig {...        externalNativeBuild {            cmake {// Add an flags if needed                cppFlags+=""                arguments+=""            }        }    }// Declare c++ sources    externalNativeBuild {        cmake {            path= file("$jllamaLib/CMakeLists.txt")            version="3.22.1"        }    }// Declare java sources    sourceSets {        named("main") {// Add source directory for java-llama.cpp            java.srcDir("$jllamaLib/src/main/java")        }    }}