Repository files navigation

KoboldCpp is an easy-to-use AI text-generation software for GGML and GGUF models, inspired by the originalKoboldAI. It's a single self-contained distributable that builds offllama.cpp and adds many additional powerful features.

• Single file executable, with no installation required and no external dependencies
• Runs on CPU or GPU, supports full or partial offloaded
• LLM text generation (Supports all GGML and GGUF models, backwards compatibility with ALL past models)
• Image Generation (Stable Diffusion 1.5, SDXL, SD3, Flux)
• Speech-To-Text (Voice Recognition) via Whisper
• Text-To-Speech (Voice Generation) via OuteTTS
• Provides many compatible APIs endpoints for many popular webservices (KoboldCppApi OpenAiApi OllamaApi A1111ForgeApi ComfyUiApi WhisperTranscribeApi XttsApi OpenAiSpeechApi)
• Bundled KoboldAI Lite UI with editing tools, save formats, memory, world info, author's note, characters, scenarios.
• Includes multiple modes (chat, adventure, instruct, storywriter) and UI Themes (aesthetic roleplay, classic writer, corporate assistant, messsenger)
• Supports loading Tavern Character Cards, importing many different data formats from various sites, reading or exporting JSON savefiles and persistent stories.
• Many other features including new samplers, regex support, websearch, RAG via TextDB and more.
• Ready-to-use binaries for Windows, MacOS, Linux. Runs directly with Colab, Docker, also supports other platforms if self-compiled (like Android (via Termux) and Raspberry PI).
• Need help finding a model? Read this!

• Windows binaries are provided in the form ofkoboldcpp.exe, which is a pyinstaller wrapper containing all necessary files.Download the latest koboldcpp.exe release here
• To run, simply executekoboldcpp.exe.
• Launching with no command line arguments displays a GUI containing a subset of configurable settings. Generally you dont have to change much besides thePresets andGPU Layers. Read the--help for more info about each settings.
• Obtain and load a GGUF model. Seehere
• By default, you can connect tohttp://localhost:5001
• You can also run it using the command line. For info, please checkkoboldcpp.exe --help

On modern Linux systems, you should download thekoboldcpp-linux-x64 prebuilt PyInstaller binary on thereleases page. Simply download and run the binary (You may have tochmod +x it first). If you have an older device, you can also try thekoboldcpp-linux-x64-oldpc instead for greatest compatibility.

Alternatively, you can also install koboldcpp to the current directory by running the following terminal command:

curl -fLo koboldcpp https://github.com/LostRuins/koboldcpp/releases/latest/download/koboldcpp-linux-x64-oldpc && chmod +x koboldcpp

After running this command you can launch Koboldcpp from the current directory using./koboldcpp in the terminal (for CLI usage, run with--help).Finally, obtain and load a GGUF model. Seehere

• PyInstaller binaries for Modern ARM64 MacOS (M1, M2, M3) are now available!Simply download the MacOS binary
• In a MacOS terminal window, set the file to executablechmod +x koboldcpp-mac-arm64 and run it with./koboldcpp-mac-arm64.
• In newer MacOS you may also have to whitelist it in security settings if it's blocked.Here's a video guide.
• Alternatively, or for older x86 MacOS computers, you can clone the repo and compile from source code, see Compiling for MacOS below.
• Finally, obtain and load a GGUF model. Seehere

• KoboldCpp now has anofficial Colab GPU Notebook! This is an easy way to get started without installing anything in a minute or two.Try it here!.
• Note that KoboldCpp is not responsible for your usage of this Colab Notebook, you should ensure that your own usage complies with Google Colab's terms of use.

• KoboldCpp can now be used on RunPod cloud GPUs! This is an easy way to get started without installing anything in a minute or two, and is very scalable, capable of running 70B+ models at afforable cost.Try our RunPod image here!.

KoboldCpp can now also be run on Novita AI, a newer alternative GPU cloud provider which has a quick launch KoboldCpp template for as well.Check it out here!

• The official docker can be found athttps://hub.docker.com/r/koboldai/koboldcpp
• If you're building your own docker, remember to enable LLAMA_PORTABLE

• KoboldCpp uses GGUF models. They are not included with KoboldCpp, but you can download GGUF files from other places such asBartowski's Huggingface. Search for "GGUF" on huggingface.co for plenty of compatible models in the.gguf format.
• For beginners, we recommend the modelsAiroboros Mistral 7B (smaller and weaker) orTiefighter 13B (larger model) orBeepo 22B (largest and most powerful)
• Alternatively, you can download the tools to convert models to the GGUF format yourself here. Runconvert-hf-to-gguf.py to convert them, thenquantize_gguf.exe to quantize the result.
• Other models for Whisper (speech recognition), Image Generation, Text to Speech or Image Recognitioncan be found on the Wiki

• GPU Acceleration: If you're on Windows with an Nvidia GPU you can get CUDA support out of the box using the--usecublas flag (Nvidia Only), or--usevulkan (Any GPU), make sure you select the correct .exe with CUDA support.
• GPU Layer Offloading: Add--gpulayers to offload model layers to the GPU. The more layers you offload to VRAM, the faster generation speed will become. Experiment to determine number of layers to offload, and reduce by a few if you run out of memory.
• Increasing Context Size: Use--contextsize (number) to increase context size, allowing the model to read more text. Note that you may also need to increase the max context in the KoboldAI Lite UI as well (click and edit the number text field).
• Old CPU Compatibility: If you are having crashes or issues, you can try running in a non-avx2 compatibility mode by adding the--noavx2 flag. You can also try reducing your--blasbatchssize (set -1 to avoid batching)

For more information, be sure to run the program with the--help flag, orcheck the wiki.

when you can't use the precompiled binary directly, we provide an automated build script which uses conda to obtain all dependencies, and generates (from source) a ready-to-use a pyinstaller binary for linux users.

• Clone the repo withgit clone https://github.com/LostRuins/koboldcpp.git
• Simply execute the build script with./koboldcpp.sh dist and run the generated binary. (Not recommended for systems that already have an existing installation of conda. Dependencies: curl, bzip2)

./koboldcpp.sh # This launches the GUI for easy configuration and launching (X11 required)../koboldcpp.sh --help # List all available terminal commands for using Koboldcpp, you can use koboldcpp.sh the same way as our python script and binaries../koboldcpp.sh rebuild # Automatically generates a new conda runtime and compiles a fresh copy of the libraries. Do this after updating Koboldcpp to keep everything functional../koboldcpp.sh dist # Generate your own precompiled binary (Due to the nature of Linux compiling these will only work on distributions equal or newer than your own.)

• To compile your binaries from source, clone the repo withgit clone https://github.com/LostRuins/koboldcpp.git
• A makefile is provided, simply runmake.
• Optional Vulkan: Link your own install of Vulkan SDK manually withmake LLAMA_VULKAN=1
• Optional CLBlast: Link your own install of CLBlast manually withmake LLAMA_CLBLAST=1
• Note: for these you will need to obtain and link OpenCL and CLBlast libraries.
  • For Arch Linux: Installcblas andclblast.
  • For Debian: Installlibclblast-dev.
• You can attempt a CuBLAS build withLLAMA_CUBLAS=1, (orLLAMA_HIPBLAS=1 for AMD). You will need CUDA Toolkit installed. Some have also reported success with the CMake file, though that is more for windows.
• For a full featured build (all backends), domake LLAMA_CLBLAST=1 LLAMA_CUBLAS=1 LLAMA_VULKAN=1. (Note thatLLAMA_CUBLAS=1 will not work on windows, you need visual studio)
• To make your build sharable and capable of working on other devices, you must useLLAMA_PORTABLE=1
• After all binaries are built, you can run the python script with the commandpython koboldcpp.py [ggml_model.gguf] [port]

• You're encouraged to use the .exe released, but if you want to compile your binaries from source at Windows, the easiest way is:
  • Get the latest release of w64devkit (https://github.com/skeeto/w64devkit). Be sure to use the "vanilla one", not i686 or other different stuff. If you try they will conflit with the precompiled libs!
  • Clone the repo withgit clone https://github.com/LostRuins/koboldcpp.git
  • Make sure you are using the w64devkit integrated terminal, then runmake at the KoboldCpp source folder. This will create the .dll files for a pure CPU native build.
  • For a full featured build (all backends), domake LLAMA_CLBLAST=1 LLAMA_VULKAN=1. (Note thatLLAMA_CUBLAS=1 will not work on windows, you need visual studio)
  • To make your build sharable and capable of working on other devices, you must useLLAMA_PORTABLE=1
  • If you want to generate the .exe file, make sure you have the python module PyInstaller installed with pip (pip install PyInstaller). Then run the scriptmake_pyinstaller.bat
  • The koboldcpp.exe file will be at your dist folder.
• Building with CUDA: Visual Studio, CMake and CUDA Toolkit is required. Clone the repo, then open the CMake file and compile it in Visual Studio. Copy thekoboldcpp_cublas.dll generated into the same directory as thekoboldcpp.py file. If you are bundling executables, you may need to include CUDA dynamic libraries (such ascublasLt64_11.dll andcublas64_11.dll) in order for the executable to work correctly on a different PC.
• Replacing Libraries (Not Recommended): If you wish to use your own version of the additional Windows libraries (OpenCL, CLBlast, Vulkan), you can do it with:
  • OpenCL - tested withhttps://github.com/KhronosGroup/OpenCL-SDK . If you wish to compile it, follow the repository instructions. You will need vcpkg.
  • CLBlast - tested withhttps://github.com/CNugteren/CLBlast . If you wish to compile it you will need to reference the OpenCL files. It will only generate the ".lib" file if you compile using MSVC.
  • Move the respectives .lib files to the /lib folder of your project, overwriting the older files.
  • Also, replace the existing versions of the corresponding .dll files located in the project directory root (e.g. clblast.dll).
  • Make the KoboldCpp project using the instructions above.

• You can compile your binaries from source. You can clone the repo withgit clone https://github.com/LostRuins/koboldcpp.git
• A makefile is provided, simply runmake.
• If you want Metal GPU support, instead runmake LLAMA_METAL=1, note that MacOS metal libraries need to be installed.
• To make your build sharable and capable of working on other devices, you must useLLAMA_PORTABLE=1
• After all binaries are built, you can run the python script with the commandpython koboldcpp.py --model [ggml_model.gguf] (and add--gpulayers (number of layer) if you wish to offload layers to GPU).

• First, Install and run Termux from F-Droid

• You can use this auto-installation script to quickly install and build everything and launch KoboldCpp with a model.Simply run:

curl -sSL https://raw.githubusercontent.com/LostRuins/koboldcpp/concedo/android_install.sh | sh

and it will install everything required. Alternatively, you can download the aboveandroid_install.sh script to file, then dochmod +x and run it interactively.

• Open termux and run the commandapt update
• Install dependencyapt install openssl
• Install other dependencies withpkg install wget git python
• Runpkg upgrade
• Clone the repogit clone https://github.com/LostRuins/koboldcpp.git
• Navigate to the koboldcpp foldercd koboldcpp
• Build the projectmake
• To make your build sharable and capable of working on other devices, you must useLLAMA_PORTABLE=1, this disables usage of ARM instrinsics.
• Grab a small GGUF model, such aswget https://huggingface.co/concedo/KobbleTinyV2-1.1B-GGUF/resolve/main/KobbleTiny-Q4_K.gguf
• Start the python serverpython koboldcpp.py --model KobbleTiny-Q4_K.gguf
• Connect tohttp://localhost:5001 on your mobile browser
• If you encounter any errors, make sure your packages are up-to-date withpkg up andpkg upgrade
• If you have trouble installing an dependency, you can try the commandtermux-change-repo and choose a different repo (e.g.Mirror by BFSU)
• GPU acceleration for Termux may be possible but I have not explored it. If you find a good cross-device solution, do share or PR it.

• For most users, you can get very decent speeds by selecting theVulkan option instead, which supports both Nvidia and AMD GPUs.
• Alternatively, you can try the ROCM fork athttps://github.com/YellowRoseCx/koboldcpp-rocm though this may be outdated.

• These unofficial resources have been contributed by the community, and may be outdated or unmaintained. No official support will be provided for them!
  • Arch Linux Packages:CUBLAS, andHIPBLAS.
  • Unofficial Dockers:korewaChino andnoneabove1182
  • Nix & NixOS: KoboldCpp is available on Nixpkgs and can be installed by adding justkoboldcpp to yourenvironment.systemPackages(or it can also be placed inhome.packages).
    • Example Nix Setup and further information
    • If you face any issues with running KoboldCpp on Nix, please open an issuehere.
• GPTLocalhost - KoboldCpp is supported by GPTLocalhost, a local Word Add-in for you to use KoboldCpp in Microsoft Word. A local alternative to "Copilot in Word."

• First, please check outThe KoboldCpp FAQ and Knowledgebase which may already have answers to your questions! Also please search through past issues and discussions.
• If you cannot find an answer, open an issue on this github, or find us on theKoboldAI Discord.

• Documentation for KoboldAI and KoboldCpp endpoints can be found here

• A public KoboldCpp demo can be found at our Huggingface Space. Please do not abuse it.

• For Windows: No installation, single file executable, (It Just Works)
• Since v1.15, requires CLBlast if enabled, the prebuilt windows binaries are included in this repo. If not found, it will fall back to a mode without CLBlast.
• Since v1.33, you can set the context size to be above what the model supports officially. It does increases perplexity but should still work well below 4096 even on untuned models. (For GPT-NeoX, GPT-J, and Llama models) Customize this with--ropeconfig.
• Since v1.42, supports GGUF models for LLAMA and Falcon
• Since v1.55, lcuda paths on Linux are hardcoded and may require manual changes to the makefile if you do not use koboldcpp.sh for the compilation.
• Since v1.60, provides native image generation with StableDiffusion.cpp, you can load any SD1.5 or SDXL .safetensors model and it will provide an A1111 compatible API to use.
• I try to keep backwards compatibility with ALL past llama.cpp models. But you are also encouraged to reconvert/update your models if possible for best results.
• Since v1.75, openblas has been deprecated and removed in favor of the native CPU implementation.

• The original GGML library, stable-diffusion.cpp and llama.cpp by ggerganov are licensed under the MIT License
• However, KoboldAI Lite is licensed under the AGPL v3.0 License
• KoboldCpp code and other files are also under the AGPL v3.0 License unless otherwise stated
• Llama.cpp source repo is athttps://github.com/ggml-org/llama.cpp (MIT)
• Stable-diffusion.cpp source repo is athttps://github.com/leejet/stable-diffusion.cpp (MIT)
• KoboldCpp source repo is athttps://github.com/LostRuins/koboldcpp (AGPL)
• KoboldAI Lite source repo is athttps://github.com/LostRuins/lite.koboldai.net (AGPL)
• For any further enquiries, contact @concedo on discord, or LostRuins on github.

• If you wish, after building the koboldcpp libraries withmake, you can rebuild the exe yourself with pyinstaller by usingmake_pyinstaller.bat
• API documentation available at/api (e.g.http://localhost:5001/api) andhttps://lite.koboldai.net/koboldcpp_api. An OpenAI compatible API is also provided at/v1 route (e.g.http://localhost:5001/v1).
• All up-to-date GGUF models are supported, and KoboldCpp also includes backward compatibility for older versions/legacy GGML.bin models, though some newer features might be unavailable.
• An incomplete list of architectures is listed, but there aremany hundreds of other GGUF models. In general, if it's GGUF, it should work.
• Llama / Llama2 / Llama3 / Alpaca / GPT4All / Vicuna / Koala / Pygmalion / Metharme / WizardLM / Mistral / Mixtral / Miqu / Qwen / Qwen2 / Yi / Gemma / Gemma2 / GPT-2 / Cerebras / Phi-2 / Phi-3 / GPT-NeoX / Pythia / StableLM / Dolly / RedPajama / GPT-J / RWKV4 / MPT / Falcon / Starcoder / Deepseek and many,many more.

• The best place to get GGUF text models is huggingface. For image models, CivitAI has a good selection. Here are some to get started.
  • Text Generation:Airoboros Mistral 7B (smaller and weaker) orTiefighter 13B (larger model) orBeepo 22B (largest and most powerful)
  • Image Generation:Anything v3 orDeliberate V2 orDreamshaper SDXL
  • Image Recognition MMproj:Pick the correct one for your model architecture here
  • Speech Recognition:Whisper models for Speech-To-Text
  • Text-To-Speech:TTS models for Narration