Repository files navigation

Try it out via thisdemo, or build and run it on your ownCPU orGPU.

bitnet.cpp is the official inference framework for 1-bit LLMs (e.g., BitNet b1.58). It offers a suite of optimized kernels, that supportfast andlossless inference of 1.58-bit models on CPU and GPU (NPU support will coming next).

The first release of bitnet.cpp is to support inference on CPUs. bitnet.cpp achieves speedups of1.37x to5.07x on ARM CPUs, with larger models experiencing greater performance gains. Additionally, it reduces energy consumption by55.4% to70.0%, further boosting overall efficiency. On x86 CPUs, speedups range from2.37x to6.17x with energy reductions between71.9% to82.2%. Furthermore, bitnet.cpp can run a 100B BitNet b1.58 model on a single CPU, achieving speeds comparable to human reading (5-7 tokens per second), significantly enhancing the potential for running LLMs on local devices. Please refer to thetechnical report for more details.

The tested models are dummy setups used in a research context to demonstrate the inference performance of bitnet.cpp.

A demo of bitnet.cpp running a BitNet b1.58 3B model on Apple M2:

• 05/20/2025BitNet Official GPU inference kernel
• 04/14/2025BitNet Official 2B Parameter Model on Hugging Face
• 02/18/2025Bitnet.cpp: Efficient Edge Inference for Ternary LLMs
• 11/08/2024BitNet a4.8: 4-bit Activations for 1-bit LLMs
• 10/21/20241-bit AI Infra: Part 1.1, Fast and Lossless BitNet b1.58 Inference on CPUs
• 10/17/2024 bitnet.cpp 1.0 released.
• 03/21/2024The-Era-of-1-bit-LLMs__Training_Tips_Code_FAQ
• 02/27/2024The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits
• 10/17/2023BitNet: Scaling 1-bit Transformers for Large Language Models

This project is based on thellama.cpp framework. We would like to thank all the authors for their contributions to the open-source community. Also, bitnet.cpp's kernels are built on top of the Lookup Table methodologies pioneered inT-MAC. For inference of general low-bit LLMs beyond ternary models, we recommend using T-MAC.

❗️We use existing 1-bit LLMs available onHugging Face to demonstrate the inference capabilities of bitnet.cpp. We hope the release of bitnet.cpp will inspire the development of 1-bit LLMs in large-scale settings in terms of model size and training tokens.

• python>=3.9
• cmake>=3.22
• clang>=18

  • For Windows users, installVisual Studio 2022. In the installer, toggle on at least the following options(this also automatically installs the required additional tools like CMake):

    • Desktop-development with C++
    • C++-CMake Tools for Windows
    • Git for Windows
    • C++-Clang Compiler for Windows
    • MS-Build Support for LLVM-Toolset (clang)

  • For Debian/Ubuntu users, you can download withAutomatic installation script

    bash -c "$(wget -O - https://apt.llvm.org/llvm.sh)"

• conda (highly recommend)

1. Clone the repo

git clone --recursive https://github.com/microsoft/BitNet.gitcd BitNet

2. Install the dependencies

# (Recommended) Create a new conda environmentconda create -n bitnet-cpp python=3.9conda activate bitnet-cpppip install -r requirements.txt

3. Build the project

# Manually download the model and run with local pathhuggingface-cli download microsoft/BitNet-b1.58-2B-4T-gguf --local-dir models/BitNet-b1.58-2B-4Tpython setup_env.py -md models/BitNet-b1.58-2B-4T -q i2_s

usage: setup_env.py [-h] [--hf-repo {1bitLLM/bitnet_b1_58-large,1bitLLM/bitnet_b1_58-3B,HF1BitLLM/Llama3-8B-1.58-100B-tokens,tiiuae/Falcon3-1B-Instruct-1.58bit,tiiuae/Falcon3-3B-Instruct-1.58bit,tiiuae/Falcon3-7B-Instruct-1.58bit,tiiuae/Falcon3-10B-Instruct-1.58bit}] [--model-dir MODEL_DIR] [--log-dir LOG_DIR] [--quant-type {i2_s,tl1}] [--quant-embd]                    [--use-pretuned]Setup the environment for running inferenceoptional arguments:  -h, --help            show this help message and exit  --hf-repo {1bitLLM/bitnet_b1_58-large,1bitLLM/bitnet_b1_58-3B,HF1BitLLM/Llama3-8B-1.58-100B-tokens,tiiuae/Falcon3-1B-Instruct-1.58bit,tiiuae/Falcon3-3B-Instruct-1.58bit,tiiuae/Falcon3-7B-Instruct-1.58bit,tiiuae/Falcon3-10B-Instruct-1.58bit}, -hr {1bitLLM/bitnet_b1_58-large,1bitLLM/bitnet_b1_58-3B,HF1BitLLM/Llama3-8B-1.58-100B-tokens,tiiuae/Falcon3-1B-Instruct-1.58bit,tiiuae/Falcon3-3B-Instruct-1.58bit,tiiuae/Falcon3-7B-Instruct-1.58bit,tiiuae/Falcon3-10B-Instruct-1.58bit}                        Model used for inference  --model-dir MODEL_DIR, -md MODEL_DIR                        Directory to save/load the model  --log-dir LOG_DIR, -ld LOG_DIR                        Directory to save the logging info  --quant-type {i2_s,tl1}, -q {i2_s,tl1}                        Quantization type  --quant-embd          Quantize the embeddings to f16  --use-pretuned, -p    Use the pretuned kernel parameters

# Run inference with the quantized modelpython run_inference.py -m models/BitNet-b1.58-2B-4T/ggml-model-i2_s.gguf -p"You are a helpful assistant" -cnv

usage: run_inference.py [-h] [-m MODEL] [-n N_PREDICT] -p PROMPT [-t THREADS] [-c CTX_SIZE] [-temp TEMPERATURE] [-cnv]Run inferenceoptional arguments:  -h, --help            show this help message and exit  -m MODEL, --model MODEL                        Path to model file  -n N_PREDICT, --n-predict N_PREDICT                        Number of tokens to predict when generating text  -p PROMPT, --prompt PROMPT                        Prompt to generate text from  -t THREADS, --threads THREADS                        Number of threads to use  -c CTX_SIZE, --ctx-size CTX_SIZE                        Size of the prompt context  -temp TEMPERATURE, --temperature TEMPERATURE                        Temperature, a hyperparameter that controls the randomness of the generated text  -cnv, --conversation  Whether to enable chat mode or not (for instruct models.)                        (When this option is turned on, the prompt specified by -p will be used as the system prompt.)

We provide scripts to run the inference benchmark providing a model.

usage: e2e_benchmark.py -m MODEL [-n N_TOKEN] [-p N_PROMPT] [-t THREADS]     Setup the environment for running the inference     required arguments:    -m MODEL, --model MODEL                          Path to the model file.    optional arguments:    -h, --help                          Show this help message and exit.   -n N_TOKEN, --n-token N_TOKEN                          Number of generated tokens.   -p N_PROMPT, --n-prompt N_PROMPT                          Prompt to generate text from.   -t THREADS, --threads THREADS                          Number of threads to use.

Here's a brief explanation of each argument:

• -m,--model: The path to the model file. This is a required argument that must be provided when running the script.
• -n,--n-token: The number of tokens to generate during the inference. It is an optional argument with a default value of 128.
• -p,--n-prompt: The number of prompt tokens to use for generating text. This is an optional argument with a default value of 512.
• -t,--threads: The number of threads to use for running the inference. It is an optional argument with a default value of 2.
• -h,--help: Show the help message and exit. Use this argument to display usage information.

For example:

python utils/e2e_benchmark.py -m /path/to/model -n 200 -p 256 -t 4

This command would run the inference benchmark using the model located at/path/to/model, generating 200 tokens from a 256 token prompt, utilizing 4 threads.

For the model layout that do not supported by any public model, we provide scripts to generate a dummy model with the given model layout, and run the benchmark on your machine:

python utils/generate-dummy-bitnet-model.py models/bitnet_b1_58-large --outfile models/dummy-bitnet-125m.tl1.gguf --outtype tl1 --model-size 125M# Run benchmark with the generated model, use -m to specify the model path, -p to specify the prompt processed, -n to specify the number of token to generatepython utils/e2e_benchmark.py -m models/dummy-bitnet-125m.tl1.gguf -p 512 -n 128

# Prepare the .safetensors model filehuggingface-cli download microsoft/bitnet-b1.58-2B-4T-bf16 --local-dir ./models/bitnet-b1.58-2B-4T-bf16# Convert to gguf modelpython ./utils/convert-helper-bitnet.py ./models/bitnet-b1.58-2B-4T-bf16

A:This is an issue introduced in recent version of llama.cpp. Please refer to thiscommit in thediscussion to fix this issue.

A:Before building the project, verify your clang installation and access to Visual Studio tools by running:

clang -v

This command checks that you are using the correct version of clang and that the Visual Studio tools are available. If you see an error message such as:

'clang' is not recognized as an internal or external command, operable program or batch file.

It indicates that your command line window is not properly initialized for Visual Studio tools.

• If you are using Command Prompt, run:

"C:\Program Files\Microsoft Visual Studio\2022\Professional\Common7\Tools\VsDevCmd.bat" -startdir=none -arch=x64 -host_arch=x64

• If you are using Windows PowerShell, run the following commands:

Import-Module "C:\Program Files\Microsoft Visual Studio\2022\Professional\Common7\Tools\Microsoft.VisualStudio.DevShell.dll" Enter-VsDevShell 3f0e31ad -SkipAutomaticLocation -DevCmdArguments "-arch=x64 -host_arch=x64"

These steps will initialize your environment and allow you to use the correct Visual Studio tools.