Repository files navigation

A standalone Python/C++/CUDA implementation of Llama for use with 4-bit GPTQ weights, designed to be fast andmemory-efficient on modern GPUs.

Disclaimer: The project is coming along, but it's still a work in progress!

I am developing on an RTX 4090 and an RTX 3090-Ti. 30-series and later NVIDIA GPUs should be well supported, butanything Pascal or older with poor FP16 support isn't going to perform well.AutoGPTQ orGPTQ-for-LLaMaare better options at the moment for older GPUs. ROCm is also theoretically supported (via HIP) though I currentlyhave no AMD devices to test or optimize on.

• Python 3.9 or newer
• torch tested on 2.0.1 and 2.1.0 (nightly) with cu118
• safetensors 0.3.2
• sentencepiece
• ninja

Additionally, only for the web UI:

• flask
• waitress

pip install --pre torch --index-url https://download.pytorch.org/whl/nightly/cu118

To run on Windows (without WSL):

1. InstallMSVC 2022. You can choose to install the wholeVisual Studio 2022 IDE, or alternatively just theBuild Tools for Visual Studio 2022 package (make sureDesktop development with C++ is ticked in the installer), it doesn't really matter which.
2. Install the appropriate version ofPyTorch, choosing one of the CUDAversions. I am developing on the nightly build, but the stable version (2.0.1) should also work.
3. Install CUDA Toolkit, (11.7 and11.8 both seem to work, just make sure to match PyTorch'sCompute Platform version).
4. For best performance, enable Hardware Accelerated GPU Scheduling.

Clone repo, install dependencies, and run benchmark:

git clone https://github.com/turboderp/exllamacd exllamapip install -r requirements.txtpython test_benchmark_inference.py -d <path_to_model_files> -p -ppl

The CUDA extension is loaded at runtime so there's no need to install it separately. It will be compiled on the firstrun and cached to~/.cache/torch_extensions/ which could take a little while. If nothing happens at first, give ita minute to compile.

Chatbot example:

python example_chatbot.py -d <path_to_model_files> -un "Jeff" -p prompt_chatbort.txt

jllllll currently maintains an installable Python modulehere which may be moresuitable for integrating ExLlama with other projects

I also made a simple web UI for it. Don't look at the JavaScript, it was mostly written by ChatGPT and it will hauntyour dreams. But it sort of works, and it's kinda fun, especially multibot mode:

To run it:

pip install -r requirements-web.txtpython webui/app.py -d <path_to_model_files>

Note that sessions are stored in~/exllama_sessions/ by default. You can change that location with-sd if you want.

For security benefits and easier deployment, it is also possible to run the web UI in an isolated docker container. Note: the docker image currently only supports NVIDIA GPUs.

• Docker
• NVIDIA Container Toolkit

It is recommended to run docker inrootless mode.

The easiest way to build the docker image is using docker compose. First, set theMODEL_PATH andSESSIONS_PATH variables in the.env file to the actual directories on the host. Then run:

docker compose build

It is also possible to manually build the image:

docker build -t exllama-web .

NOTE: by default, the service inside the docker container is run by a non-root user. Hence, the ownership of bind-mounted directories (/data/model and/data/exllama_sessions in the defaultdocker-compose.yml file) is changed to this non-root user in the container entrypoint (entrypoint.sh). To disable this, setRUN_UID=0 in the.env file if usingdocker compose, or the following command if you manually build the image:

docker build -t exllama-web --build-arg RUN_UID=0 .

Using docker compose:

docker compose up

The web UI can now be accessed on the host athttp://localhost:5000.

The configuration can be viewed indocker-compose.yml and changed by creating adocker-compose.override.yml file.

Run manually:

docker run --gpus all -p 5000:5000 -v <path_to_model_dir>:/data/model/ -v <path_to_session_dir>:/data/exllama_sessions --rm -it exllama-web --host 0.0.0.0:5000

¹ Can not achieve full sequence length without OoM

All tests done on stock RTX 4090 / 12900K, running with a desktop environment, with a few other apps also using VRAM.

"Prompt" speed is inference over the sequence length listed minus 128 tokens."Worst" is the average speed forthe last 128 tokens of the full context (worst case) and"Best" lists the speed for the first 128 tokens in anempty sequence (best case.)

VRAM usage is as reported by PyTorch and does not include PyTorch's own overhead (CUDA kernels,internal buffers etc.) This is somewhat unpredictable anyway. Best bet is to just optimize VRAM usage by the model,probably aiming for 20 GB on a 24 GB GPU to ensure there is room for a desktop environment and all of Torch'sinternals.

Perplexity is measured only to verify that the models are working. The dataset used is a particular, small sample fromWikiText, so scores are not comparable to other Llama benchmarks and only useful for comparing the different Llamamodels to one another.

The following benchmarks are from a 4090 + 3090-Ti with-gs 17.2,24:

Note that perplexity scores may not be strictly apples-to-apples between Llama and Llama 2 due to their differentpretraining datasets.

Moved the todo listhere.

Here is a list of models confirmed to be working right now.

2023-01-09: Added rope_theta parameter for (at least partial) CodeLlama support. If you were using alpha = 97or similar, you would no longer need that for CodeLlama models. Still stuff to sort out regarding the extendedvocabulary.

2023-08-09: Added support for sharded models.config.model_path now accepts either a filename or a list offilenames.model_init() will detect multiple .safetensors files if given a model directory. Note the change in thevarious examples:model_path = glob.glob(st_pattern)[0] becomes simplymodel_path = glob.glob(st_pattern). Alsothere's a little script inutil/shard.py to split large .safetensors files. It also produces an index.json file forthe sharded model, just for completeness, although ExLlama doesn't need it to read the shards. Note that thesafetensors dependency was bumped to version 0.3.2.

2023-08-12: Preliminary, initial and tentative release ofExLlamaV2.It doesn't do all the things that ExLlamaV1 does, yet, but it's better at what it does do. So check it out!