Repository files navigation

📘Documentation |🛠️Quick Start |🤔Reporting Issues

English |简体中文 |日本語

👋 join us on

LMDeploy is a toolkit for compressing, deploying, and serving LLM, developed by theMMRazor andMMDeploy teams. It has the following core features:

• Efficient Inference: LMDeploy delivers up to 1.8x higher request throughput than vLLM, by introducing key features like persistent batch(a.k.a. continuous batching), blocked KV cache, dynamic split&fuse, tensor parallelism, high-performance CUDA kernels and so on.

• Effective Quantization: LMDeploy supports weight-only and k/v quantization, and the 4-bit inference performance is 2.4x higher than FP16. The quantization quality has been confirmed via OpenCompass evaluation.

• Effortless Distribution Server: Leveraging the request distribution service, LMDeploy facilitates an easy and efficient deployment of multi-model services across multiple machines and cards.

• Interactive Inference Mode: By caching the k/v of attention during multi-round dialogue processes, the engine remembers dialogue history, thus avoiding repetitive processing of historical sessions.

• Excellent Compatibility: LMDeploy supportsKV Cache Quant,AWQ andAutomatic Prefix Caching to be used simultaneously.

LMDeploy has developed two inference engines -TurboMind andPyTorch, each with a different focus. The former strives for ultimate optimization of inference performance, while the latter, developed purely in Python, aims to decrease the barriers for developers.

They differ in the types of supported models and the inference data type. Please refer tothis table for each engine's capability and choose the proper one that best fits your actual needs.

It is recommended installing lmdeploy using pip in a conda environment (python 3.8 - 3.12):

conda create -n lmdeploy python=3.8 -yconda activate lmdeploypip install lmdeploy

The default prebuilt package is compiled onCUDA 12 since v0.3.0.For more information on installing on CUDA 11+ platform, or for instructions on building from source, please refer to theinstallation guide.

importlmdeploywithlmdeploy.pipeline("internlm/internlm3-8b-instruct")aspipe:response=pipe(["Hi, pls intro yourself","Shanghai is"])print(response)

For more information about inference pipeline, please refer tohere.

Please reviewgetting_started section for the basic usage of LMDeploy.

For detailed user guides and advanced guides, please refer to ourtutorials:

• User Guide
  • LLM Inference pipeline
  • VLM Inference pipeline
  • LLM Serving
  • VLM Serving
  • Quantization
• Advance Guide
  • Inference Engine - TurboMind
  • Inference Engine - PyTorch
  • Customize chat templates
  • Add a new model
  • gemm tuning
  • Long context inference
  • Multi-model inference service

• Deploying LLMs offline on the NVIDIA Jetson platform by LMDeploy:LMDeploy-Jetson

• Example project for deploying LLMs using LMDeploy and BentoML:BentoLMDeploy

We appreciate all contributions to LMDeploy. Please refer toCONTRIBUTING.md for the contributing guideline.

• FasterTransformer
• llm-awq
• vLLM
• DeepSpeed-MII

@misc{2023lmdeploy,title={LMDeploy: A Toolkit for Compressing, Deploying, and Serving LLM},author={LMDeploy Contributors},howpublished ={\url{https://github.com/InternLM/lmdeploy}},year={2023}}

This project is released under theApache 2.0 license.