Repository files navigation

|Blog|Documentation|Join Slack|Interest Form|Roadmap

🔥NEW: For enterprise-scale deployment of LMCache and vLLM, please check out vLLMProduction Stack. LMCache is also officially supported inllm-d andKServe!

LMCache is anLLM serving engine extension toreduce TTFT andincrease throughput, especially under long-context scenarios. By storing the KV caches of reusable texts across various locations, including (GPU, CPU DRAM, Local Disk), LMCache reuses the KV caches ofany reused text (not necessarily prefix) inany serving engine instance. Thus, LMCache saves precious GPU cycles and reduces user response delay.

By combining LMCache with vLLM, developers achieve 3-10x delay savings and GPU cycle reduction in many LLM use cases, including multi-round QA and RAG.

• 🔥 Integration with vLLM v1 with the following features:
  • High performance CPU KVCache offloading
  • Disaggregated prefill
  • P2P KVCache sharing
• LMCache is supported in thevLLM production stack,llm-d, andKServe
• Stable support for non-prefix KV caches
• Storage support as follows:
  • CPU
  • Disk
  • NIXL
• Installation support through pip and latest vLLM

To use LMCache, simply installlmcache from your package manager, e.g. pip:

pip install lmcache

Works on Linux NVIDIA GPU platform.

Moredetailed installation instructions are available in the docs.

The best way to get started is to checkout theQuickstart Examples in the docs.

Check out the LMCachedocumentation which is available online.

We also post regularly inLMCache blogs.

Go hands-on with ourexamples,demonstrating how to address different use cases with LMCache.

Fill out theinterest form,sign up for our newsletter,join LMCache slack,check out LMCache website, ordrop an email, and our team will reach out to you!

Thecommunity meeting for LMCache is hosted weekly. All are welcome to join!

Meetingsalternate weekly between these two times:

• Tuesdays at 9:00 AM PT –Add to Calendar
• Tuesdays at 6:30 PM PT –Add to Calendar

We keep notes from each meeting on thisdocument for summaries of standups, discussion, and action items.

Recordings of meetings are available on theYouTube LMCache channel.

We welcome and value all contributions and collaborations. Please check outContributing Guide on how to contribute.

If you use LMCache for your research, please cite our papers:

@inproceedings{liu2024cachegen,  title={Cachegen: Kv cache compression and streaming for fast large language model serving},  author={Liu, Yuhan and Li, Hanchen and Cheng, Yihua and Ray, Siddhant and Huang, Yuyang and Zhang, Qizheng and Du, Kuntai and Yao, Jiayi and Lu, Shan and Ananthanarayanan, Ganesh and others},  booktitle={Proceedings of the ACM SIGCOMM 2024 Conference},  pages={38--56},  year={2024}}@article{cheng2024large,  title={Do Large Language Models Need a Content Delivery Network?},  author={Cheng, Yihua and Du, Kuntai and Yao, Jiayi and Jiang, Junchen},  journal={arXiv preprint arXiv:2409.13761},  year={2024}}@inproceedings{10.1145/3689031.3696098,  author = {Yao, Jiayi and Li, Hanchen and Liu, Yuhan and Ray, Siddhant and Cheng, Yihua and Zhang, Qizheng and Du, Kuntai and Lu, Shan and Jiang, Junchen},  title = {CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion},  year = {2025},  url = {https://doi.org/10.1145/3689031.3696098},  doi = {10.1145/3689031.3696098},  booktitle = {Proceedings of the Twentieth European Conference on Computer Systems},  pages = {94–109},}

Linkedin |Twitter |Youtube

The LMCache codebase is licensed under Apache License 2.0. See theLICENSE file for details.