This repo hosts code for vLLM CI & Performance Benchmark infrastructure.

TPU inference for vLLM, with unified JAX and PyTorch support.

A high-throughput and memory-efficient inference and serving engine for LLMs

A framework for efficient model inference with omni-modality models

vLLM Daily Summarization of Merged PRs

Community maintained hardware plugin for vLLM on Ascend

Fast and memory-efficient exact attention

System Level Intelligent Router for Mixture-of-Models at Cloud, Data Center and Edge

Evaluate and Enhance Your LLM Deployments for Real-World Inference Needs

Community maintained hardware plugin for vLLM on Intel Gaudi