Background#

Multimodal LLMs typically handle non-text inputs by combining a multimodal encoder with an LLM decoder. The encoder first transforms non-text modality input into embeddings, which are then fused with text embeddings and fed into the LLM decoder for downstream inference. Compared to standard LLM inference, multimodal LLM inference involves three additional stages to support non-text modalities.

• Multimodal Input Processor: Preprocess raw multimodal input into a format suitable for the multimodal encoder, such as pixel values for vision models.

• Multimodal Encoder: Encodes the processed input into embeddings that are aligned with the LLM’s embedding space.

• Integration with LLM Decoder: Fuses multimodal embeddings with text embeddings as the input to the LLM decoder.

Optimizations#

TensorRT LLM incorporates some key optimizations to enhance the performance of multimodal inference:

• In-Flight Batching: Batches multimodal requests within the GPU executor to improve GPU utilization and throughput.

• CPU/GPU Concurrency: Asynchronously overlaps data preprocessing on the CPU with image encoding on the GPU.

• Raw data hashing: Leverages image hashes and token chunk information to improve KV cache reuse and minimize collisions.

Further optimizations are under development and will be updated as they become available.

Model Support Matrix#

Please refer to the latest multimodalsupport matrix.

Examples#

The following examples demonstrate how to use TensorRT LLM’s multimodal support in various scenarios, including quick run examples, serving endpoints, and performance benchmarking.

python3quickstart_multimodal.py--model_dirEfficient-Large-Model/NVILA-8B--modalityimage--disable_kv_cache_reuse

trtllm-serveQwen/Qwen2-VL-7B-Instruct--backendpytorch