Describe Your Changes

High-level design

• vLLM is an inference engine for large-scale (many GPUs)
• cortex will spawn an vLLM subprocess and route the requests to vLLM

cortex engines install vllm

• Download uv tocortexcpp/python_engines/bin/uv if uv is not installed
• (via uv) Setup venv atcortexcpp/python_engines/envs/vllm/<version>/.venv
• (via uv) Download vllm and its deps
• Known issues:
  • Progress streaming is not supported (since download is done via uv instead ofDownloadService).
  • It's not async since we need to wait for subprocess to finish (perhaps we will need a newSubprocessService in the future which handles asyncWaitProcess())
  • Hence, stopping and resuming download also does not work.

Note:

• All cached Python packages are stored incortexcpp/python_engines/cache/uv. The purpose is that when we removepython_engines folder, we are sure that we don't leave anything behind.

cortex models start <model>

• Spawnvllm serve

TODO:

• cortex engines install vllm (TODO: async install in separate thread)
• Set default engine variant
• cortex engines load vllm
• cortex engines list
• cortex engines uninstall vllm: deletecortexcpp/python_engines/envs/vllm/<version>
• cortex pull <model>
• cortex models list
• cortex models start <model>: spawnvllm serve
• cortex models stop <model>
• cortex ps
• Chat completion
  • Non-streaming
  • Streaming
• Embeddings
• cortex run

Fixes Issues

• ClosesvLLM backend for Cortex #1890

Self Checklist

• Added relevant comments, esp in complex areas
• Updated docs (for bug fixes / features)
• Created issues for follow-up changes or refactoring needed