vLLM: A high-throughput and memory-efficient inference and serving engine for LLMs

Python SDK, Proxy Server (LLM Gateway) to call 100+ LLM APIs in OpenAI format - [Bedrock, Azure, OpenAI, VertexAI, Cohere, Anthropic, Sagemaker, HuggingFace, Replicate, Groq]

SkyPilot: Run AI and batch jobs on any infra (Kubernetes or 12+ clouds). Get unified execution, cost savings, and high GPU availability via a simple interface.

The collaborative spreadsheet for AI. Chain cells into powerful pipelines, experiment with prompts and models, and evaluate LLM responses in real-time. Work together seamlessly to build and iterate on AI applications.

A high-throughput and memory efficient inference and serving engine for Omni-modality models

AI Tensor Engine for ROCm

This is a repository containing the tools for testing vLLM correctness and perf regression

A high-throughput and memory-efficient inference and serving engine for LLMs

Typescript Documentation of JamAISDK