Repository files navigation

KServe is a standardized distributed generative and predictive AI inference platform for scalable, multi-framework deployment on Kubernetes.

KServe is beingused by many organizations and is aCloud Native Computing Foundation (CNCF) incubating project.

For more details, visit theKServe website.

Single platform that unifies Generative and Predictive AI inference on Kubernetes. Simple enough for quick deployments, yet powerful enough to handle enterprise-scale AI workloads with advanced features.

Generative AI

• 🧠LLM-Optimized: OpenAI-compatible inference protocol for seamless integration with large language models
• 🚅GPU Acceleration: High-performance serving with GPU support and optimized memory management for large models
• 💾Model Caching: Intelligent model caching to reduce loading times and improve response latency for frequently used models
• 🗂️KV Cache Offloading: Advanced memory management with KV cache offloading to CPU/disk for handling longer sequences efficiently
• 📈Autoscaling: Request-based autoscaling capabilities optimized for generative workload patterns
• 🔧Hugging Face Ready: Native support for Hugging Face models with streamlined deployment workflows

Predictive AI

• 🧮Multi-Framework: Support for TensorFlow, PyTorch, scikit-learn, XGBoost, ONNX, and more
• 🔀Intelligent Routing: Seamless request routing between predictor, transformer, and explainer components with automatic traffic management
• 🔄Advanced Deployments: Canary rollouts, inference pipelines, and ensembles with InferenceGraph
• ⚡Autoscaling: Request-based autoscaling with scale-to-zero for predictive workloads
• 🔍Model Explainability: Built-in support for model explanations and feature attribution to understand prediction reasoning
• 📊Advanced Monitoring: Enables payload logging, outlier detection, adversarial detection, and drift detection
• 💰Cost Efficient: Scale-to-zero on expensive resources when not in use, reducing infrastructure costs

To learn more about KServe, how to use various supported features, and how to participate in the KServe community,please follow theKServe website documentation.Additionally, we have compiled a list ofpresentations and demos to dive through various details.

• Standard Kubernetes Installation: Compared to Serverless Installation, this is a morelightweight installation. However, this option does not support canary deployment and request based autoscaling with scale-to-zero.
• Knative Installation: KServe by default installs Knative forserverless deployment for InferenceService.
• ModelMesh Installation: You can optionally install ModelMesh to enablehigh-scale,high-density andfrequently-changing model serving use cases.
• Quick Installation: Install KServe on your local machine.

KServe is an important addon component of Kubeflow, please learn more from theKubeflow KServe documentation. Check out the following guides for runningon AWS oron OpenShift Container Platform.

Thanks to all of our amazing contributors!