llm-inference

Superduper: Build end-to-end AI applications and agent workflows on your existing data infrastructure and preferred tools - without migrating your data.

python data database ai mongodb chatbot ml transformers inference torch pytorch pretrained-models semantic-search rag mlops distributed-ml vector-search llmops llm-serving llm-inference

UpdatedMar 24, 2025
Jupyter Notebook

kserve /kserve

Star4k

Standardized Serverless ML Inference Platform on Kubernetes

kubernetes machine-learning tensorflow sklearn pytorch artificial-intelligence xgboost k8s service-mesh hacktoberfest istio model-serving kubeflow mlops knative model-interpretability kserve genai llm-inference

UpdatedMar 24, 2025
Python

xlite-dev /Awesome-LLM-Inference

Star3.7k

📖A curated list of Awesome LLM/VLM Inference Papers with codes: WINT8/4, FlashAttention, PagedAttention, MLA, Parallelism, etc. 🎉🎉

mla vllm llm-inference awesome-llm flash-attention tensorrt-llm paged-attention deepseek flash-attention-3 deepseek-v3 minimax-01 deepseek-r1 flash-mla

UpdatedMar 4, 2025

neuralmagic /deepsparse

Star3.1k

Sparsity-aware deep learning inference runtime for CPUs

nlp performance computer-vision inference machinelearning pruning object-detection pretrained-models quantization cpus onnx sparsification llm-inference deepsparse

UpdatedJul 19, 2024
Python

NVIDIA /GenerativeAIExamples

Star2.9k

Generative AI reference workflows optimized for accelerated infrastructure and microservice architecture.

microservice gpu-acceleration nemo tensorrt rag triton-inference-server large-language-models llm llm-inference retrieval-augmented-generation

UpdatedMar 18, 2025
Python

FellouAI /eko

Star2.9k

Eko (Eko Keeps Operating) - Build Production-ready Agentic Workflow with Natural Language - eko.fellou.ai

agent workflow agents natural-language-inference browser-automation ai-agents rag computer-automation prompt-engineering chain-of-thought genai llm-inference llm-agents llmapi agentic-framework agentic-workflow agentic-ai agentic-ai-development computeruse browseruse

UpdatedMar 24, 2025
TypeScript

predibase /lorax

Star2.8k

Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs

transformers pytorch llama gpt lora model-serving fine-tuning llm llmops llm-serving llm-inference

UpdatedMar 7, 2025
Python

databricks /dbrx

Star2.5k

Code examples and resources for DBRX, a large language model developed by Databricks

databricks llm generative-ai gen-ai llm-training llm-inference mosaic-ai

UpdatedMay 1, 2024
Python

FasterDecoding /Medusa

Star2.5k

Medusa: Simple Framework for Accelerating LLM Generation with Multiple Decoding Heads

llm llm-inference

UpdatedJun 25, 2024
Jupyter Notebook

Improve this page

Add a description, image, and links to thellm-inference topic page so that developers can more easily learn about it.

Curate this topic

Add this topic to your repo

To associate your repository with thellm-inference topic, visit your repo's landing page and select "manage topics."

Learn more

Movatterモバイル変換

Navigation Menu

Search code, repositories, users, issues, pull requests...

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

llm-inference

Here are 843 public repositories matching this topic...

nomic-ai /gpt4all

ray-project /ray

gitleaks /gitleaks

liguodongiot /llm-action

Lightning-AI /litgpt

bentoml /OpenLLM

mistralai /mistral-inference

SJTU-IPADS /PowerInfer

openvinotoolkit /openvino

bentoml /BentoML

InternLM /lmdeploy

superduper-io /superduper