LiteLLM Proxy Server (LLM Gateway) | Hosted Proxy |Enterprise Tier

Usage (Docs)

pip install litellm

fromlitellmimportcompletionimportos## set ENV variablesos.environ["OPENAI_API_KEY"]="your-openai-key"os.environ["ANTHROPIC_API_KEY"]="your-anthropic-key"messages= [{"content":"Hello, how are you?","role":"user"}]# openai callresponse=completion(model="openai/gpt-4o",messages=messages)# anthropic callresponse=completion(model="anthropic/claude-sonnet-4-20250514",messages=messages)print(response)

Response (OpenAI Format)

{"id":"chatcmpl-1214900a-6cdd-4148-b663-b5e2f642b4de","created":1751494488,"model":"claude-sonnet-4-20250514","object":"chat.completion","system_fingerprint":null,"choices": [        {"finish_reason":"stop","index":0,"message": {"content":"Hello! I'm doing well, thank you for asking. I'm here and ready to help with whatever you'd like to discuss or work on. How are you doing today?","role":"assistant","tool_calls":null,"function_call":null            }        }    ],"usage": {"completion_tokens":39,"prompt_tokens":13,"total_tokens":52,"completion_tokens_details":null,"prompt_tokens_details": {"audio_tokens":null,"cached_tokens":0        },"cache_creation_input_tokens":0,"cache_read_input_tokens":0    }}

Async (Docs)

fromlitellmimportacompletionimportasyncioasyncdeftest_get_response():user_message="Hello, how are you?"messages= [{"content":user_message,"role":"user"}]response=awaitacompletion(model="openai/gpt-4o",messages=messages)returnresponseresponse=asyncio.run(test_get_response())print(response)

Streaming (Docs)

fromlitellmimportcompletionmessages= [{"content":"Hello, how are you?","role":"user"}]# gpt-4oresponse=completion(model="openai/gpt-4o",messages=messages,stream=True)forpartinresponse:print(part.choices[0].delta.contentor"")# claude sonnet 4response=completion('anthropic/claude-sonnet-4-20250514',messages,stream=True)forpartinresponse:print(part)

Response chunk (OpenAI Format)

{"id":"chatcmpl-fe575c37-5004-4926-ae5e-bfbc31f356ca","created":1751494808,"model":"claude-sonnet-4-20250514","object":"chat.completion.chunk","system_fingerprint":null,"choices": [        {"finish_reason":null,"index":0,"delta": {"provider_specific_fields":null,"content":"Hello","role":"assistant","function_call":null,"tool_calls":null,"audio":null            },"logprobs":null        }    ],"provider_specific_fields":null,"stream_options":null,"citations":null}

Logging Observability (Docs)

fromlitellmimportcompletion## set env variables for logging tools (when using MLflow, no API key set up is required)os.environ["LUNARY_PUBLIC_KEY"]="your-lunary-public-key"os.environ["HELICONE_API_KEY"]="your-helicone-auth-key"os.environ["LANGFUSE_PUBLIC_KEY"]=""os.environ["LANGFUSE_SECRET_KEY"]=""os.environ["ATHINA_API_KEY"]="your-athina-api-key"os.environ["OPENAI_API_KEY"]="your-openai-key"# set callbackslitellm.success_callback= ["lunary","mlflow","langfuse","athina","helicone"]# log input/output to lunary, langfuse, supabase, athina, helicone etc#openai callresponse=completion(model="openai/gpt-4o",messages=[{"role":"user","content":"Hi 👋 - i'm openai"}])

LiteLLM Proxy Server (LLM Gateway) - (Docs)

📖 Proxy Endpoints -Swagger Docs

Quick Start Proxy - CLI

pip install'litellm[proxy]'

Step 1: Start litellm proxy

$ litellm --model huggingface/bigcode/starcoder#INFO: Proxy running on http://0.0.0.0:4000

Step 2: Make ChatCompletions Request to Proxy

Important

💡Use LiteLLM Proxy with Langchain (Python, JS), OpenAI SDK (Python, JS) Anthropic SDK, Mistral SDK, LlamaIndex, Instructor, Curl

importopenai# openai v1.0.0+client=openai.OpenAI(api_key="anything",base_url="http://0.0.0.0:4000")# set proxy to base_url# request sent to model set on litellm proxy, `litellm --model`response=client.chat.completions.create(model="gpt-3.5-turbo",messages= [    {"role":"user","content":"this is a test request, write a short poem"    }])print(response)

Proxy Key Management (Docs)

# Get the codegit clone https://github.com/BerriAI/litellm# Go to foldercd litellm# Add the master key - you can change this after setupecho'LITELLM_MASTER_KEY="sk-1234"'> .env# Add the litellm salt key - you cannot change this after adding a model# It is used to encrypt / decrypt your LLM API Key credentials# We recommend - https://1password.com/password-generator/# password generator to get a random hash for litellm salt keyecho'LITELLM_SALT_KEY="sk-1234"'>> .envsource .env# Startdocker compose up

Request

curl'http://0.0.0.0:4000/key/generate' \--header'Authorization: Bearer sk-1234' \--header'Content-Type: application/json' \--data-raw'{"models": ["gpt-3.5-turbo", "gpt-4", "claude-2"], "duration": "20m","metadata": {"user": "ishaan@berri.ai", "team": "core-infra"}}'

Expected Response

{"key":"sk-kdEXbIqZRwEeEiHwdg7sFA",# Bearer token"expires":"2023-11-19T01:38:25.838000+00:00"# datetime object}

Supported Providers (Website Supported Models |Docs)

Run in Developer mode

Services

Backend

Frontend

Enterprise

Contributing

Quick Start for Contributors

git clone https://github.com/BerriAI/litellm.gitcd litellmmake install-dev# Install development dependenciesmake format# Format your codemake lint# Run all linting checksmake test-unit# Run unit testsmake format-check# Check formatting only

Code Quality / Linting

Support / talk with founders

Why did we build this

Contributors