Movatterモバイル変換

[0]ホーム
URL:

Skip to main content
Optimization
Evaluations
  • Overview
Deployment

Deploy the TensorZero Gateway

Learn how to deploy and customize the TensorZero Gateway.

The TensorZero Gateway is the core component that handles inference requests and collects observability data.It’s easy to get started with the TensorZero Gateway.
You need to only deploy a standalone gateway if you plan to use the TensorZero UI or interact with the gateway using programming languages other than Python.The TensorZero Python SDK includes a built-in embedded gateway, so you don’t need to deploy a standalone gateway if you’re only using Python.See theClients page for more details on how to interact with the TensorZero Gateway.

Deploy

The gateway requires one of the following command line arguments:
  • --default-config: Use default configuration settings.
  • --config-file path/to/tensorzero.toml: Use a custom configuration file.
    --config-file supports glob patterns, e.g.--config-file /path/to/**/*.toml.
  • --run-clickhouse-migrations: Run ClickHouse database migrations and exit.
  • --run-postgres-migrations: Run PostgreSQL database migrations and exit.
There are many ways to deploy the TensorZero Gateway.Here are a few examples:
You can easily run the TensorZero Gateway locally using Docker.If you don’t have custom configuration, you can use:
Running with Docker (default configuration)
docker run \  --env-file .env \  -p 3000:3000 \  tensorzero/gateway \  --default-config
If you have custom configuration, you can use:
Running with Docker (custom configuration)
docker run \  -v "./config:/app/config" \  --env-file .env \  -p 3000:3000 \  tensorzero/gateway \  --config-path config/tensorzero.toml
We provide an example production-gradedocker-compose.yml for reference.
We provide a reference Helm chart in ourGitHub repository.You can use it to run TensorZero in Kubernetes.The chart is available onArtifactHub.
You can build the TensorZero Gateway from source and run it directly on your host machine usingCargo.
Building from source
cargo run --profile performance --bin gateway -- --config-file path/to/your/tensorzero.toml
See theoptimizing latency and throughput guide to learn how to configure the gateway for high-performance deployments.

Configure

Set up model provider credentials

The TensorZero Gateway accepts the following environment variables for provider credentials.Unless you specify an alternative credential location in your configuration file, these environment variables are required for the providers that are used in a variant with positive weight.If required credentials are missing, the gateway will fail on startup.Unless customized in your configuration file, the following credentials are used by default:
ProviderEnvironment Variable(s)
AnthropicANTHROPIC_API_KEY
AWS BedrockAWS_ACCESS_KEY_ID,AWS_SECRET_ACCESS_KEY (seedetails)
AWS SageMakerAWS_ACCESS_KEY_ID,AWS_SECRET_ACCESS_KEY (seedetails)
Azure OpenAIAZURE_OPENAI_API_KEY
FireworksFIREWORKS_API_KEY
GCP Vertex AI AnthropicGCP_VERTEX_CREDENTIALS_PATH (seedetails)
GCP Vertex AI GeminiGCP_VERTEX_CREDENTIALS_PATH (seedetails)
Google AI Studio GeminiGOOGLE_AI_STUDIO_GEMINI_API_KEY
GroqGROQ_API_KEY
HyperbolicHYPERBOLIC_API_KEY
MistralMISTRAL_API_KEY
OpenAIOPENAI_API_KEY
OpenRouterOPENROUTER_API_KEY
TogetherTOGETHER_API_KEY
xAIXAI_API_KEY
See.env.example for a complete example with every supported environment variable.

Set up custom configuration

Optionally, you can use a configuration file to customize the behavior of the gateway.SeeConfiguration Reference for more details.
TensorZero collectspseudonymous usage analytics to help our team improve the product.The collected data includesaggregated metrics about TensorZero itself, but does NOT include your application’s data.To be explicit: TensorZero does NOT share any inference input or output.TensorZero also does NOT share the name of any function, variant, metric, or similar application-specific identifiers.Seehowdy.rs in the GitHub repository to see exactly what usage data is collected and shared with TensorZero.To disable usage analytics, set the following configuration in thetensorzero.toml file:
tensorzero.toml
[gateway]disable_pseudonymous_usage_analytics =true
Alternatively, you can also set the environment variableTENSORZERO_DISABLE_PSEUDONYMOUS_USAGE_ANALYTICS=1.

Set up observability with ClickHouse

Optionally, the TensorZero Gateway can collect inference and feedback data for observability, optimization, evaluations, and experimentation.Afterdeploying ClickHouse, you need to configure theTENSORZERO_CLICKHOUSE_URL environment variable with the connection details.If you don’t provide this environment variable, observability will be disabled.We recommend setting up observability early to monitor your LLM application and collect data for future optimization, but this can be done incrementally as needed.

Customize the logging format

Optionally, you can provide the following command line argument to customize the gateway’s logging format:
  • --log-format: Set the logging format to eitherpretty (default) orjson.

Add a status or health check

The TensorZero Gateway exposes endpoints for status and health checks.The/status endpoint checks that the gateway is running successfully.
GET /status
{"status":"ok" }
The/health endpoint additionally checks that it can communicate with ClickHouse (if observability is enabled).
GET /health
{"gateway":"ok","clickhouse":"ok" }
⌘I
[8]ページ先頭
©2009-2025 Movatter.jp