Gateway
- Overview
- Call any LLM
- Call the OpenAI Responses API
- Configure models & providers
- Configure functions & variants
- Create a prompt template
- Generate structured outputs
- Generate embeddings
- Batch Inference
- Episodes
- Inference Caching
- Inference-Time Optimizations
- Metrics & Feedback
- Multimodal Inference
- Retries & Fallbacks
- Streaming Inference
- Tool Use (Function Calling)
- Benchmarks
- Clients
- Configuration Reference
- Data Model
Optimization
Evaluations
Experimentation
Deployment
Gateway
Inference Caching
Learn how to use inference caching with TensorZero Gateway.
The TensorZero Gateway supports caching of inference responses to improve latency and reduce costs.When caching is enabled, identical requests will be served from the cache instead of being sent to the model provider, resulting in faster response times and lower token usage.
Usage
The TensorZero Gateway supports the following cache modes:write_only(default): Only write to cache but don’t serve cached responsesread_only: Only read from cache but don’t write new entrieson: Both read from and write to cacheoff: Disable caching completely
Example
from tensorzeroimport TensorZeroGatewaywith TensorZeroGateway.build_http(gateway_url="http://localhost:3000")as client: response= client.inference( model_name="openai::gpt-4o-mini", input={ "messages": [ { "role":"user", "content":"What is the capital of Japan?", } ] }, cache_options={ "enabled":"on",# read and write to cache "max_age_s":3600,# optional: cache entries >1h (>3600s) old are disregarded for reads }, )print(response)Technical Notes
- The cache applies to individual model requests, not inference requests.This means that the following will be cached separately:multiple variants of the same function;multiple calls to the same function with different parameters;individual model requests for inference-time optimizations;and so on.
- The
max_age_sparameter applies to the retrieval of cached responses.The cache does not automatically delete old entries (i.e. not a TTL). - When the gateway serves a cached response, the usage fields are set to zero.
- The cache data is stored in ClickHouse.
- For batch inference, the gateway only writes to the cache but does not serve cached responses.
- Inference caching also works for embeddings, using the same cache modes and options as chat completion inference.Caching works for single embeddings.Batch embedding requests (multiple inputs) will write to the cache but won’t serve cached responses.
⌘I