Repository files navigation

A Python library for automatically optimizing Kafka producer configurations based on topic-specific recommendations.

Superstream Clients works as a Python import hook that intercepts Kafka producer creation and applies optimized configurations without requiring any code changes in your application. It dynamically retrieves optimization recommendations from Superstream and applies them based on impact analysis.

• kafka-python
• aiokafka
• confluent-kafka
• Faust
• FastAPI event publishers
• Celery Kafka backends
• Any custom wrapper around these Kafka clients

• Zero-code integration: No code changes required in your application
• Dynamic configuration: Applies optimized settings based on topic-specific recommendations
• Intelligent optimization: Identifies the most impactful topics to optimize
• Graceful fallback: Falls back to default settings if optimization fails

pip install superstream-clients&& python -m superclient install_pth

That's it! Superclient will now automatically load and optimize all Kafka producers in your Python environment.

After installation, superclient works automatically. Just use your Kafka clients as usual.

When using Superstream Clients with containerized applications, include the package in your Dockerfile:

FROM python:3.8-slim# Install superclientRUN pip install superstream-clientsRUN python -m superclient install_pth# Your application codeCOPY . /appWORKDIR /app# Run your applicationCMD ["python","your_app.py"]

• SUPERSTREAM_TOPICS_LIST: Comma-separated list of topics your application produces to

• SUPERSTREAM_LATENCY_SENSITIVE: Set to "true" to prevent any modification to linger.ms values
• SUPERSTREAM_DISABLED: Set to "true" to disable optimization
• SUPERSTREAM_DEBUG: Set to "true" to enable debug logs

Example:

export SUPERSTREAM_TOPICS_LIST=orders,payments,user-eventsexport SUPERSTREAM_LATENCY_SENSITIVE=true

• Python 3.8 or higher
• Kafka cluster that is connected to the Superstream's console
• Read and write permissions to thesuperstream.* topics

Apache License 2.0