- Notifications
You must be signed in to change notification settings - Fork75
Description
Bug Description
Using latest version of Bytewax (0.21.1), latest version of helm chart (0.0.9) and deploying this in EKS, I encountered a rather odd behaviour. I use kustomize and ArgoCD to deploy the Bytewax helm chart with 1 operator. My Bytewax flow is designed to process messages from Kafka.
If I start with 2 replicas, they both run fine. I then update the number of replicas to 3. It seems the last pod (pod-2) will always fail to connect properly to pod-1.
worker 2: error connecting to worker 1: Connection refused (os error 111); retryingworker 2: error connecting to worker 1: Connection refused (os error 111); retryingworker 2: error connecting to worker 1: Connection refused (os error 111); retryingworker 2: error connecting to worker 1: Connection refused (os error 111); retryingworker 2: error connecting to worker 1: Connection refused (os error 111); retryingworker 2: error connecting to worker 1: Connection refused (os error 111); retryingworker 2: error connecting to worker 1: Connection refused (os error 111); retryingworker 2: error connecting to worker 1: Connection refused (os error 111); retryingworker 2: error connecting to worker 1: Connection refused (os error 111); retryingworker 2: error connecting to worker 1: Connection refused (os error 111); retryingworker 2: error connecting to worker 1: Connection refused (os error 111); retryingworker 2: error connecting to worker 1: Connection refused (os error 111); retryingworker 2: error connecting to worker 1: Connection refused (os error 111); retryingworker 2: error connecting to worker 1: Connection refused (os error 111); retryingworker 2: error connecting to worker 1: Connection refused (os error 111); retryingworker 2: error connecting to worker 1: Connection refused (os error 111); retryingworker 2: error connecting to worker 1: Connection refused (os error 111); retryingworker 2: error connecting to worker 1: Connection refused (os error 111); retryingworker 2: error connecting to worker 1: Connection refused (os error 111); retryingworker 2: error connecting to worker 1: Connection refused (os error 111); retryingworker 2: error connecting to worker 1: Connection refused (os error 111); retryingWhen this happens, all instances of my Bytewax flow will never receive any messages from Kafka. The only way to get things right is to completely delete the deployment and redeploy it. After delete + redeploy, everything works as normal and my Bytewax flow starts receiving messages from Kafka.
Is this a bug?
Python version (python -V)
Using bytewax:0.21.1 docker image
Bytewax version (pip list | grep bytewax)
0.21.1
Operating System version (uname -morp)
Using bytewax:0.21.1 docker image
Relevant log output
See above
Steps to Reproduce
See above