Repository files navigation

Django + Kafka + Celery 🚀

• pip install confluent-kafka
• Download confluent platform from:https://www.confluent.io/download/ and unzip the file.
• export CONFLUENT_HOME=<path-to-confluent>
• export PATH=$PATH:$CONFLUENT_HOME/bin
• $CONFLUENT_HOME/bin/confluent-hub install \ --no-prompt confluentinc/kafka-connect-datagen:latest

The output would be:

Running in a "--no-prompt" mode...Completed

• Start the service:confluent local startThe output should resemble:

Starting ZookeeperZookeeper is [UP]Starting KafkaKafka is [UP]Starting Schema RegistrySchema Registry is [UP]Starting Kafka RESTKafka REST is [UP]Starting ConnectConnect is [UP]Starting KSQL ServerKSQL Server is [UP]Starting Control CenterControl Center is [UP]

• Stop services:confluent local stop

Or set-up Kafka and Zookeeper Separately

• Simply put, Kafka is a distributed publish-subscribe messaging system that maintains feeds of messages in partitioned and replicated topics.
• In the simplest way there are three players in the Kafka ecosystem: producers, topics (run by brokers) and consumers.

• brew install zookeeper
• Where is installation directory of zookeeper :/usr/local/Cellar/zookeeper

• In foregroundzkServer start
• In backgroundbrew services start zookeeper

• brew install kafka
• Where is installation directory of Kafka :/usr/local/Cellar/kafka

• In foregroundbrew services start kafka
• In backgroundzookeeper-server-start /usr/local/etc/kafka/zookeeper.properties & kafka-server-start /usr/local/etc/kafka/server.properties

For details on how to set-up a django project with best practices:https://pyblog.xyz/django-initial-setup/

• pip install amqp

• pip install celery==4.4.0 (Note: Celery 5.0 is not compatible with django-celery-beat)

• pip install django-celery-beat

• Start worker and beatcelery -A django_kafka beat -l INFO -S django

• Start RabbitMQbrew services start rabbitmq

INSTALLED_APPS = [    ...,    'django_celery_beat',]

• Run migrations:python manage.py migrate django_celery_beat

Note: In this project, the same Django project is the Producer and Consumer, but you can choose you have a standalone consumer.

• https://docs.confluent.io/current/getting-started.html
• https://www.instaclustr.com/apache-kafka-architecture/
• https://github.com/confluentinc/confluent-kafka-python
• https://docs.confluent.io/current/schema-registry/index.html
• https://docs.confluent.io/current/quickstart/ce-quickstart.html