Repository files navigation

This is a series of Jupyter Notebooks on how to start with Apache Kafka® and Python.You can try these notebooks in order to learn the basic concepts of Apache Kafka in an environment containing markdown text, media and executable code on the same page.

The notebooks are based on a managed Apache Kafka instance created onAiven's website, but can be also customised to any Apache Kafka instance running locally with SSL authentication. Aiven's offer 300$ of free credit that you can redeem by creating your account onAiven's website.

If you have any question or improvement suggestion regarding the notebooks, please open an issue. Any contributions are welcome!

You can access the notebooks via Jupyterlab, this example will be based on docker

1. clone the repository
2. open a terminal
3. go to the folder where the repository has been cloned
4. run the following

docker run --rm -p 8888:8888 \  -e JUPYTER_ENABLE_LAB=yes  \  -v "$PWD":/home/jovyan/work \  jupyter/datascience-notebook

You'll see a folder namedwork on the top left, under it you'll find the list of notebooks.

This repository contains the following notebooks.

• 00 - Aiven Setup.ipynb: Creates requires Apache Kafka and PostgreSQL instances on Aiven website
• 01 - Producer.ipynb: Creates a Python Producers and produces the first messages
• 02 - Consumer.ipynb: reads the messages produced by01 - Producer.ipynb
• 03 - 00 - Partition Producer.ipynb: creates a topic with two partitions and pushes records in each partition
• 03 - 01 - Consumer - Partition 0.ipynb: reads messages pushed by03 - 00 - Partition Producer.ipynb on partition 0
• 03 - 02 - Consumer - Partition 1.ipynb: reads messages pushed by03 - 00 - Partition Producer.ipynb on partition 1
• 04 - New Consumer Group.ipynb: creates a new consumer part of a new consumer group and reads messages produced by01 - Producer.ipynb
• 05 - Kafka Connect.ipynb: creates a new topic containing JSON messages with schema and payload, and then creates a Kafka Connect JDBC Connector to PostgreSQL
• ON - Aiven - Delete Services.ipynb: deletes all the instances created by00 - Aiven Setup.ipynb

The notebooks are divided per Apache Kafka functionality.

00 - Aiven Setup.ipynb notebook downloadsAiven's command line interface and creates an Apache Kafka and a PostgreSQL instance.

Please change<INSERT_TOKEN_HERE> and<INSERT_EMAIL_HERE> with a valid email address and token created onAiven's website. The notebook creates the instances and also stores all the required connection credentials locally.

01 - Producer.ipynb Creates a Python Apache Kafka Producer and produces the first messages. After the first message is produced, open the02 - Consumer.ipynb notebook and pace it alongside the Producer.

02 - Consumer.ipynb reads from the topic where01 - Producer wrote. But it does it from the point in time that it attaches to Apache Kafka, not going back to history.

If you want to read messages created with01 - Producer you need to run02 - Consumer.ipynb's last code block before producing any messages on01 - Producer. This behaviour is Apache Kafka's default and can be changed by adding a line'auto.offset.reset'='earliest' to the consumer properties.

Partitions is Apache Kafka are a way to divide messages belonging to the same topic in sub-logs.

• 03 - 00 - Partition Producer.ipynb creates a topic with two partitions usingKafkaAdmin and sends a message to each partition.We can then open both03 - 01 - Consumer - Partition 0.ipynb and03 - 02 - Consumer - Partition 1.ipynb which will read messages fromPartition 0 andPartition 1 respectively.

Messages in Apache Kafka are not deleted when read from a consumer. This makes them available for other consumers to be read.04 - New Consumer Group.ipynb creates a new consumer part of the a new Consumer Group and reads from the topic where01 - Producer wrote. We can check now, by sending a message from the01 - Producer notebook, that we can receive it both in02 - Consumer.ipynb and04 - New Consumer Group.

Apache Kafka Connect® is a prebuilt framework enabling an easy integration of Apache Kafka with existing data sources or sinks. Aiven providesKafka connect as managed service making the integration a matter of a single config file.05 - Kafka Connect.ipynb: Creates a new Kafka topic containing messages with both schema and payload, and then pushes them to a PostgreSQL database via Apache Kafka Connect.

Once you're done, you can delete all the services create onAiven's website by executing the code inON - Aiven - Delete Services.ipynb

We maintain some other resources that you may also find useful:

• Command Line Magic with avn
• Teach Yourself Apache Kafka with Jupyter Notebooks

This project is licensed under theApache License, Version 2.0.

Apache Kafka is either a registered trademark or trademark of the Apache Software Foundation in the United States and/or other countries. Aiven has no affiliation with and is not endorsed by The Apache Software Foundation.