Repository files navigation

Building streaming event pipeline around Apache Kafka and its ecosystem (REST Proxy, Kafka Connect), that allows to simulate and display the status of train lines in real time.

You are constructing a streaming event pipeline around Apache Kafka and its ecosystem. Using public data from theChicago Transit Authority you will construct an event pipeline around Kafka that allows us to simulate and display the status of train lines in real time.When the project is completed, you will be able to monitor a website to watch trains move from station to station.

• InstallDocker, make sure Docker Compose is installed too
• If you are on Windows machine, install: Windows Subsystem for Linux (WSL) version 2link
• Install Ubuntu 20.04
• If you are on Windows machine, also installlibrdkafka librarylink, make sure to install what is needed fromthis link too
• Inside Ubuntu, in a terminal instance rundocker-compose up
  • You can check status of the environment by runningdocker-compose ps in a new terminal instance

There are 2 pieces of the simulation,producer andconsumer, each of them can be run separately. To run end-to-end simulation, run all the pieces together(in different terminal windows):

1. To run theproducer:

   • cd producers
   • virtualenv venv
   • . venv/bin/activate
   • pip install -r requirements.txt
   • python simulation.pyHitCtrl+C at any time to exit.

2. To run the Faust Stream Processing Application:

   • cd consumers
   • virtualenv venv
   • . venv/bin/activate
   • pip install -r requirements.txt
   • faust -A faust_stream worker -l info

3. To run the KSQL Creation Script:

   • cd consumers
   • virtualenv venv
   • . venv/bin/activate
   • pip install -r requirements.txt
   • python ksql.py

4. To run theconsumer:

   • cd consumers
   • virtualenv venv
   • . venv/bin/activate
   • pip install -r requirements.txt
   • cpython server.pyHitCtrl+C at any time to exit.

• To stop Docker rundocker-compose stop in a terminal instance
• To clean up the containers to reclaim disk space, rundocker-compose rm -v

The output of the project looks like this