Repository files navigation

Get your data via REST into Apache Kafka

Often in your environment you might not be allowed to directly access your kafka from web (e.g. you have a web- and application-zone, but kafa is installed in application zone and not accessible via direct-path). But still you have the need to get data coming from outside into your topics. So one easy way is: kafka-rest-proxy. It provides a simple RESTful API, forwarding your payload to kafka.

1. InstallApache Kafka download and unarchive orbrew install kafka

2. Start kafka

   bin/zookeeper-server-start.sh config/zookeeper.propertiesbin/kafka-server-start.sh config/server.properties

   or if installed via brew

   zookeeper-server-start /usr/local/etc/kafka/zookeeper.propertieskafka-server-start /usr/local/etc/kafka/server.properties

3. Create a kafka topic

   Homebrew:

   kafka-topics --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

4. Check if topic is there

   Homebrew:

   kafka-topics --list --zookeeper localhost:2181

5. Start a console conusmer to check your posted messsages

   Homebrew:

   kafka-console-consumer --topic test --zookeeper localhost:2181

1. Clone git repository
2. Build executable./gradlew clean build
3. Start./gradlew bootRun

Configuration file can be found insrc/main/resources and is namedapplication.yml.

• Put all Apache Kafka properties beneath nodeproducer (seeConfiguration)
• For configuration ofSpring Boot seeConfiguration

curl -XPOST -H "Content-Type:application/json" -d "{\"message\": \"kafka-rest-proxy is on github.\"}" http://localhost:8080/topics/testcurl -XPOST -H "Content-Type:application/json" -d "{\"message\": \"kafka-rest-proxy is on github.\"}" http://localhost:8080/topics/test?key=key

curl -XGET http://localhost:8080/topicslist  //since consumer is not thread-safe topic-list gets up-to-date approx. every 2 sec in a scheduled task.curl -XGET http://localhost:8080/metricscurl -XGET http://localhost:8080/topicsinfo/test

curl -XGET http://localhost:8080/actuator/health

* Note: spring-boot-2.0.1, spring-kafka-2.2.0, java version 10.0.2 (2018-07-17)