Repository files navigation

git.io/nagios-plugin-kafka

Kafka 0.9+ API CLI Tester & Advanced Nagios Plugin with Kerberos support, written in Scala.

Tested on Hortonworks HDP 2.4.0 with Kerberos + Ranger ACLs and Apache Kafka 0.8.x / 0.9.0.1docker images with regular ACLs.

You may need to change the Kafka library version inpom.xml /build.sbt /build.gradle before building to match your deployed Kafka server / cluster otherwise it may hang when run due to version / protocol mismatch.

SeeThe Advanced Nagios Plugins Collection for many more related enterprise monitoring programs.

Hari Sekhon

Cloud & Big Data Contractor, United Kingdom

(ex-Cloudera, former Hortonworks Consultant)

(you're welcome to connect with me on LinkedIn)

This project builds a single self-contained Java jar file with all dependencies included and can simply be run on the command line with full switch option support:

java -jar check_kafka.jar --help

and there is an optional convenience shell wrapper script at the top level to make commands shorter:

./check_kafka --help

Run against one or more Kafka brokers, comma separated:

$ ./check_kafka --brokers localhost:9092 --topictestOK: Kafka broker successfully returned unique message via topic'test' partition'0', writetime = 0.185s,readtime = 0.045s, totaltime = 1.729s| write_time=0.185s read_time=0.045s total_time=1.729s

Use the--verbose switch to also show the brokers list that were tested. If you have specified one of the kerberos switches (or edited the consumer/producer properties files to do so) then the output will additionally contain the markerwith sasl authentication to let you know that it was a secure configuration that was tested (originally I called thiswith kerberos but technically it may not be in future).

OK: Kafka broker '<hortonworks_host>:6667' successfully returned unique message via topic 'topic3' partition '0' with sasl authentication, write time = 0.148s, read time = 0.043s, total time = 0.691s | write_time=0.148s read_time=0.043s total_time=0.691s

This program only supports Kafka 0.9+ as the API changed (again) and Kerberos security was only added in the 0.9 API. For Kafka versions before 0.9 you can find Python and Perl versions of this program in theAdvanced Nagios Plugins Collection that support 0.8 onwards (they dosn't support Kafka <= 0.7 as the API changed in 0.8 too and the underlying libraries in those languages don't support Kafka <= 0.7).

It appears that several errors are caught too early in the new Kafka Java API and result in embedded looping retry behaviour on encountering errors (visible in debug level logging of the base library).

I haven't found a great way of handle that behaviour as it's not exposed to the client code so it ends up being handled via my generic default self timeout mechanism that I apply to all my tools. Hence if you specify an incorrect--brokers <host>:<port> or the Kafka brokers are down or you fail to negotiate the protocol due to security settings you will only receive a genericUNKNOWN: self timed out after 10 secs message as the code self terminates.

Otherwise the Kafka API would just hang there indefintely as it keeps retrying deeper in the library. I've tried various settings to get it to time out but nothing worked and I even posted to the Kafka users mailing list without an answer. If you know of a setting that will make the Kafka Client library time out and return the more specific error then please let me know and I'll update this code accordingly.

See theconf/ directory for JAAS kerberos configurations.

If you're running the code on a Hortonworks Kafka broker it'll auto-detect the HDP configuration and use that.

A Dockerized pre-built version is available onDockerHub.

If you have docker installed this one command will download and run it:

docker run harisekhon/nagios-plugin-kafka check_kafka --help

curl -L https://git.io/nagios-plugin-kafka-bootstrap| sh

OR

Maven, Gradle and SBT automated builds are all provided.

A self-contained jar file with all dependencies will be created and symlinked tocheck_kafka.jar at the top level.

The Maven and Gradle builds are best as they will auto bootstap and run with no prior installed dependencies other than Java andmake to kick it off.

The defaultmake build will trigger a Gradle bootstrap from scratch with an embedded checksum for security:

make

You can call any one of the 3 major build systems explicitly instead, which will recurse to build the library submodule using the same mechanism:

Maven:

make mvn

Gradle:

make gradle

SBT:

make sbt

If using bespoke internal domains such as.local,.intranet,.vm,.cloud etc. that aren't part of the official IANA TLD list then this is additionally supported via a custom configuration filelib/resources/custom_tlds.txt containing one TLD per line, with support for # comment prefixes. Just add your bespoke internal TLD to the file and it will then pass the host/domain/fqdn validations.

Continuous Integration is run on this repo with tests for success and failure scenarios:

• unit tests for the custom supportingjava library
• integration tests of the top level programs using the libraries for things like option parsing
• functional tests for the top level programs usingDocker containers

To trigger all tests run:

maketest

which will start with the underlying libraries, then move on to top level integration tests and functional tests using docker containers if docker is available.

TheAdvanced Nagios Plugins Collection has both Perl and Python predecessors to this program which work with Kafka 0.8+. The main differenitator with this Scala version is that it uses the new native 0.9+ Java API which has Kerberos support (the dynamic language versions were built on libraries for Kafka 0.8).

git.io/nagios-plugin-kafka

The rest of my original source repos arehere.

Pre-built Docker images are available on myDockerHub.