Repository files navigation

This example shows how to create aConfluent Kafka Cloud Producer/Consumer in GousingConfluent's Golang Client for Apache Kafka client and deploy it to Kubernetes.

• Linux (Ubuntu) or Mac OS

• gvm Go 1.24.2

  gvm install go1.24.2 --prefer-binary --with-build-tools --with-protobufgvm use go1.24.2 --default

• [Cross compilation on Ubuntu with CGO ] Optional

  InstallLLVM Compiler Infrastructure, release 17

  ./scripts/install-clang-17-ubuntu.sh

  Install libraries for cross compilation (Windows, etc.)

  ./scripts/install-cross-libs-ubuntu.sh

  Installosxcross for MacOS cross compilation

  ./scripts/install-osxcross-ubuntu.sh

• [Build ARM Images on x86 Hosts] Optional

  sudo apt-get update&& sudo apt-get install -y --no-install-recommends qemu-user-static binfmt-supportupdate-binfmts --enable qemu-armupdate-binfmts --display qemu-armdocker buildx build --platform linux/arm64 --file Dockerfile.consumer -t kafka-confluent-go-consumer:latest.

• Confluent Kafka CLI and tools

  curl -sL --http1.1 https://cnfl.io/cli| sh -s -- latestconfluent update# log in to a Confluent Cloud organizationconfluent login --save

• docker Optional

• GoReleaser Optional

• kubectl Optional

Following steps are required.

• Create Environment and Cluster -https://confluent.cloud/home

• Export Confluent Environment ID as CONFLUENT_ENV

  xdg-open https://confluent.cloud/environmentsexport CONFLUENT_ENV=

• Export Confluent cluster ID as CONFLUENT_CLUSTER

  xdg-open https://confluent.cloud/environments/$CONFLUENT_ENV/clustersexport CONFLUENT_CLUSTER=

• Select Environment

  confluent environment use$CONFLUENT_ENV

• Select Cluster

  confluent kafka cluster use$CONFLUENT_CLUSTERconfluent login --save

• Create a new API key and secret pair

  confluent api-key create --resource$CONFLUENT_CLUSTER

• Export previously created KEY and SECRET

  export CONFLUENT_API_KEY=export CONFLUENT_API_SECRET=

• Use an API key and secret in the CLI

  confluent api-key use$CONFLUENT_API_KEY --resource$CONFLUENT_CLUSTER

• Export Confluent Kafka Cluster Bootstrap Server -Cluster settings -> Endpoints -> Bootstrap server, createkafka.properties from the template and add it to the git repository.

  xdg-open https://confluent.cloud/environments/$CONFLUENT_ENV/clusters/$CONFLUENT_CLUSTER/settings/kafkaexport CONFLUENT_BOOTSTRAP_SERVER=sed -e"s%BTSTRP%$CONFLUENT_BOOTSTRAP_SERVER%g" ./tmpl/kafka.properties.tmpl> ./kafka.properties

• Create .env file

  sed -e"s%BTSTRP%$CONFLUENT_BOOTSTRAP_SERVER%g" -e"s%APIKEY%$CONFLUENT_API_KEY%g" -e"s%APISECRET%$CONFLUENT_API_SECRET%g" ./tmpl/.env.tmpl> ./.env

• Create Confluent Kafka topic

  confluent kafka topic create test-topic

confluent kafka topic listconfluent kafka topic produce test-topicconfluent kafka topic consume -b test-topic

To deploy on Kubernetes create configmap and secret:

# create configmap from Kafka properties filekubectl create configmap kafka-config --from-file kafka.properties -o yaml --dry-run=client>./k8s/cm.yaml# store $CONFLUENT_API_KEY and $CONFLUENT_API_SECRET as k8s secretsed -e"s%USR%`echo -n$CONFLUENT_API_KEY|base64 -w0`%g" -e"s%PWD%`echo -n$CONFLUENT_API_SECRET|base64 -w0`%g" ./tmpl/sc.yaml.tmpl> ./k8s/sc.yaml

and then run

make k8s-deploy

make consumer-image-run

make runp

Commands :help                 - List available tasksclean                - Cleanupbuild                - Buildtest                 - Run testsupdate               - Update dependency packages to latest versionsget                  - Download and install dependency packagesrelease              - Create and push a new tagversion              - Print current version(tag)consumer-image-build - Build Consumer Docker imageconsumer-image-run   - Run a Docker imageconsumer-image-stop  - Run a Docker imagerunp                 - Run producerrunc                 - Run consumerk8s-deploy           - Deploy to Kubernetesk8s-undeploy         - Undeploy from Kubernetes