Repository files navigation

Setup which shows the following ideas

• Sending nginx logs to elasticsearch via fluentd and viewing in kibana.
• Sending the nginx logs to kafka, store the extracted data in mongodb, watch for changes in node.js socket.io server via mongodb changestreams, send events to front end via socket.io connection.

Its just a demo application.

• Docker
• Minikube
• Kubectl
• Helm
• Terraform

• Setup and Teardown
• Worflow

• nginx-logs-producer
  • nginx, fluentd with elasticsearch and kafka plugins
  • fluentd takes nginx logs and sends to elasticsearch and kafka
• kafka-nginx-consumer
  • subscribes to the nginx-access topic
  • Upon receiving messages, parses and stores in mongodb
• realtime-publisher
  • node.js socket.io server which listens to mongodb changestreams and emits the changes to socket
• realtime-viewer
  • html page which have socket.io client listens for event and updates the table of nginx events
• kibana
  • logs which are sent to elasticsearch, are viewed in kibana

• Prebuild images - Dockerhub
  • aravindkumars/realtime-viewer:1.0
  • aravindkumars/kafka-nginx-consumer:1.0
  • aravindkumars/nginx-fluentd:1.0
  • aravindkumars/nginx:1.0
• Building docker images
  • realtime-viewer

cd realtime-viewerdocker build -t aravindkumars/realtime-viewer:1.0 -t aravindkumars/realtime-viewer:latest. docker push aravindkumars/realtime-viewer:1.0docker push aravindkumars/realtime-viewer:latest

• kafka-consumer

cd kafka-consumerdocker build -t aravindkumars/kafka-nginx-consumer:1.0 -t aravindkumars/kafka-nginx-consumer:latest. docker push aravindkumars/kafka-nginx-consumer:1.0docker push aravindkumars/kafka-nginx-consumer:latest

• nginx-fluentd

cd fluentd-dockerdocker build -t aravindkumars/nginx-fluentd:1.0 -t aravindkumars/nginx-fluentd:latest. docker push aravindkumars/nginx-fluentd:1.0docker push aravindkumars/nginx-fluentd:latest

• nginx

cd nginx-dockerdocker build -t aravindkumars/nginx-docker:1.0 -t aravindkumars/nginx-docker:latest. docker push aravindkumars/nginx-docker:1.0 docker push aravindkumars/nginx-docker:latest

• Directory

cd terraform/

• Prerequisites
  • Change the backend in main.tf
  • Change the kubernetes and helm providers in main.tf
• Backend Consul setup
  • based on the consul container ip view in browser. ex:http://172.17.0.2:8500

docker run -d --name=dev-consul -e CONSUL_BIND_INTERFACE=eth0 consul

• Docker images
  • Change the docker images in variables.tf
• Infra setup

terraform initterrafor apply

• Infra teardown
  • after destroying using terraform, need to manually delete persistent volumes. below command deletes all the pvc. use it carefully

terraform destroyforiin$(kubectl get pvc -o go-template --template"{{ range .items }} {{ .metadata.name }} {{ end }}");do kubectl delete"pvc/${i}";done

• Forward kibana port and view in browserhttp://localhost:5601

export KIBANA_POD_NAME=$(kubectl get pods --namespace default -l"app=kibana,release=kibana" -o jsonpath="{.items[0].metadata.name}")kubectl port-forward --namespace default$KIBANA_POD_NAME 5601:5601

• Get NodePort for nginx and replace the minikube ip

minikube ipterraform output logs_producer_url

• Get NodePort for realtime viewer and replace the minikube ip

minikube ipterraform output realtime_viewer_url

• Github -https://github.com/aravindkumarsvg
• Github Gists -https://gist.github.com/aravindkumarsvg
• Dockerhub -https://hub.docker.com/u/aravindkumars
• Vagrant Boxes -https://app.vagrantup.com/aravindkumarsvgj
• Asciinema -https://asciinema.org/~aravindkumarsvgm