Last year I had written about theimportance of service monitoring and observability.
While service monitoring is important, service monitoring requires active involvement from application maintainers. When supporting applications and services which service real users, it's important to become aware of problems when they first happen.
The longer your service is down, experiencing errors or performance issues, or otherwise buggy, the more negative an experience you provide for your users.
Alerting provides application developers a passive mechanism for becoming aware of problems involving their applications.
Instead of being informed by users that a problem is occurring in an application, developers receive automated alarms which inform them a problem is occurring.

Kubernetes Alerting

Alerting isn't natively provided in Kubernetes. Fortunately there are a few open source applications which can satisfy the alerting needs. As mentioned in my previous post,Linkerd provides many monitoring tools out-of-the box, but some customizations need to be made in order to provide alerting.

• Prometheus is used to provide metrics monitoring (Memory usage, CPU usage, Network usage, etc.)
• Alertmanager is used to create and manage alarms using Prometheus metrics
• Grafana A visualization application which displays graphs and charts based on data received from metrics. Grafana also provides alerting.
• Prometheus Operator a project which expresses Prometheus services, rules, and alarms as Kubernetes Custom Resource Definitions (CRDs). Prometheus Operator is a Kubernetes-idiomatic way of declaring Prometheus services.

Alerting Channels

Before going through the process of setting up alarms, it is important do decide how you want to be notified.
For this tutorial, I'll be setting up email alerts.
Simple Mail Transfer Protocol (SMTP) is the service which allows applications, in this case our monitoring services, to send emails to the people supporting the applications. By default, most Internet Service Providers (ISPs), Email Providers, and Cloud Providers block the port that SMTP sends emails on.

Setting up a SMTP Relay

A SMTP Relay is a service which can be used as a proxy for SMTP traffic when you want to send emails external to your Local Area Network (LAN). I've researched a few SMTP relays and ultimately decided on Dynu.
Dynu won me over through providingeasy-to-use documentation on how to set up their SMTP relay service. Also at $9/year, the service is not unaffordable.

After opening an account with Dynu, my next step was to deploy the SMTP service onto Kubernetes.
Since I've started working with Kubernetes, I've discovered the convenience of working with Helm charts.ArtifuctHub is now my go-to for finding quick and easy helm charts to install onto Kubernetes.
I decided to deploy SMTP using thebokysan postfix chart.
I started out with these initialvalues.yml for the helm chart:

config:general:TZ:America/New_YorkLOG_FORMAT:jsonRELAYHOST:"{{relay_host}}"ALLOWED_SENDER_DOMAINS:"cluster.local{{domain_name}}"secret:RELAYHOST_USERNAME:"{{relay_username}}"RELAYHOST_PASSWORD:"{{relay_password}}"

I installed the helm release usingAnsible, but you can install with the following helm commands:

helm repo add docker-postfix https://bokysan.github.io/docker-postfix/helminstall--values values.yml--namespace=mail--create-namespace sender docker-postfix/mail

Forking the Postfix Base Image

Prometheus Alertmanager is built on Golang. A limitation of the SMTP integration in Alertmanager is thatTLS is required for remote connections.
I attempted to use thecerts.create value from the helm chart values, but the resulting certificate was not accepted by Alertmanager.
As a result, I had to create anew image from bokysan's Postfix image.

The new image detects for the following files to be mounted in/mnt/certs:

• tls.crt
• tls.key
• ca.crt

If the files are present, they are copied into/etc/ssl and the postfix configuration at/etc/postfix/main.cf is updated to enable the SSL settings.

This set up is useful if usingcert-manager to set up certificates using Kubernetes Secrets.

⚠️ When using the Postfix image, please note that thesigning algorithm for your certificates needs to be RSA

I'm still able to use Bokysan's Helm chart with the following values:

image:repository:mikeyglitz/postfixtag:latestextraVolumes:-name:tls-certsecret:secretName:mail-tlsextraVolumeMounts:-name:tls-certmountPath:/mnt/certsreadOnly:trueconfig:general:TZ:America/New_YorkLOG_FORMAT:jsonRELAYHOST:"{{relay_host}}"ALLOWED_SENDER_DOMAINS:"cluster.local{{domain_name}}"secret:RELAYHOST_USERNAME:"{{relay_username}}"RELAYHOST_PASSWORD:"{{relay_password}}"

Installing Prometheus Operator

Prometheus Operator is a suite of applications leveraging theOperator Pattern for managing Prometheus applications in a way that is idiomatic for Kubernetes.
Configurations, Prometheus instances, and Alertmanager instances are managed usingCustom Resource Definitions.

Before we can create the Prometheus Stack, we need to create a namespace where the prometheus resources will live

apiVersion:v1kind:Namespacemetadata:name:monitoringannotations:linkerd.io/inject:enabled

Installation of the Prometheus Operator is handled using thePrometheus Operator Helm Chart.
The Prometheus Operator Helm Chart installs the entire Prometheus Stack:

• Prometheus
• Alertmanager
• Grafana
• Prometheus-Operator
• Prometheus NodeExporter

Thevalues.yml for the chart is quite extensive and will take up a lot of space in the subsequent sections. I've broken down the values that go into the helm chart so that they can be better comprehended.
With Linkerd, additional customizations need to be made to ensure that the Linkerd dashboards work by installing the correct Prometheus rules. This can be accomplished with the followingvalues.yaml

Prometheus Operator Values

Since I'm usingLinkerd as my service mesh, and have created the monitoring namespace where I will be installing the Helm Chart using thelinkerd.io/inject annotation, I will need to set the values for the Prometheus Operator so that the webhook is skipped from linkerd injection.

# Configure Prometheus -- we need to skip Linkerd injection or# the operator will not installprometheusOperator:admissionWebhooks:patch:podAnnotations:linkerd.io/inject:disabledcertManager:enabled:"true"issuerRef:name:monitoring-issuerkind:Issuer

This configuration also utilizescert-manager to inject certificates into the Prometheus Webhook. The certificates generated by themonitoring-issuer would be signed by ourRoot CA.

Prometheus Values

The default prometheus configuration must be modified to includecustom scrapers so that Prometheus can export metrics to Linkerd. Without this configuration, theLinkerd Dashboard will not function properly.

prometheus:prometheusSpec:evaluationInterval:10sscrapeInterval:10sscrapeTimeout:10sresources:requests:memory:4GiadditionalScrapeConfigs:-job_name:'prometheus'static_configs:-targets:['localhost:9090']-job_name:'grafana'kubernetes_sd_configs:-role:podnamespaces:names:['monitoring']relabel_configs:-source_labels:-__meta_kubernetes_pod_container_nameaction:keepregex:^grafana$#  Required for: https://grafana.com/grafana/dashboards/315-job_name:'kubernetes-nodes-cadvisor'scheme:httpstls_config:ca_file:/var/run/secrets/kubernetes.io/serviceaccount/ca.crtinsecure_skip_verify:truebearer_token_file:/var/run/secrets/kubernetes.io/serviceaccount/tokenkubernetes_sd_configs:-role:noderelabel_configs:-action:labelmapregex:__meta_kubernetes_node_label_(.+)-target_label:__address__replacement:kubernetes.default.svc:443-source_labels:[__meta_kubernetes_node_name]regex:(.+)target_label:__metrics_path__replacement:/api/v1/nodes/$1/proxy/metrics/cadvisormetric_relabel_configs:-source_labels:[__name__]regex:'(container|machine)_(cpu|memory|network|fs)_(.+)'action:keep-source_labels:[__name__]regex:'container_memory_failures_total'# unneeded large metricaction:drop-job_name:'linkerd-controller'kubernetes_sd_configs:-role:podnamespaces:names:-'linkerd'-'monitoring'relabel_configs:-source_labels:-__meta_kubernetes_pod_container_port_nameaction:keepregex:admin-http-source_labels:[__meta_kubernetes_pod_container_name]action:replacetarget_label:component-job_name:'linkerd-service-mirror'kubernetes_sd_configs:-role:podrelabel_configs:-source_labels:-__meta_kubernetes_pod_label_linkerd_io_control_plane_component-__meta_kubernetes_pod_container_port_nameaction:keepregex:linkerd-service-mirror;admin-http$-source_labels:[__meta_kubernetes_pod_container_name]action:replacetarget_label:component-job_name:'linkerd-proxy'kubernetes_sd_configs:-role:podrelabel_configs:-source_labels:-__meta_kubernetes_pod_container_name-__meta_kubernetes_pod_container_port_name-__meta_kubernetes_pod_label_linkerd_io_control_plane_nsaction:keepregex:^linkerd-proxy;linkerd-admin;linkerd$-source_labels:[__meta_kubernetes_namespace]action:replacetarget_label:namespace-source_labels:[__meta_kubernetes_pod_name]action:replacetarget_label:pod# special case k8s' "job" label, to not interfere with prometheus' "job"# label# __meta_kubernetes_pod_label_linkerd_io_proxy_job=foo =># k8s_job=foo-source_labels:[__meta_kubernetes_pod_label_linkerd_io_proxy_job]action:replacetarget_label:k8s_job# drop __meta_kubernetes_pod_label_linkerd_io_proxy_job-action:labeldropregex:__meta_kubernetes_pod_label_linkerd_io_proxy_job# __meta_kubernetes_pod_label_linkerd_io_proxy_deployment=foo =># deployment=foo-action:labelmapregex:__meta_kubernetes_pod_label_linkerd_io_proxy_(.+)# drop all labels that we just made copies of in the previous labelmap-action:labeldropregex:__meta_kubernetes_pod_label_linkerd_io_proxy_(.+)# __meta_kubernetes_pod_label_linkerd_io_foo=bar =># foo=bar-action:labelmapregex:__meta_kubernetes_pod_label_linkerd_io_(.+)# Copy all pod labels to tmp labels-action:labelmapregex:__meta_kubernetes_pod_label_(.+)replacement:__tmp_pod_label_$1# Take `linkerd_io_` prefixed labels and copy them without the prefix-action:labelmapregex:__tmp_pod_label_linkerd_io_(.+)replacement:__tmp_pod_label_$1# Drop the `linkerd_io_` originals-action:labeldropregex:__tmp_pod_label_linkerd_io_(.+)# Copy tmp labels into real labels-action:labelmapregex:__tmp_pod_label_(.+)

Grafana Options

ℹ️ At the time of writing, using an external instance of Grafana
is only supported by an upcoming release of Linkerd 2.12.
Linkerd 2.12 can only be found on theedge branch, not thestable.
Subsequent sections will cover how to adjust the Linkerd install
to support an external Grafana instance in more detail.

Grafana needs to be updated to pre-install the Linkerd dashboard.
As per theMonitoring and Observability post,Loki also needs to be added as a data source for Grafana.

To supportGrafana's Alerting Functionality Alertmanager can be added as a data source so that Alertmanager rules will appear in grafana.

ℹ️ At the time of writing, Alertmanager is only supported in Grafana plugins alpha.
Grafana plugins alpha can only be added in the grafana configuration via theplugins.enable_alpha option ingrafana.ini.

# Grafana options -- pre-install Linkerd dashboards# and configure datasourcesgrafana:grafana.ini:server:root_url:'%(protocol)s://%(domain)s:/grafana/'auth:disable_login_form:falseauth.anonymous:enabled:trueorg_role:Editorauth.basic:enabled:trueanalytics:check_for_updates:falsepanels:disable_sanitize_html:truelog:mode:consolelog.console:format:textlevel:infoplugins:enable_alpha:truesmtp:enabled:truefrom_address:grafana@<domain>host:sender-mail.mail-sender:587skip_verify:truedashboardProviders:dashboardproviders.yaml:apiVersion:1providers:-name:'default'orgId:1folder:''type:filedisableDeletion:falseeditable:trueoptions:path:/var/lib/grafana/dashboards/defaultdashboards:default:# Logging dashboard - https://grafana.com/grafana/dashboards/7752logging:gnetId:7752revision:5datasource:prometheus# all these charts are hosted at https://grafana.com/grafana/dashboards/{id}top-line:gnetId:15474revision:3datasource:prometheushealth:gnetId:15486revision:2datasource:prometheuskubernetes:gnetId:15479revision:2datasource:prometheusnamespace:gnetId:15478revision:2datasource:prometheusdeployment:gnetId:15475revision:5datasource:prometheuspod:gnetId:15477revision:2datasource:prometheusservice:gnetId:15480revision:2datasource:prometheusroute:gnetId:15481revision:2datasource:prometheusauthority:gnetId:15482revision:2datasource:prometheuscronjob:gnetId:15483revision:2datasource:prometheusjob:gnetId:15487revision:2datasource:prometheusdaemonset:gnetId:15484revision:2datasource:prometheusreplicaset:gnetId:15491revision:2datasource:prometheusstatefulset:gnetId:15493revision:2datasource:prometheusreplicationcontroller:gnetId:15492revision:2datasource:prometheusprometheus:gnetId:15489revision:2datasource:prometheusprometheus-benchmark:gnetId:15490revision:2datasource:prometheusmulticluster:gnetId:15488revision:2datasource:prometheusadditionalDataSources:-name:alertmanagertype:alertmanagerurl:http://metrics-kube-prometheus-st-alertmanager:9093access:proxyorgId:1jsonData:implementation:prometheus-name:lokitype:lokiaccess:proxydefault:falseeditable:trueurl:http://loki:3100maximumLines:"300"orgId:1jsonData:manageAlerts:truealertmanagerUid:alertmanager

Configuring Alertmanager

The final piece of the puzzle is configuring Alertmanager.
Alertmanager manages Prometheus alerts and is responsible for forwarding messages to the various alert receivers. Majority of the values below were extracted from the default configuration that the Alertmanager helm chart creates.

The notable differences are the root route. By default the root route doesn't allow a matcher. If we attached a receiver to the root route, we will be constantly notified. The constant notifications would trigger spam blockers to block our Alertmanager emails.

Under the root route, we create a nestedroutes field. Theroutes field is set up so that based on thematchers criteria we will receive an alert via email. In this case our alert will trigger whenever there is an alert where its severity is critical.

alertmanager:config:global:resolve_timeout:5mroute:group_by:['job']group_wait:15sgroup_interval:5mrepeat_interval:12hreceiver:'null'routes:-group_by:['alertname']group_wait:15sgroup_interval:10srepeat_interval:12hmatchers:-severity="critical"receiver:emailreceivers:-name:'null'-name:emailemail_configs:-to:<recipient@mail.com>from:alertmanager@haus.netsmarthost:sender-mail.mail-sender:587require_tls:truetls_config:insecure_skip_verify:truetemplates:-'/etc/alertmanager/config/*.tmpl'alertmanagerSpec:externalUrl:https://monitoring.haus.net/alarmslogFormat:jsonalertmanagerConfigNamespaceSelector:matchLabels:alertmanagerconfig:enabledalertmanagerConfigSelector:matchLabels:role:alertmanager-config

Installing the chart

The chart may be installed with the following command:

helm repo add prometheus-community https://prometheus-community.github.io/helm-chartshelminstall--values values.yml--namespace monitoring metrics prometheus-community/prometheus-stack

Configuring Linkerd

Inmy previous post, I had leveraged kustomize to install Grafana with an additional data source.
This time around I'll be installing thelinkerd-viz helm chart

First I begin by installing the edge repo. The current stable branch of Linkerd does not support bringing your own Grafana instance to linkerd

source:https://linkerd.io/2.11/tasks/grafana/
These notes apply only to recent Linkerd Edge releases and the upcoming Linkerd 2.12 stable release, which have stripped off the embedded Grafana instance, recommending users to install it separately as explained below.

helm repo add linkerd-edge https://helm.linkerd.io/edge

Based on thevalues.yml from the helm chart, I can set the update the grafana settings with thegrafanaUrl parameter. I also set up Jaeger so that I can visualize application tracing.

values.yml

jaegerUrl:jaeger.linkerd-jaeger:16686prometheusUrl:http://metrics-kube-prometheus-st-prometheus.monitoring:9090grafanaUrl:metrics-grafana.monitoring:80# Since we're bringing our own Prometheus and Grafana instances,# we have to disable the embedded Prometheus and Grafana instancesprometheus:enabled:falsegrafana:enabled:false

Install the linkerd-viz helm chart with the following command:

helminstall--values values.yml--namespace linkerd-viz--create-namespace linkerd-viz linkerd-edge/linkerd-viz

Once the install completes, you will be able to access Grafana through Linkerd.

Alerts and Monitoring with Logging Operator

I use theBanzaicloud Logging Operator to provide idiomatic configurable logging from applications in my Kubernetes cluster to Grafana Loki. Logging Operator can provide Service Monitors and alarms for Logging instances with the following configuration:

apiVersion:logging.banzaicloud.io/v1beta1kind:Loggingmetadata:name:my-loggernamespace:my-namespacespec:fluentd:metrics:serviceMonitor:trueprometheusRules:truefluentbit:metrics:serviceMonitor:trueprometheusRules:true

TheserviceMonitor tag sets up a Prometheus Service monitor.
TheprometheusRules set up the Prometheus rules for alerting on certain thresholds.

The default alerting rules trigger alerts when:

• Prometheus cannot access the Fluentd node
• Fluentd buffers are quickly filling up
• Traffic to Fluentd is increasing at a high rate
• The number of Fluent Bit or Fluentd errors or retries is high
• Fluentd buffers are over 90% full

Next Steps

Once everything has been set up, we should have a foundation for receiving alarms from our Kubernetes cluster.
From this point, we could enhance the alarms by creating additional rules, such as FluentD rules with the Logging Operator.
Perhaps you would like additional Alertmanager configurations to send alarms to messaging integrations such as Slack.

In my next post, I'll demonstrate how to set up an Alertmanager configuration to work with Discord/Guilded WebHooks.

References

• Kubernetes Service Monitoring and Observability
• Dynu SMTP relay
• Bokysan SMTP Helm Chart
• Postfix Dockerhub
• Postfix Gitlab Repo
• Linkerd - Bring your own Prometheus
• Linkerd - Grafana
• Linkerd - Alerting on Error rates
• Logging Operator - Monitoring
• Logging Operator - Alerting