Repository files navigation

Notification Manager manages notifications in multi-tenant K8s environment. It receives alerts, cloud event, and others (such as auditing, k8s events)from different senders and then send notifications to various tenant receivers based on tenant label likenamespace oruser.

Supported senders includes:

• Prometheus Alertmanager
• Custom sender
• Prometheus (Coming soon)
• Cloud Event (Coming soon)

Supported receivers includes:

• DingTalk
• Email
• Feishu
• Pushover
• SMS (Short Message Service)
• Slack
• Webhook
• WeChat
• Discord

Notification Manager usesReceiver andConfig CRDs to store notification configslike email, WeChat and slack. It also includes an operator to create and reconcileNotificationManagerCRD which watches allreceivers andconfigs, updates notification settings accordingly and sends notifications to users.

The incoming data (alert, cloud event and others) will cache in the cache firstly, then goes through steps such assilence, inhibit (coming soon),route,filter,aggregation, etc. Notifications will generate from data usingtemplate,then send to receivers andhistory webhook (if set).

Silence is a straightforward way to simply mute notifications for a given time. It usesSilence CRD to definethe silence policy. If incoming data matches an active silence, no notifications will be sent out for that data.

Route find all receivers the notifications will send to.There are two ways to determine which receivers the notifications will send to, one is viaRouter CRD,and the other is auto-matching via thenamespace label in the notification.

Usually the incoming data contains anamespace label, Notification Manager uses this label to decide which receiver to use for sending notifications:

• For KubeSphere, Notification Manager will try to find tenants with the right to access the namespace fromsidecarand then find receivers withuser = xxx label.
• For Kubernetes, Notification Manager will try to find receivers withnamespace = xxx label.

For data without anamespace label, for example alerts of node or kubelet, user can set up a receiver withtype = global label to receive notifications without anamespace label. A global receiver sends notifications for all notifications received regardless any label. A global receiver usually set for an admin role.

How the two methods work together is determined by theroutePolicy.

Filter filters the notifications sent to receivers. There are two ways to filter notifications. One is usingalertSelector in the receiver,the other is usingtenant silence.

Aggregation groups notifications bygroupLabels. Notifications in the same group will send together.

History is a webhook used to collect all notifications sent to receivers, it can be set viahistory.

We assume you already have a Kubernetes cluster (v1.16+). You can install one usingKubeKey if you haven't.

# Deploy CRDs and the Notification Manager Operator:kubectl apply -f https://github.com/kubesphere/notification-manager/releases/download/v2.5.2/bundle.yaml# Deploy default template:kubectl apply -f https://github.com/kubesphere/notification-manager/releases/download/v2.5.2/template.yaml# Deploy built-in language packs.kubectl apply -f https://github.com/kubesphere/notification-manager/releases/download/v2.5.2/zh-cn.yaml

helm install notification-manager --create-namespace -n kubesphere-monitoring-system https://github.com/kubesphere/notification-manager/releases/download/v2.5.2/notification-manager.tgz

NotificationManager CRD Defines the desired notification manager deployment. The Notification Manager Operatorensures a deployment meeting the resource requirements is running.

We should create a NotificationManager CR first, skip this when using helm install.

kubectl apply -f https://github.com/kubesphere/notification-manager/releases/download/v2.5.2/notification_manager.yaml

Notification Manager uses port19093 and API path/api/v2/alerts to receive alerts sent from Alertmanager.

To receive Alertmanager alerts, add webhook config like below to thereceivers section of Alertmanager configuration file:

"receivers":     -"name":"notification-manager""webhook_configs":       -"url":"http://notification-manager-svc.kubesphere-monitoring-system.svc:19093/api/v2/alerts"

Below is the data structure passed to the notification manager, please refer toData for more details.

{"alerts":[{"status":"firing","labels":{"alertname":"KubePodCrashLooping","container":"busybox-3jb7u6","instance":"10.233.71.230:8080","job":"kube-state-metrics","namespace":"pp1","pod":"dd1-0","prometheus":"kubesphere-monitoring-system/k8s","severity":"critical"},"annotations":{"message":"Pod pp1/dd1-0 (busybox-3jb7u6) is restarting 1.07 times / 5 minutes.",},"startsAt":"2020-02-26T07:05:04.989876849Z","endsAt":"0001-01-01T00:00:00Z",}],}

A custom senders can send notifications using notification manager simply by sending data tohttp://notification-manager-svc.kubesphere-monitoring-system.svc:19093/api/v2/alerts.

Now it's time to create the receiver and config to receive notifications, you can find guides to create them inreceiver andconfig.

To customize the notification template, please refer totemplate.

# Build notification-manager-operator and notification-manager docker imagesmake build # Push built docker images to docker registrymake push

• API documentation.