Repository files navigation

Alert dashboard forPrometheus Alertmanager.

Alertmanager>=0.22.0 is required.

SeeGitHub Releases for releasechangelog.

Alertmanager UI is useful for browsing alerts and managing silences, but it'slacking as a dashboard tool - karma aims to fill this gap.

Starting with the0.7.0 release it can aggregate alerts from multipleAlertmanager instances, running either in HA mode or separate. Unique alerts aredisplayed by filtering duplicates. Each alert is tagged with the names of allAlertmanager instances it was found at and can be filtered based on those tags(@alertmanager). Note that@alertmanager tags will be visible only if karmais configured with multiple Alertmanager instances.If alertmanger is configured to useHA clustersthen@cluster will be available as well, to set a custom name for each clusterseeCONFIGURATION.md.

Alerts are displayed grouped preservinggroup_byconfiguration option in Alertmanager.Note that a unique alert group will be created for each receiver it uses inalertmanager as they can have differentgroup_by settings.If a group contains multiple alerts onlythe first few alerts will be presented. Alerts are expanded or hiddenusing - / + buttons. The default number of alerts can be configured in the UIsettings module.Each group can be collapsed to only show the title bar using top right toggleicon.Each individual alert will show unique labels and annotations. Labelsand annotations that are shared between all alerts are moved to the footer.

Active alerts will show recently expired silences, to allow re-silincingif needed.This is controlled viasilences:expired setting.10m value would showsilences expired in the last 10 minutes but only for alerts that started firingmore than 10 minutes ago.

Alertmanager doesn't currently provide any long term storage of alert eventsor a way to query for historical alerts, but each Prometheus server sendingalerts stores metrics related to triggered alerts.Whenhistory:enabled istrue karma will usesource fields from each alertto try querying alert related metrics on remote Prometheus servers.The result is the number of times given alert group triggered an alert per hourin the last 24h, displayed as 24 blocks. The darker the color the more alertswere triggered in that hour, as compared by all other hours.

For this feature to work karma must be able to connect to all Prometheus serverssending alerts. Be sure to set--web.external-url Prometheus flag to a publiclyreachable URL of each server.

Inhibited alerts (suppressed by other alerts,see Alertmanager docs)will have a "muted" button.

Clicking on that button will bring a modal with a list of inhibiting alerts.

If all alerts in a group were suppressed by the same silence then, to savescreen space, the silence will also be moved to the footer.

To help separate alerts from different environments or with different level ofseverity multi-grid mode can be enabled, which adds another layer of visuallygrouping alert groups.To enable this mode go to the configuration modal and select a label name, allalerts will be grouped by that label, each label value will have a dedicatedgrid, including an extra grid for alerts without that label present.

Silence modal allows to create new silences and manage all silences alreadypresent in Alertmanager.Silence ACL rules can be used to control silence creation and editing, seeACLs docs for more details.

Clicking on the alert counter in the top left corner will open the overviewmodal, which allows to quickly get an overview of the top label values forall current alerts.

Starting withv0.50 karma can create short lived silences to acknowledgealerts with a single button click. To create silences that will resolve itselfonly after all alerts are resolved you can usekthxbye.Seeconfiguration docs fordetails.

Starting withv0.78 karma can be configured to check forDead Man’s Switchstyle alerts (alert that is always firing). If no alert is found in givenalertmanager karma will show an error in the UI.Seehealthcheck:filters option onconfiguration docsfor details.

Starting withv0.52 release karma includes both light and dark themes.By default it will follow browser preference usingprefers-color-schememedia queries.

Online demo is running latest main branch.It might include features that are experimental and not yet ready to be included.

Release notes can be found onGitHub Release Page.

To get notifications about new karma releases go toGitHub karma page, clickWatch andselectReleases only. This requires GitHub user account.To subscribe to email notifications without GitHub account you can subscribe tothe RSS feed thatGitHub provides.To get email notifications from those feeds use one of the free servicesprovidingRSS to email notifications, likeBlogtrottr.

I created karma while working forCloudflare,originally it was calledunsee.This project is based on that code but the UI part was rewritten from scratchusingReact. New UI required changes to the backend sothe API is also incompatible.Given that the React rewrite resulted in roughly 50% of new code and to avoidconfusion for user I've decided to rename it to karma, especially that theoriginal project wasn't being maintained anymore.

Alertmanager's API isn't stable yet and can change between releases, seeVERSIONS ininternal/mock/Makefile for list of allAlertmanager releases that are tested and supported by karma.Due to API differences between those releases some features will workdifferently or be missing, it's recommended to use the latest supportedAlertmanager version.

karma doesn't in any way alter alerts in any Alertmanager instance it collectsdata from. This is true for both the backend and the web UI.The web UI allows to manage silences by sending requests to Alertmanagerinstances, this can be done directly (browser to Alertmanager API) or byproxying such requests via karma backend (browser to karma backend toAlertmanager API) ifproxy mode is enabled in karma config.

If you wish to deploy karma as a read-only tool without giving users any abilityto modify data in Alertmanager instance, then please ensure that:

• the karma process is able to connect to the Alertmanager API
• read-only users are able to connect to the karma web interface
• read-only users are NOT able to connect to the Alertmanager API
• readonly is set totrue inalertmanager:servers config sectionfor all alertmanager instances, this options will disable any UI elements thatcould trigger updates (like silence management)

To restrict some users from creating silences or enforce some matcher rules usesilence ACL rules. This feature requiresproxy to be enabled.

karma process metrics are accessible under/metrics path by default.If you set the--listen.prefix option a path relative to it will beused.

To clone git repo and build the binary yourself run:

git clone https://github.com/prymitive/karma $GOPATH/src/github.com/prymitive/karmacd $GOPATH/src/github.com/prymitive/karma

To finally compilekarma the binary run:

make

Note that building locally from sources requires Go, nodejs and yarn.See Docker build options below for instructions on building from withing dockercontainer.

karma can be configured using config file, command line flags or environmentvariables. Config file is the recommended method, it's also the only way toconfigure karma to use multiple Alertmanager servers for collecting alerts.To run karma with a single Alertmanager server setALERTMANAGER_URIenvironment variable or pass--alertmanger.uri flag on the command line, withAlertmanager URI as argument, example:

ALERTMANAGER_URI=https://alertmanager.example.com karmakarma --alertmanager.uri https://alertmanager.example.com

There is a make target which will compile and run a demo karma docker image:

make run-demo

By default it will listen on port8080 and will have mock alerts.

Official docker images are built and hosted onGithub.

Images are built automatically for:

• release tags in git -ghcr.io/prymitive/karma:vX.Y.Z
• main branch commits -ghcr.io/prymitive/karma:latest

NOTE karma usesuber-go/automaxprocsto automatically adjustGOMAXPROCS to match Linux container CPU quota.

To start a release image run:

docker run -e ALERTMANAGER_URI=https://alertmanager.example.com ghcr.io/prymitive/karma:vX.Y.Z

Latest release details can be found onGitHub.

To start docker image build from lastet main branch run:

docker run -e ALERTMANAGER_URI=https://alertmanager.example.com ghcr.io/prymitive/karma:latest

Note that latest main branch might have bugs or breaking changes. Usingrelease images is strongly recommended for any production use.

make docker-image

This will build a Docker image locally from sources.

/health endpoint can be used for health check probes, it always responds with200 OK code andPong response body.

Please seeCONFIGURATION for full list of availableconfiguration options andexample.yaml for a config fileexample.

Please seeCONTRIBUTING for details.

Apache License 2.0, please seeLICENSE.