Repository files navigation

Nomad Events Sink is an events collection agent which usesNomad Events SDK to fetch events.

Events can help debug the cluster state and alert operators about new deployments, failing allocations, node updates etc. Nomad emits these events in a channel and provides an SDK to access these events.

The missing piece was a tool to ingest these events and analyze/visualize them later. This is wherenomad-events-sink agent comes into picture and helps to ingest these events in multiple configurableSinks.

The tool is designed to be generic in order to handle many use-cases of processing events. MultipleSink Providers can be created and events will be handled by them. Common usecases included storing events for long term use (Vector->Loki), alerting on cluster state changes (Slack/Rocketchat webhooks) etc.

• A subscription for each topic is created and new events are watched.
• Whenever a new event comes, it gets added to aSink channel.
• Multiple background workers (Goroutines) are listening to this channel, processing events in FIFO.
• These workers batch events and push to upstream providers (for eg HTTP/S3 etc).
• A background worker commits the event index state to a file on disk. This is done so that the program is able to pickup events from where it left before shutting down.

This program uses 2 batch strategies in order to avoid an inefficient process of making a request for every incoming event:

• idle_timeout: If a batch is in memory for more thanidle_timeout duration, it is flushed to providers.
• events_count: If a batch has more events thanevents_counts, it is flushed to providers.

You can choose one of the various deployment options:

To deploy on a Nomad cluster, you can refer tothis jobspec.

Grab the latest release fromReleases.

To run:

$ ./nomad-events-sink.bin --config config.toml

docker pull ghcr.io/mr-karan/nomad-events-sink:latest

Refer tothis example for quickly getting started with ingesting Deployment Events to Loki using Vector.

Refer toconfig.sample.toml for a list of configurable values.

All config variables can also be populated as env vairables by prefixingNOMAD_EVENTS_SINK_ and replacing. with__.

For eg:app.data_dir becomesNOMAD_EVENTS_SINK_app__data_dir.

Nomad API client reads the following environment variables:

• NOMAD_TOKEN
• NOMAD_ADDR
• NOMAD_REGION
• NOMAD_NAMESPACE
• NOMAD_HTTP_AUTH
• NOMAD_CACERT
• NOMAD_CAPATH
• NOMAD_CLIENT_CERT
• NOMAD_CLIENT_KEY
• NOMAD_TLS_SERVER_NAME
• NOMAD_SKIP_VERIFY

You can read about them in detailhere.

• Currently onlyHTTP Provider is implemented. There's a fullworking example of ingesting events to the HTTP server provided by Vector and dumping them to Loki.

Please feel free to open a new issue for bugs, new features (like Sink Providers), feedback etc.

LICENSE