Repository files navigation

OpenTelemetryCollector for device monitoring to a Prometheus and Loki backend, packaged as abalenaBlock per architecture. (See block foraarch64,amd64,armv7hf.) Includes Containermetrics and device journallogs. Requires 120 MB disk space and 40 MB memory.

There are two parts to getting started: setting up the Prometheus and Loki backend integration, and configuring the collector for your fleet.

For Prometheus and Loki it's easy to create a free account atGrafana Cloud. Note theteam name created for you is used in the URL for your Grafana account, likehttps://grafana.com/orgs/{team}, and in the URL for your cloud dashboards, likehttps://{team}.grafana.net .

After registering, install the Linux Server integration (instructions) for your cloud, which will create pre-configured dashboards based on Prometheus and Loki. On the setup page,don't install the agent. Instead, at the bottom of the page, selectInstall in the sectionInstall Dashboards and Alerts (screenshot). Then you can view your dashboards although they are not receiving data yet.

Next, create a docker-compose file withotel-collector andnode-exporter service entries that reference the block, likethis example. In balenaCloud dashboard, create a fleet for your devices, andpush that service composition to your fleet to generate a release.

Finally, set fleetvariables for theotel-collector service for Prometheus username, password, and remote write URL using the values from your Grafana Cloud Prometheus service page (screenshot). The service page is available from your Grafana account site (grafana.com/orgs/{team}), not the cloud dashboard. Optionally to collect journal logs and forward to Loki, set variables for the Loki user and remote URL using the Grafana Cloud Loki service page (screenshot).

With this setup in place, you now can add devices to your fleet. They will start to send metrics data to Grafana Cloud, which you may view in the fleet overview dashboard, like below. Clicking on an item in the Instance column displays the Node Overview dashboard, like at the top of this document.

The collector also captures metrics for your application containers in balenaEngine. You can view these metrics in Grafana by importing a dashboard definition file in thedocs/dashboards directory. The screenshot below is based oncontainers.json. See the Grafanainstructions for importing a dashboard.

The collector also captures journal logs for the device. The Linux Server integration already includes a Logs dashboard, but you shouldimport the dashboard definition indocs/dashboard/node-logs.json. The screenshot below is based on this file. Notice that by selecting a line in the Logs panel, the container that generated the log line diplays.

Presently the collector block does not have a switch to enable debug logging. However, you can add this logging if you build the collector binary locally. Try the tools below and read the collector Troubleshootingdocs.

The OpenTelemetry collector can generate internal logs. Add a stanza like below to theservice section. See the collector's Internal Telemetrydocs for more.

  telemetry:    logs:      level: "debug"

To see what an exporter is sending to the backend, add this stanza to theexporters section. See theLocal exporterssection of the collector's Troubleshooting docs.

  debug:    verbosity: normal

Then add "debug" to the exporters list in the pipeline of interest, for example:

service:  pipelines:    metrics/prom:      receivers: [prometheus]      processors: [batch]      exporters: [prometheusremotewrite, debug]