Repository files navigation

Prometheus exporter for hardware and OS metrics exposed by *NIX kernels, writtenin Go with pluggable metric collectors.

TheWindows exporter is recommended for Windows users.To expose NVIDIA GPU metrics,prometheus-dcgmcan be used.

If you are new to Prometheus andnode_exporter there is asimple step-by-step guide.

Thenode_exporter listens on HTTP port 9100 by default. See the--help output for more options.

For automated installs withAnsible, there is thePrometheus Community role.

Thenode_exporter is designed to monitor the host system. Deploying in containers requiresextra care in order to avoid monitoring the container itself.

For situations where containerized deployment is needed, some extra flags must be used to allowthenode_exporter access to the host namespaces.

Be aware that any non-root mount points you want to monitor will need to be bind-mountedinto the container.

If you start container for host monitoring, specifypath.rootfs argument.This argument must match path in bind-mount of host root. The node_exporter will usepath.rootfs as prefix to access host filesystem.

docker run -d \  --net="host" \  --pid="host" \  -v"/:/host:ro,rslave" \  quay.io/prometheus/node-exporter:latest \  --path.rootfs=/host

For Docker compose, similar flag changes are needed.

---version:'3.8'services:node_exporter:image:quay.io/prometheus/node-exporter:latestcontainer_name:node_exportercommand:      -'--path.rootfs=/host'network_mode:hostpid:hostrestart:unless-stoppedvolumes:      -'/:/host:ro,rslave'

On some systems, thetimex collector requires an additional Docker flag,--cap-add=SYS_TIME, in order to access the required syscalls.

There is varying support for collectors on each operating system. The tablesbelow list all existing collectors and the supported systems.

Collectors are enabled by providing a--collector.<name> flag.Collectors that are enabled by default can be disabled by providing a--no-collector.<name> flag.To enable only some specific collector(s), use--collector.disable-defaults --collector.<name> ....

A few collectors can be configured to include or exclude certain patterns using dedicated flags. The exclude flags are used to indicate "all except", while the include flags are used to say "none except". Note that these flags are mutually exclusive on collectors that support both.

Example:

--collector.filesystem.mount-points-exclude=^/(dev|proc|sys|var/lib/docker/.+|var/lib/kubelet/.+)($|/)

List:

node_exporter also implements a number of collectors that are disabled by default. Reasons for this vary bycollector, and may include:

• High cardinality
• Prolonged runtime that exceeds the Prometheusscrape_interval orscrape_timeout
• Significant resource demands on the host

You can enable additional collectors as desired by adding them to yourinit system's or service supervisor's startup configuration fornode_exporter but caution is advised. Enable at most one at a time,testing first on a non-production system, then by hand on a singleproduction node. When enabling additional collectors, you shouldcarefully monitor the change by observing the scrape_duration_seconds metric to ensure that collection completesand does not time out. In addition, monitor thescrape_samples_post_metric_relabeling metric to see the changes incardinality.

These collectors are deprecated and will be removed in the next major release.

Theperf collector may not work out of the box on some Linux systems due to kernelconfiguration and security settings. To allow access, set the followingsysctlparameter:

sysctl -w kernel.perf_event_paranoid=X

• 2 allow only user-space measurements (default since Linux 4.6).
• 1 allow both kernel and user measurements (default before Linux 4.6).
• 0 allow access to CPU-specific data but not raw tracepoint samples.
• -1 no restrictions.

Depending on the configured value different metrics will be available, for mostcases0 will provide the most complete set. For more information seeman 2 perf_event_open.

By default, theperf collector will only collect metrics of the CPUs thatnode_exporter is running on (ieruntime.NumCPU. If this isinsufficient (e.g. if you runnode_exporter with its CPU affinity set tospecific CPUs), you can specify a list of alternate CPUs by using the--collector.perf.cpus flag. For example, to collect metrics on CPUs 2-6, youwould specify:--collector.perf --collector.perf.cpus=2-6. The CPUconfiguration is zero indexed and can also take a stride value; e.g.--collector.perf --collector.perf.cpus=1-10:5 would collect on CPUs1, 5, and 10.

Theperf collector is also able to collecttracepointcounts when using the--collector.perf.tracepoint flag. Tracepoints can befound usingperf list orfrom debugfs. And example usage of this would be--collector.perf.tracepoint="sched:sched_process_exec".

Thesysctl collector can be enabled with--collector.sysctl. It supports exposing numeric sysctl valuesas metrics using the--collector.sysctl.include flag and string values as info metrics by using the--collector.sysctl.include-info flag. The flags can be repeated. For sysctl with multiple numeric values,an optional mapping can be given to expose each value as its own metric. Otherwise anindex label is usedto identify the different fields.

Using--collector.sysctl.include=vm.user_reserve_kbytes:vm.user_reserve_kbytes = 131072 ->node_sysctl_vm_user_reserve_kbytes 131072

A sysctl can contain multiple values, for example:

net.ipv4.tcp_rmem = 40961310726291456

Using--collector.sysctl.include=net.ipv4.tcp_rmem the collector will expose:

node_sysctl_net_ipv4_tcp_rmem{index="0"} 4096node_sysctl_net_ipv4_tcp_rmem{index="1"} 131072node_sysctl_net_ipv4_tcp_rmem{index="2"} 6291456

If the indexes have defined meaning like in this case, the values can be mapped to multiple metrics by appending the mapping to the --collector.sysctl.include flag:Using--collector.sysctl.include=net.ipv4.tcp_rmem:min,default,max the collector will expose:

node_sysctl_net_ipv4_tcp_rmem_min 4096node_sysctl_net_ipv4_tcp_rmem_default 131072node_sysctl_net_ipv4_tcp_rmem_max 6291456

String values need to be exposed as info metric. The user selects them by using the--collector.sysctl.include-info flag.

kernel.core_pattern = core ->node_sysctl_info{key="kernel.core_pattern_info", value="core"} 1

Given the following sysctl:

kernel.seccomp.actions_avail = kill_process kill_thread trap errno trace log allow

Setting--collector.sysctl.include-info=kernel.seccomp.actions_avail will yield:

node_sysctl_info{key="kernel.seccomp.actions_avail", index="0", value="kill_process"} 1node_sysctl_info{key="kernel.seccomp.actions_avail", index="1", value="kill_thread"} 1...

Thetextfile collector is similar to thePushgateway,in that it allows exporting of statistics from batch jobs. It can also be usedto export static metrics, such as what role a machine has. The Pushgatewayshould be used for service-level metrics. Thetextfile module is for metricsthat are tied to a machine.

To use it, set the--collector.textfile.directory flag on thenode_exporter commandline. Thecollector will parse all files in that directory matching the glob*.promusing thetextformat.Note: Timestamps are not supported.

To atomically push completion time for a cron job:

echo my_batch_job_completion_time $(date +%s) > /path/to/directory/my_batch_job.prom.$$mv /path/to/directory/my_batch_job.prom.$$ /path/to/directory/my_batch_job.prom

To statically set roles for a machine using labels:

echo 'role{role="application_server"} 1' > /path/to/directory/role.prom.$$mv /path/to/directory/role.prom.$$ /path/to/directory/role.prom

Thenode_exporter will expose all metrics from enabled collectors by default. This is the recommended way to collect metrics to avoid errors when comparing metrics of different families.

For advanced use thenode_exporter can be passed an optional list of collectors to filter metrics. The parameterscollect[] andexclude[] can be used multiple times (but cannot be combined). In Prometheus configuration you can use this syntax under thescrape config.

Collect onlycpu andmeminfo collector metrics:

  params:    collect[]:      - cpu      - meminfo

Collect all enabled collector metrics but excludenetdev:

  params:    exclude[]:      - netdev

This can be useful for having different Prometheus servers collect specific metrics from nodes.

Prerequisites:

• Go compiler
• RHEL/CentOS:glibc-static package.

Building:

git clone https://github.com/prometheus/node_exporter.gitcd node_exportermake build./node_exporter <flags>

To see all available configuration flags:

./node_exporter -h

make test

EXPERIMENTAL

The exporter supports TLS via a new web configuration file.

./node_exporter --web.config.file=web-config.yml

See theexporter-toolkit web-configuration for more details.