Repository files navigation

• Table of Contents
  • Overview
  • Setup
    • Installation
    • Configuration
      • Parameters
        • output_dir
        • collection_frequency
        • retention_days
        • Metrics Server Parameters
        • metrics_server_type
        • metrics_server_hostname
        • metrics_server_port
        • metrics_server_db_name
        • override_metrics_command
  • Usage
    • Searching Metrics
      • Searching Puppetserver Metrics
      • Searching PuppetDB Metrics
    • Sharing Metrics Data
  • Reference
    • Directory Layout
    • Systemd Timers
  • Alternate Setup
    • Temporary Installation
    • Manual Configuration of Hosts
      • Monolithic Infrastructure with Compilers
        • Hiera Data Example
        • Class Declaration Example
    • Configuration for Distributed Metrics Collection
  • How to Report an issue or contribute to the module
• Supporting Content
  • Articles
  • Videos

This module collects metrics provided by the status endpoints of Puppet Enterprise services.The metrics can be used to identify performance issues that may be addressed by performance tuning.

For PE versions older than 2019.8.5, access to the/metrics/v2 API endpoint is restricted tolocalhost as a mitigation forCVE-2020-7943. This module requires access the/metrics/v2 API to collect a complete set of performance metrics from PuppetDB. Refer toConfiguration for Distributed Metrics Collection for a workaround.

Install this module withpuppet module install puppetlabs-puppet_metrics_collector or add it to your Puppetfile.

To activate this module, classify your Primary Server with thepuppet_metrics_collector class using your preferred classification method.Below is an example usingsite.pp.

node'primary.example.com' {include puppet_metrics_collector}

Optionally, you can gather basic system metrics.Unlike service metrics, system metrics have to be enabled locally on each PE Infrastructure Host, and the resulting data will be stored locally on that host.This functionality depends onsysstat.

node'primary.example.com' {include puppet_metrics_collectorinclude puppet_metrics_collector::system}node'compilerA.example.com','compilerB.example.com,' {include puppet_metrics_collector::system}

Note: Do notinclude the top-levelpuppet_metrics_collector class on any PE Infrastructure Host other than the Primary Server, otherwise it will collect the same data as the Primary Server.

This module automatically configures the hosts it collects metrics from by querying PuppetDB for PE Infrastructure Hosts.If there is an error with the automatic configuration of hosts, refer toManual Configuration of Hosts.

For each Puppet Enterprise service (Puppet Server, PuppetDB, Orchestrator, Ace, Bolt, and ActiveMQ) there are associated<service_name>_ensure,<service_name>_hosts, and<service_name>_port parameters.Refer tomanifests/init.pp for details.

String: Output directory for collected metrics.

Defaults to/opt/puppetlabs/puppet-metrics-collector.

Integer: How often to collect metrics, in minutes.

Defaults to5.

Integer: How long to retain collect metrics, in days.

Defaults to90.

The following set of parameters begining withmetrics_server_ allows for the specification of a server type to use to generate and (in some cases) send data to a specified metrics server.Currently, bothinfluxdb andgraphite types allow for the transfer of data whilesplunk_hec only generates data.

OptionalEnum['influxdb','graphite','splunk_hec']: The metrics server type to send data to.

Currently, this module supportsinfluxdb,graphite, andsplunk_hec metrics server types.

For theinfluxdb metrics server type, ametrics_server_db_name must be provided.

For thesplunk_hec metrics server type, data cannot be sent to a server, however the command will format the JSON output using thesplunk_hec module, which is a requirement for this option. Thesplunk_hec module can be found on theForge orGitHub.Setup instructions for thesplunk_hec module can be found within that module's README.

OptionalString: The hostname of the metrics server to send data to.

Defaults toundef.

OptionalInteger: The port number of the metrics server to send data to.

Defaults toundef.

OptionalString: The database name on the metrics server to send data to.

Required formetrics_server_type ofinfluxdb.

Defaults toundef.

OptionalString: Allows you to define the command that is executed to gather metrics.

Defaults toundef.

Metrics are formatted as a JSON hash on one line.In order to convert the metric files into a multi-line format, they can be processed withpython -m json.tool as per below.

cd /opt/puppetlabs/puppet-metrics-collectorforiin<service_name>/primary.example.com/*.json;doecho"$(python -m json.tool<$i)">$i;done

You can search for useful information by performing agrep, run from inside the directory containing the metrics

cd /opt/puppetlabs/puppet-metrics-collectorgrep -oP'"<metric_name>,*?,'<service_name>/primary.example.com/*.json

or JQ if available

cd /opt/puppetlabs/puppet-metrics-collectorjq'.. |."<metric_name>"? | select(. != null)| input_filename , .' --<service_name>/primary.example.com/*.json

Since the metrics are archived once per day, you can only search metrics for the current day.To search older metrics, decompress the archived files into a subdirectory of/tmp and run your search from inside that directory.

Example:

grep -oP'"average-free-jrubies.*?,' puppetserver/primary.example.com/*.jsonpuppetserver/primary.example.com/20190404T170501Z.json:"average-free-jrubies":0.9950009285369501,puppetserver/primary.example.com/20190404T171001Z.json:"average-free-jrubies":0.9999444653324225,puppetserver/primary.example.com/20190404T171502Z.json:"average-free-jrubies":0.9999993830655706,

jq'.. |."average-free-jrubies"? | select(. != null)| input_filename , .' -- puppetserver/primary.example.com/*.json"puppetserver/primary.example.com/20190404T170501Z.json"0.9950009285369501"puppetserver/primary.example.com/20190404T171001Z.json"0.9999444653324225,"puppetserver/primary.example.com/20190404T171502Z.json"0.9999993830655706,

Example:

grep -oP'"queue_depth.*?,' puppetdb/primary.example.com/*.jsonpuppetdb/primary.example.com/20190404T170501Z.json:"queue_depth": 0,puppetdb/primary.example.com/20190404T171001Z.json:"queue_depth": 0,puppetdb/primary.example.com/20190404T171502Z.json:"queue_depth": 0,

jq'.. |."queue_depth "? | select(. != null)| input_filename , .' -- puppetdb/primary.example.com/*.json"puppetdb/primary.example.com/20190404T170501Z.json"0"puppetdb/primary.example.com/20190404T171001Z.json"0"puppetdb/primary.example.com/20190404T171502Z.json" 0

When working with Support, you may be asked for an archive of collected metrics data.

This module provides a script,create-metrics-archive to archive metrics data for sending to Support.

/opt/puppetlabs/puppet-metrics-collector/scripts/create-metrics-archive

This script creates the archive in the current working directory.

It takes an optional-m or--metrics-directory parameter (default/opt/puppetlabs/puppet-metrics-collector) to specify an alterate metrics directory to archive.

It takes an optional-r or--retention-days parameter (default:30) to limit the number of days to include in the archive.

[root@primary~]# /opt/puppetlabs/puppet-metrics-collector/scripts/create-metrics-archiveCreated metrics archive: /root/puppet-metrics-collector-20200203T123456Z.tar.gz

This module creates an output directory with one subdirectory for each Puppet Enterprise service (Puppet Server, PuppetDB, Orchestrator, Ace, Bolt, and ActiveMQ) that this module has been configured to collect.Each service directory has one subdirectory for each host.Each host directory contains one JSON file, collected every 5 minutes.Once per day, the metrics for each service are archived and compressed.

Example:

/opt/puppetlabs/puppet-metrics-collector/puppetserver├── primary.example.com│   ├── 20190404T020001Z.json│   ├── ...│   ├── 20190404T170501Z.json│   └── 20190404T171001Z.json└── puppetserver-2019.04.04.02.00.01.tar.gz/opt/puppetlabs/puppet-metrics-collector/puppetdb└── primary.example.com│   ├── 20190404T020001Z.json│   ├── ...│   ├── 20190404T170501Z.json│   ├── 20190404T171001Z.json└── puppetdb-2019.04.04.02.00.01.tar.gz

This module creates two systemd timers for each Puppet Enterprise service:

• One to collect the metrics
  • Runs as percollection_frequency
• One to archive collected metrics and delete metrics older than the retention period, as perretention_days
  • Runs at randomly selected time between 12:00 AM and 3:00 AM

While a permanent installation is recommended, this module can be temporarily installed with the following commands.

puppet module install puppetlabs-puppet_metrics_collector --modulepath /tmp;puppet apply -e"class { 'puppet_metrics_collector': }" --modulepath /tmp;

If necessary, you can manually configure this module by specifying parameters via the class declaration or via Hiera data.The preferred method is via Hiera data.The following examples show you how to specify those parameters for different infrastructures, and assumes you declare this module on the Primary Server.

puppet_metrics_collector::puppetserver_hosts: -'primary.example.com' -'compiler-1.example.com' -'compiler-2.example.com'puppet_metrics_collector::puppetdb_hosts: -'primary.example.com'

class {'puppet_metrics_collector':puppetserver_hosts=> ['primary.example.com','compiler-1.example.com','compiler-2.example.com'  ],puppetdb_hosts=> ['primary.example.com'],}

This option collect metrics on each PE Infrastructure Host instead of collecting metrics centrally on the Primary Server.This option is discouraged, but allows for the collection of metrics when the Primary Server cannot access the API endpoints of the other PE Infrastructure Hosts.Classify each PE Infrastructure Host with this module, specifying the following parameters.

When classifying a Compiler, specify these additional parameters:

class {'puppet_metrics_collector':puppetserver_hosts=> ['127.0.0.1'],puppetdb_metrics_ensure=>absent,orchestrator_metrics_ensure=>absent,ace_metrics_ensure=>absent,bolt_metrics_ensure=>absent,}

When classifying a PuppetDB Host, specify these additional parameters:

class {'puppet_metrics_collector':puppetdb_hosts=> ['127.0.0.1'],puppetserver_metrics_ensure=>absent,orchestrator_metrics_ensure=>absent,ace_metrics_ensure=>absent,bolt_metrics_ensure=>absent,}

If you are a PE user and need support using this module or are encountering issues, our Support team would be happy to help you resolve your issue and help reproduce any bugs. Just raise a ticket on thesupport portal.If you have a reproducible bug or are a community user you can raise it directly on the Github issues page of the modulehere. We also welcome PR contributions to improve the module. Please see further details about contributinghere.

TheSupport Knowledge base is a searchable repository for technical information and how-to guides for all Puppet products.

This Module has the following specific Article(s) available:

1. Troubleshoot and fix performance issues with the puppetlabs-puppet_metrics_collector module in Puppet Enterprise
2. Manage the installation and configuration of metrics dashboards using the puppetlabs-puppet_metrics_dashboard module for Puppet Enterprise 2016.4 to 2019.1
3. Troubleshooting potential issues in Puppet Enterprise: How to learn more

TheSupport Video Playlist is a resource of content generated by the support team

This Module has the following specific video content available:

1. Puppet Metrics Overview

2. Collecting and Displaying Puppet Metrics

3. Interpreting Puppet Metrics

   ---