This product or feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of theService Specific Terms. Pre-GA products and features are available "as is" and might have limited support. For more information, see thelaunch stage descriptions.

This guide explains how you can configure the Monitoring agent torecognize and export your application metrics to Cloud Monitoring.

The Monitoring agent is acollectd daemon. In addition toexporting many predefined system and third-party metrics toCloud Monitoring, theagent can export your own collectd application metrics toMonitoringasuser-defined metrics. Your collectd plugins can alsoexport to Monitoring.

An alternative way to export application metrics to Monitoringis touseStatsD. Cloud Monitoring provides a default configuration that mapsStatsD metrics to user-defined metrics. If you are satisfied with that mapping,then you don't need the customization steps described below.For more information, see theStatsD plugin.

For more information about metrics, see the following documents:

• Metrics, time series, and resources.
• Structure of time series.
• User-defined metrics overview.

This functionality is only available for agents running on Linux. It isnot available on Windows.

Before you begin

• Install the most recent Monitoring agent on a VM instance andverify it is working. To update your agent, seeUpdating theagent.

• Configurecollectd to get monitoring data from your application.Collectd supports many application frameworks and standard monitoringendpoints through itsread plugins. Find a read plugin thatworks for you.

• (Optional) As a convenience, add the agent's collectd referencedocumentation to your system'sman pages by updating theMANPATHvariable and then runningmandb:

  exportMANPATH="$MANPATH:/opt/stackdriver/collectd/share/man"sudomandb

  The man pages are forstackdriver-collectd.

Important files and directories

The following files and directories, created by installing the agent, arerelevant to using the Monitoring agent (collectd):

/etc/stackdriver/collectd.conf

The collectd configuration file used by the agent. Edit this file tochange general configuration.

Note: The system-default collectd configuration,/etc/collectd.conf,isn't used by the Monitoring agent.

/etc/stackdriver/collectd.d/

The directory for user-added configuration files. To send user-definedmetrics from the agent, you place the required configuration files, discussedbelow, in this directory. For backward compatibility, the agent also looks forfiles in/opt/stackdriver/collectd/etc/collectd.d/.

/opt/stackdriver/collectd/share/man/*

The documentation for the agent's version of collectd. You can addthese pages to your system's set ofman pages; seeBefore you beginfor details.

/etc/init.d/stackdriver-agent

The init script for the agent.

How Monitoring handles collectd metrics

As background, the Monitoring agent processes collectd metricsand sends them to Monitoring, which treats each metric as amember of one of the following categories:

• User-defined metrics. Collectd metrics that have the metadata keystackdriver_metric_type and a singledata source are handledasuser-defined metrics and sent toMonitoring using theprojects.timeSeries.create methodin the Monitoring API.

• Curated metrics. All other collectd metrics are sent toMonitoring using an internal API. Only the metricsinthe list of curated metrics areaccepted and processed.

• Discarded metrics. Collectd metrics that aren't in the curated metricslist and aren't user-defined metrics are silently discarded byMonitoring. The agent itself isn't aware of which metricsare accepted or discarded.

Write user-defined metrics with the agent

You configure the agent to send metric data points toMonitoring. Each point must be associated with auser-defined metric, which you define with ametric descriptor.These concepts are introduced inMetrics, time series, and resourcesand described in detail atStructure of time series andUser-defined metrics overview.

You can have a collectd metric treated as a user-defined metric byadding the proper metadata to the metric:

• stackdriver_metric_type : (required) the name of the exported metric.Example:custom.googleapis.com/my_custom_metric.

• label:[LABEL] : (optional) additional labels for the exportedmetric. For example, if you want a Monitoring STRING labelnamedcolor, then your metadata key would belabel:color and the valueof the key could be"blue". You can have up to 10 labels per metric type.

You can use acollectd filter chain to modify the metadata foryour metrics. Because filter chains can't modify the list of data sources anduser-defined metrics only support a single data source, any collectd metricsthat you want to use with this facility must have a single data source.

Example

In this example we will monitor active Nginx connections from two Nginxservices,my_service_a andmy_service_b. We will send these toMonitoring using a user-defined metric.We will take the following steps:

1. Identify the collectd metrics for each Nginx service.

2. Define a Monitoring metric descriptor.

3. Configure acollectd filter chain to add metadata to thecollectd metrics, to meet the expectations of the Monitoringagent.

Incoming collectd metrics

Collectd expects metrics to consist of the following components. The first fivecomponents make up the collectdidentifier for the metric:

    Host, Plugin, Plugin-instance, Type, Type-instance, [value]

In this example, the metrics you want to send as a user-defined metrichave the following values:

Notes:
¹ In the example, this value encodes both the application (Nginx) andthe connected service name.
² The value is typically a timestamp and double-precision number.Monitoring handles the details of interpreting the various kindsof values. Compound values aren't supported by theMonitoring agent.

Monitoring metric descriptor and time series

On the Monitoring side, design a metric descriptor for youruser-defined metric. The following descriptor is a reasonable choice for thedata in this example:

• Name:custom.googleapis.com/nginx/active_connections
• Labels:
  • service_name (STRING): The name of the service connected to Nginx.
• Kind: GAUGE
• Type: DOUBLE

After you've designed the metric descriptor, you can create it by usingprojects.metricDescriptors.create,or you can let it be created for youfrom the time series metadata, discussed below. For more information,seeCreating metric descriptors on this page.

The time series data for this metric descriptor must contain the followinginformation, because of the way the metric descriptor is defined:

• Metric type:custom.googleapis.com/nginx/active_connections
• Metric label values:
  • service_name: either"my_service_a" or"my_service_b"

Other time series information, including theassociatedmonitored resource—the VMinstance sending the data—and the metric's data point, is automaticallyobtained by the agent for all metrics. You don't have to do anything special.

Your filter chain

LoadPluginmatch_regexLoadPlugintarget_setLoadPlugintarget_replace#Insertanewruleinthedefault"PreCache"chain,todivertyourmetrics.PreCacheChain"PreCache"<Chain"PreCache"><Rule"jump_to_custom_metrics_from_curl_json">#Ifthepluginnameandinstancematch,thisisPROBABLYametricwe'relookingfor:<Matchregex>Plugin"^curl_json$"PluginInstance"^nginx_"</Match><Target"jump">#Goexecutethefollowingchain;thencomeback.Chain"PreCache_curl_json"</Target></Rule>#Continueprocessingmetricsinthedefault"PreCache"chain.</Chain>#FollowingisaNEWfilterchain,justforyourmetric.#Itisonlyexecutedifthedefaultchain"jumps"here.<Chain"PreCache_curl_json">#Thefollowingruledoesalltheworkforyourmetric:<Rule"rewrite_curl_json_my_special_metric">#Doacarefulmatchforjustyourmetrics;ifitfails,dropdown#tothenextrule:<Matchregex>Plugin"^curl_json$"#Matchonplugin.PluginInstance"^nginx_my_service_.*$"#Matchonplugininstance.Type"^gauge$"#Matchontype.TypeInstance"^active-connections$"#Matchontypeinstance.</Match><Target"set">#Specifythemetricdescriptortype:MetaData"stackdriver_metric_type""custom.googleapis.com/nginx/active_connections"#Specifyavalueforthe"service_name"label;cleanitupinthenextTarget:MetaData"label:service_name""%{plugin_instance}"</Target><Target"replace">#Removethe"nginx_"prefixintheservice_nametogettherealservicename:MetaData"label:service_name""nginx_"""</Target></Rule>#Thefollowingruleisrunafterrewritingyourmetric,or#ifthemetricwasn'toneofyouruser-definedmetrics.Therulereturns#tothedefault"PreCache"chain.Thedefaultprocessing#willwriteallmetricstoCloudMonitoring,#whichwilldropanyunrecognizedmetrics:onesthataren't#inthelistofcuratedmetricsanddon'thave#theuser-definedmetricmetadata.<Rule"go_back">Target"return"</Rule></Chain>

sudo service stackdriver-agent restart

For an introduction to Cloud Monitoring metrics, seeMetrics, time series, and resources.More details are available inUser-defined metrics overviewandStructure of time series.

Metric descriptors. A metric descriptor has the following significantpieces:

• Atype of the formcustom.googleapis.com/[NAME1]/.../[NAME0]. For example:

  custom.googleapis.com/my_measurementcustom.googleapis.com/instance/network/received_packets_countcustom.googleapis.com/instance/network/sent_packets_count

  Therecommended naming is hierarchical to make the metrics easier for people tokeep track of. Metric types can't contain hyphens; for the exact namingrules, seeNaming metric types and labels.

• Up to 10 labels to annotate the metric data, such asdevice_name,fault_type, orresponse_code. The values of thelabels aren't specified in the metric descriptor.

• The kind and value type of the data points, such as "a gauge value of typedouble". For more information, seeMetricKind andValueType.

Time series. A metric data point has the following significant pieces:

• The type of the associated metric descriptor.

• Values for all of the metric descriptor's labels.

• A timestamped value consistent with the metric descriptor's value type andkind.

• Themonitored resourcethe data came from, typically a VM instance. Space for the resource is builtin, so the descriptor doesn't need a separate label for it.

Creating metric descriptors

You don't have to create a metric descriptor ahead of time.When a data point arrives in Monitoring, the point'smetric type, labels, and the point's value can be used to automaticallycreate a gauge or cumulative metric descriptor. For more information, seeAuto-creation of metric descriptors.

However, there are advantages to creating your own metric descriptor:

• You can include some thoughtful documentation for the metric and its labels.

• You can specify additional kinds and types of metrics. The only (kind,type) combinations supported by the agent are (GAUGE, DOUBLE) and(CUMULATIVE, INT64). For more information,seeMetric kinds and value types.

• You can specify label types other than STRING.

If you write a data point to Monitoring that uses a metric typethat isn't defined, then a new metric descriptor is created for the datapoint. This behavior can be a problem when you are debugging the code thatwrites metric data—misspelling the metric type results in spurious metricdescriptors.

After you create a metric descriptor, or after it is created for you, it cannotbe changed. For example, you can't add or remove labels. You can only deletethe metric descriptor—which deletes all its data—and then recreatethe descriptor the way you want.

For more details about creating metric descriptors, seeCreating your metric.

Pricing

To learn about pricing for Cloud Monitoring, see theGoogle Cloud Observability pricing page.

Limits

Cloud Monitoring has limits on the number of metric timeseries and the number of user-defined metric descriptors in each project.For details, seeQuotas and limits.

If you discover that you have created metric descriptors you no longer want, youcan find and delete the descriptors using the Monitoring API. For moreinformation, seeprojects.metricDescriptors.

Troubleshooting

This section explains how to configure the Monitoring agent'swrite_log plugin to dump out the full set of metric points, includingmetadata. This can be used to determine what points need to be transformed, aswell as to ensure your transformations behave as expected.

Dec  815:13:45test-write-logcollectd[1061]:write_logvalues:#012[{"values":[1933524992],"dstypes":["gauge"],"dsnames":["value"],"time":1481210025.252,"interval":60.000,"host":"test-write-log.c.test-write-log.internal","plugin":"df","plugin_instance":"udev","type":"df_complex","type_instance":"free"}]Dec  815:13:45test-write-logcollectd[1061]:write_logvalues:#012[{"values":[0],"dstypes":["gauge"],"dsnames":["value"],"time":1481210025.252,"interval":60.000,"host":"test-write-log.c.test-write-log.internal","plugin":"df","plugin_instance":"udev","type":"df_complex","type_instance":"reserved"}]