Multiprocess Mode

Prometheus client libraries presume a threaded model, where metrics are sharedacross workers. This doesn’t work so well for languages such as Python whereit’s common to have processes rather than threads to handle large workloads.

To handle this the client library can be put in multiprocess mode.This comes with a number of limitations:

• Registries can not be used as normal:
  • all instantiated metrics are collected
  • Registering metrics to a registry later used by aMultiProcessCollectormay cause duplicate metrics to be exported
  • Filtering on metrics works if and only if the constructor was called withsupport_collectors_without_names=True and it but might be inefficient.
• Custom collectors do not work (e.g. cpu and memory metrics)
• Gauges cannot useset_function
• Info and Enum metrics do not work
• The pushgateway cannot be used
• Gauges cannot use thepid label
• Exemplars are not supported
• Remove and Clear of labels are currently not supported in multiprocess mode.

There’s several steps to getting this working:

1. Deployment:

ThePROMETHEUS_MULTIPROC_DIR environment variable must be set to a directorythat the client library can use for metrics. This directory must be wipedbetween process/Gunicorn runs (before startup is recommended).

This environment variable should be set from a start-up shell script,and not directly from Python (otherwise it may not propagate to child processes).

2. Metrics collector:

The application must initialize a newCollectorRegistry, and store themulti-process collector inside. It is a best practice to create this registryinside the context of a request to avoid metrics registering themselves to acollector used by aMultiProcessCollector. If a registry with metricsregistered is used by aMultiProcessCollector duplicate metrics may beexported, one for multiprocess, and one for the process serving the request.

fromprometheus_clientimportmultiprocessfromprometheus_clientimportgenerate_latest,CollectorRegistry,CONTENT_TYPE_LATEST,CounterMY_COUNTER=Counter('my_counter','Description of my counter')# Expose metrics.defapp(environ,start_response):registry=CollectorRegistry(support_collectors_without_names=True)multiprocess.MultiProcessCollector(registry)data=generate_latest(registry)status='200 OK'response_headers=[('Content-type',CONTENT_TYPE_LATEST),('Content-Length',str(len(data)))]start_response(status,response_headers)returniter([data])

3. Gunicorn configuration:

Thegunicorn configuration file needs to include the following function:

fromprometheus_clientimportmultiprocessdefchild_exit(server,worker):multiprocess.mark_process_dead(worker.pid)

4. Metrics tuning (Gauge):

WhenGauges are used in multiprocess applications,you must decide how to handle the metrics reported by each process.Gauges have several modes they can run in, which can be selected with themultiprocess_mode parameter.

• ‘all’: Default. Return a timeseries per process (alive or dead), labelled by the process’spid (the label is added internally).
• ‘min’: Return a single timeseries that is the minimum of the values of all processes (alive or dead).
• ‘max’: Return a single timeseries that is the maximum of the values of all processes (alive or dead).
• ‘sum’: Return a single timeseries that is the sum of the values of all processes (alive or dead).
• ‘mostrecent’: Return a single timeseries that is the most recent value among all processes (alive or dead).

Prepend ’live’ to the beginning of the mode to return the same result but only considering living processes(e.g., ’liveall, ’livesum’, ’livemax’, ’livemin’, ’livemostrecent’).

fromprometheus_clientimportGauge# Example gaugeIN_PROGRESS=Gauge("inprogress_requests","help",multiprocess_mode='livesum')