Prometheus provides a functional query language called PromQL (Prometheus QueryLanguage) that lets the user select and aggregate time series data in realtime.

When you send a query request to Prometheus, it can be aninstant query, evaluated at one point in time,or arange query at equally-spaced steps between a start and an end time. PromQL works exactly the samein each case; the range query is just like an instant query run multiple times at different timestamps.

In the Prometheus UI, the "Table" tab is for instant queries and the "Graph" tab is for range queries.

Other programs can fetch the result of a PromQL expression via theHTTP API.

Examples

This document is a Prometheus basic language reference. For learning, it may be easier tostart with a couple ofexamples.

Samples

The value of a sample at a given timestamp returned by PromQL may be a float oranative histogram. Afloat sample is a simple floating point number, whereas a native histogramssample contains a full histogram including count, sum, and buckets.

Note that the term “histogram sample” in the PromQL documentation always refersto a native histogram. The term "classic histogram" refers to a set of timeseries containing float samples with the_bucket,_count, and_sumsuffixes that together describe a histogram. From the perspective of PromQL,these contain just float samples, there are no “classic histogram samples”.

Both float samples and histogram samples can have a counter or a gauge “flavor”.Float samples with a counter or gauge flavor are generally simply called“counters” or “gauges”, respectively, while their histogram counterparts arecalled “counter histograms” or “gauge histograms”. Float samples do not storetheir flavor, leaving it to the user to take their flavor into account whenwriting PromQL queries. (By convention, time series containing float countershave a name ending on_total to help with the distinction.)

Since histogram samples “know” their counter or gauge flavor, this allowsreliable warnings about mismatched operations. For example, applying theratefunction to gauge floats will most likely produce anonsensical result, but the query will be processed without complains. However,if applied to gauge histograms, the result of the query will beannotated with a warning.

Expression language data types

In Prometheus's expression language, an expression or sub-expression canevaluate to one of four types:

• Instant vector - a set of time series containing a single sample for each time series, all sharing the same timestamp
• Range vector - a set of time series containing a range of data points over time for each time series
• Scalar - a simple numeric floating point value
• String - a simple string value; currently unused

Depending on the use case (e.g. when graphing vs. displaying the output of anexpression), only some of these types are legal as the result of auser-specified expression.Forinstant queries, any of the above data types are allowed as the root of the expression.Range queries only support scalar-typed and instant-vector-typed expressions.

Both vectors and time series may contain a mix of float samples and histogramsamples.

Reconciliation of histogram bucket layouts

Native histograms can have different bucket layouts, but they are generallyconvertible to compatible versions to apply binary and aggregation operationsto them. Functions acting on range vectors that are applicable to nativehistograms also perform such reconciliation. In binary operations thisreconciliation is performed pairwise, in aggregation operations and functionsall histogram samples are reconciled to one compatible bucket layout.

Not all bucket layouts can be reconciled, if incompatible histograms areencountered in an operation, the corresponding output vector element is removedfrom the result, flagged with a warn-level annotation.More details can be found in thenative histogram specification.

Literals

The following section describes literal values of various kinds.Note that there is no “histogram literal”.

String literals

String literals are designated by single quotes, double quotes or backticks.

PromQL follows the sameescaping rules asGo . For string literals in single or double quotes, abackslash begins an escape sequence, which may be followed bya,b,f,n,r,t,v or\. Specific characters can be provided using octal(\nnn) or hexadecimal (\xnn,\unnnn and\Unnnnnnnn) notations.

Conversely, escape characters are not parsed in string literals designated by backticks. It is important to note that, unlike Go, Prometheus does not discard newlines inside backticks.

Example:

"this is a string"'these are unescaped: \n \\ \t'`these are not unescaped: \n ' " \t`

Float literals and time durations

Scalar float values can be written as literal integer or floating-point numbersin the format (whitespace only included for better readability):

[-+]?(      [0-9]*\.?[0-9]+([eE][-+]?[0-9]+)?    | 0[xX][0-9a-fA-F]+    | [nN][aA][nN]    | [iI][nN][fF])

Examples:

23-2.433.4e-90x8f-InfNaN

Additionally, underscores (_) can be used in between decimal or hexadecimaldigits to improve readability.

Examples:

1_000_000.123_456_7890x_53_AB_F3_82

Float literals are also used to specify durations in seconds. For convenience,decimal integer numbers may be combined with the followingtime units:

• ms – milliseconds
• s – seconds – 1s equals 1000ms
• m – minutes – 1m equals 60s (ignoring leap seconds)
• h – hours – 1h equals 60m
• d – days – 1d equals 24h (ignoring so-called daylight saving time)
• w – weeks – 1w equals 7d
• y – years – 1y equals 365d (ignoring leap days)

Suffixing a decimal integer number with one of the units above is a differentrepresentation of the equivalent number of seconds as a bare float literal.

Examples:

1s # Equivalent to 1.2m # Equivalent to 120.1ms # Equivalent to 0.001.-2h # Equivalent to -7200.

The following examples donot work:

0xABm # No suffixing of hexadecimal numbers.1.5h # Time units cannot be combined with a floating point.+Infd # No suffixing of ±Inf or NaN.

Multiple units can be combined by concatenation of suffixed integers. Unitsmust be ordered from the longest to the shortest. A given unit must only appearonce per float literal.

Examples:

1h30m # Equivalent to 5400s and thus 5400.12h34m56s # Equivalent to 45296s and thus 45296.54s321ms # Equivalent to 54.321.

Time series selectors

These are the basic building-blocks that instruct PromQL what data to fetch.

Instant vector selectors

Instant vector selectors allow the selection of a set of time series and asingle sample value for each at a given timestamp (point in time). In the simplestform, only a metric name is specified, which results in an instant vectorcontaining elements for all time series that have this metric name.

The value returned will be that of the most recent sample at or before thequery's evaluation timestamp (in the case of aninstant query)or the current step within the query (in the case of arange query).The@ modifier allows overriding the timestamp relative to whichthe selection takes place. Time series are only returned if their most recent sample is less than thelookback period ago.

This example selects all time series that have thehttp_requests_total metricname, returning the most recent sample for each:

http_requests_total

It is possible to filter these time series further by appending a comma-separated list of labelmatchers in curly braces ({}).

This example selects only those time series with thehttp_requests_totalmetric name that also have thejob label set toprometheus and theirgroup label set tocanary:

http_requests_total{job="prometheus",group="canary"}

It is also possible to negatively match a label value, or to match label valuesagainst regular expressions. The following label matching operators exist:

• =: Select labels that are exactly equal to the provided string.
• !=: Select labels that are not equal to the provided string.
• =~: Select labels that regex-match the provided string.
• !~: Select labels that do not regex-match the provided string.

Regex matches are fully anchored. A match ofenv=~"foo" is treated asenv=~"^foo$".

For example, this selects allhttp_requests_total time series forstaging,testing, anddevelopment environments and HTTP methods other thanGET.

http_requests_total{environment=~"staging|testing|development",method!="GET"}

Label matchers that match empty label values also select all time series thatdo not have the specific label set at all. It is possible to have multiple matchers for the same label name.

For example, given the dataset:

http_requests_totalhttp_requests_total{replica="rep-a"}http_requests_total{replica="rep-b"}http_requests_total{environment="development"}

The queryhttp_requests_total{environment=""} would match and return:

http_requests_totalhttp_requests_total{replica="rep-a"}http_requests_total{replica="rep-b"}

and would exclude:

http_requests_total{environment="development"}

Multiple matchers can be used for the same label name; they all must pass for a result to be returned.

The query:

http_requests_total{replica!="rep-a",replica=~"rep.*"}

Would then match:

http_requests_total{replica="rep-b"}

Vector selectors must either specify a name or at least one label matcherthat does not match the empty string. The following expression is illegal:

{job=~".*"} # Bad!

In contrast, these expressions are valid as they both have a selector that does notmatch empty label values.

{job=~".+"}              # Good!{job=~".*",method="get"} # Good!

Label matchers can also be applied to metric names by matching against the internal__name__ label. For example, the expressionhttp_requests_total is equivalent to{__name__="http_requests_total"}. Matchers other than= (!=,=~,!~) may also be used.The following expression selects all metrics that have a name starting withjob::

{__name__=~"job:.*"}

The metric name must not be one of the keywordsbool,on,ignoring,group_left andgroup_right. The following expression is illegal:

on{} # Bad!

A workaround for this restriction is to use the__name__ label:

{__name__="on"} # Good!

Range Vector Selectors

Range vector literals work like instant vector literals, except that theyselect a range of samples back from the current instant. Syntactically, afloat literal is appended in squarebrackets ([]) at the end of a vector selector to specify for how many secondsback in time values should be fetched for each resulting range vector element.Commonly, the float literal uses the syntax with one or more time units, e.g.[5m]. The range is a left-open and right-closed interval, i.e. samples withtimestamps coinciding with the left boundary of the range are excluded from theselection, while samples coinciding with the right boundary of the range areincluded in the selection.

In this example, we select all the values recorded less than 5m ago for alltime series that have the metric namehttp_requests_total and ajob labelset toprometheus:

http_requests_total{job="prometheus"}[5m]

Offset modifier

Theoffset modifier allows changing the time offset for individualinstant and range vectors in a query.

For example, the following expression returns the value ofhttp_requests_total 5 minutes in the past relative to the currentquery evaluation time:

http_requests_total offset 5m

Note that theoffset modifier always needs to follow the selectorimmediately, i.e. the following would be correct:

sum(http_requests_total{method="GET"} offset 5m) // GOOD.

While the following would beincorrect:

sum(http_requests_total{method="GET"}) offset 5m // INVALID.

The same works for range vectors. This returns the 5-minuteratethathttp_requests_total had a week ago:

rate(http_requests_total[5m] offset 1w)

When querying for samples in the past, a negative offset will enable temporal comparisons forward in time:

rate(http_requests_total[5m] offset -1w)

Note that this allows a query to look ahead of its evaluation time.

@ modifier

The@ modifier allows changing the evaluation time for individual instantand range vectors in a query. The time supplied to the@ modifieris a Unix timestamp and described with a float literal.

For example, the following expression returns the value ofhttp_requests_total at2021-01-04T07:40:00+00:00:

http_requests_total @ 1609746000

Note that the@ modifier always needs to follow the selectorimmediately, i.e. the following would be correct:

sum(http_requests_total{method="GET"} @ 1609746000) // GOOD.

While the following would beincorrect:

sum(http_requests_total{method="GET"}) @ 1609746000 // INVALID.

The same works for range vectors. This returns the 5-minute rate thathttp_requests_total had at2021-01-04T07:40:00+00:00:

rate(http_requests_total[5m] @ 1609746000)

The@ modifier supports all representations of numeric literals described above.It works with theoffset modifier where the offset is applied relative to the@modifier time. The results are the same irrespective of the order of the modifiers.

For example, these two queries will produce the same result:

# offset after @http_requests_total @ 1609746000 offset 5m# offset before @http_requests_total offset 5m @ 1609746000

Additionally,start() andend() can also be used as values for the@ modifier as special values.

For a range query, they resolve to the start and end of the range query respectively and remain the same for all steps.

For an instant query,start() andend() both resolve to the evaluation time.

http_requests_total @ start()rate(http_requests_total[5m] @ end())

Note that the@ modifier allows a query to look ahead of its evaluation time.

Subquery

Subquery allows you to run an instant query for a given range and resolution. The result of a subquery is a range vector.

Syntax:<instant_query> '[' <range> ':' [<resolution>] ']' [ @ <float_literal> ] [ offset <float_literal> ]

• <resolution> is optional. Default is the global evaluation interval.

Operators

Prometheus supports many binary and aggregation operators. These are describedin detail in theexpression language operators page.

Functions

Prometheus supports several functions to operate on data. These are describedin detail in theexpression language functions page.

Comments

PromQL supports line comments that start with#. Example:

    # This is a comment

Regular expressions

All regular expressions in Prometheus useRE2 syntax .

Regex matches are always fully anchored.

Gotchas

Staleness

The timestamps at which to sample data, during a query, are selectedindependently of the actual present time series data. This is mainly to supportcases like aggregation (sum,avg, and so on), where multiple aggregatedtime series do not precisely align in time. Because of their independence,Prometheus needs to assign a value at those timestamps for each relevant timeseries. It does so by taking the newest sample that is less than the lookback period ago.The lookback period is 5 minutes by default, but can beset with the--query.lookback-delta flagor overridden on an individual query via thelookback_delta parameter.

If a target scrape or rule evaluation no longer returns a sample for a timeseries that was previously present, this time series will be marked as stale.If a target is removed, the previously retrieved time series will be marked asstale soon after removal.

If a query is evaluated at a sampling timestamp after a time series is markedas stale, then no value is returned for that time series. If new samples aresubsequently ingested for that time series, they will be returned as expected.

A time series will go stale when it is no longer exported, or the target nolonger exists. Such time series will disappear from graphsat the times of their latest collected sample, and they will not be returnedin queries after they are marked stale.

Some exporters, which put their own timestamps on samples, get a different behaviour:series that stop being exported take the last value for (by default) 5 minutes beforedisappearing. Thetrack_timestamps_staleness setting can change this.

Avoiding slow queries and overloads

If a query needs to operate on a substantial amount of data, graphing it mighttime out or overload the server or browser. Thus, when constructing queriesover unknown data, always start building the query in the tabular view ofPrometheus's expression browser until the result set seems reasonable(hundreds, not thousands, of time series at most). Only when you have filteredor aggregated your data sufficiently, switch to graph mode. If the expressionstill takes too long to graph ad-hoc, pre-record it via arecordingrule.

This is especially relevant for Prometheus's query language, where a baremetric name selector likeapi_http_requests_total could expand to thousandsof time series with different labels. Also, keep in mind that expressions thataggregate over many time series will generate load on the server even if theoutput is only a small number of time series. This is similar to how it wouldbe slow to sum all values of a column in a relational database, even if theoutput value is only a single number.