This page provides general information about Monitoring Query Language (MQL), includingthe following topics:

• Query shortcuts for table operations and functions.
• Variations on MQLfetch operations.
• Strict-form queries.
• Rules for matching theresource.project_idcolumn.
• How the“edge effect” can affect ratioscomputed across metric types.
• Date formats supported by MQL.
• Limits on querylength and complexity.
• Macros in MQL.

For information about the structure of MQL queries, seeExamples.

Shortcuts for table operations and functions

Queries usually consist of connected chains of table operations connected bypipes (|), each of which starts with the name of the table operationfollowed by a list of expressions. Expressions can contain function calls thatlist all of their arguments explicitly. But Monitoring Query Language allows queries to beexpressed with a number of shortcuts.

This section describes shortcuts for table operations, using functionsas table operations, and a shortcut for value columns as arguments to functions.

For a full list, seeTable operation shortcuts.

Shortcuts for table operations

When using thefetch,group_by, andfilter operations, you can omit theexplicit table operation when the arguments are sufficient to determine theintended operation. For example, the following query:

gce_instance::compute.googleapis.com/instance/cpu/utilization

is equivalent to:

fetch gce_instance::compute.googleapis.com/instance/cpu/utilization

The followinggroup_by operations are equivalent:

         [zone], mean(val())group_by [zone], mean(val())

You can omit the wordfilter if you parenthesize the filter test. For example,the following twofilter operations are the equivalent:

       (instance_name =~ 'apache.*')filter instance_name =~ 'apache.*'

You can combine these shortcut forms in your queries. For example, thefollowing query:

gce_instance::compute.googleapis.com/instance/cpu/utilization|(instance_name=~'apache.*')|[zone],mean(val())

is equivalent to this more explicit form:

fetchgce_instance::compute.googleapis.com/instance/cpu/utilization|filterinstance_name=~'apache.*'|group_by[zone],mean(val())

For more information on shortcuts for table operations, seeTable operation shortcuts in the MQL reference.

Using a function as a table operation

A table operation usually starts with the name of a table operation. ButMQL allows a table operation to start with a function name instead.

You can use a function name when the named function is to perform sometransformation of the value columns of the input table. This replacement is ashortcut for thegroup_by,align, orvalue table operations, depending on the kind of function whose name is given.

The general form is:

|FUNCTION_NAMEARG,ARG ...

In the table operation, the function takes the value columns of the input table as arguments, arguments followed by any arguments for the function itself.When using a function as a table operation, you specify the arguments in thetable-operation form, as a comma-separated list, rather than with thesurrounding parentheses (()) ordinarily used with functions.

The full table operation generated by expanding the shortcut depends on thekind of function:

• group_by: If you are using anaggregating functionwith agroup_by operation that aggregates away all thetime-series identifier columns (that is, it uses[] for grouping), thenyou can use the function as a shortcut. For example,

  | distribution powers_of(1.1)

  is a shortcut for

  | group_by [], distribution(val(0), powers_of(1.1))

• align: If you are using analigning function as anargument to thealign operation, you can use the functionas a shortcut. For example,

  | delta

  is a shortcut for

  | align delta()

  Similarly,

  | rate 10m

  is a shortcut for

  | align rate(10m)

  Note that aligner functions take the input time series as an implicitargument, so the value columns are not given explicitly here.

• value: All other functions can act as shortcuts for thevalue table operation. For example,

  | mul 3.3

  is a shortcut for

  | value mul(val(0), 3.3)

  Similarly,

  | div

  is a shortcut for

  | value div(val(0), val(1))

  Note that thediv shortcut an input table operation with two valuecolumns and produces a table operation with one value column that is theratio.

Shortcut for value-column functions

You can use.function as a shortcutforfunction(val()) if there is a single valuecolumn in the input, or as a shortcut forfunction(val(0), val(1)) if there there are two valuecolumns, and so forth.

The leading dot means, “Call the following function, supplying theinput-point value column (or columns) as the argument(s) to the function.”

For example,.mean is a shortcut formean(val()). The following areequivalent:

group_by [zone], .meangroup_by [zone], mean(val())

If the input table has multiple value columns, each column becomes an argumentto the function in this shortcut. For example, if the input table has twovalue columns, then

.div

is a shortcut for

div(val(0), val(1))

With this shortcut, you can supply arguments that do not refer to valuecolumns. The additional arguments are supplied after the value-columnarguments. For example, if the input table has one value column, then

.div(3)

is equivalent to

div(val(0), 3)

Variations on afetch

Thefetch operation usually returns a time-series table named by a pair ofmonitored-resource and metric types. For example:

fetch gce_instance :: compute.googleapis.com/instance/cpu/utilization

If the metric applies only to one monitored-resource type, then you can omitthe monitored resource from the query. The following query is equivalent tothe previous query, because the CPU-utilization metric applies only togce_instance monitored resources:

fetch compute.googleapis.com/instance/cpu/utilization

Thefetch operation can specify only a monitored-resource type, with themetric specified in a subsequentmetric operations. Forexample, this example is equivalent to the previousfetch examples:

fetch gce_instance| metric metric compute.googleapis.com/instance/cpu/utilization

Splitting thefetch this way can be useful when you want to fetch twodifferent metrics for the same monitored resource. For example, the followingquery computes the number of packets per CPU-second consumed:

fetch gce_instance| {    metric compute.googleapis.com/instance/network/received_packets_count ;    metric compute.googleapis.com/instance/cpu/usage_time  }| ratio

Splitting thefetch also lets you apply filtering only to the labels ofthe monitored resource:

fetch gce_instance| filter resource.zone =~ "asia.*"| {    metric compute.googleapis.com/instance/network/received_packets_count ;    metric compute.googleapis.com/instance/cpu/usage_time  }| ratio

Afetch that names only a monitored-resource type must be followed by ametric operation, perhaps with interveningfilter operations.

Strict-form queries

A strict query is one with none of the shortcuts or implicit values usedin concise queries. Strict queries have the following characteristics:

• All shortcuts are replaced.
• All implicit arguments are made explicit.
• Columns are referred to by full names.
• New columns are explicitly given names.
• Any implicitly supplied alignment operations are given explicitly.

Using the strict form makes the query more resilient to changes in thestructure of input tables, and it can make it clearer what the query is doing.Putting a query in strict form does not make the query any more efficient.

When you save a query for a chart, it is converted to strict form.The confirmation dialog for the save operation displays the strict form.

Concise queries for alerting policies are not converted to strict form.Queries for alerting policies are stored as you provide them; you can useeither concise or strict form.

With both shortcuts and strict forms available, you might encounter equivalentMQL queries that look very different from each other. For example,the following query, which computes the number of received packets perconsumed CPU second, uses many shortcuts:

gce_instance| (zone =~ ".*-a")| {    compute.googleapis.com/instance/network/received_packets_count ;    compute.googleapis.com/instance/cpu/usage_time  }| join| div

When you save this query for a chart or as part of an alerting policy, theresulting strict form query does exactly the same thing. However, the strictform might look quite different, as shown in the following example:

fetchgce_instance|filter(resource.zone=~'.*-a')|{t_0:metric'compute.googleapis.com/instance/network/received_packets_count'|aligndelta();t_1:metric'compute.googleapis.com/instance/cpu/usage_time'|aligndelta()}|join|value[v_0:div(t_0.value.received_packets_count,t_1.value.usage_time)]

When you edit the saved definition of the chart, the code editor displaysthe strict form.

Matching theresource.project_id column

Google Cloud projects have a display name, which appears in menus but does notuniquely identify the project. A project display name might be "Monitoringdemo".

Projects also have two fields that act as identifiers:

• Project ID: a unique string identifier. This is often based on the displayname. Project IDs are set when the project is created, usually byconcatenating the elements of the project name and possibly adding digits tothe end, if needed for uniqueness. A project ID might have the form"monitoring-demo" or "monitoring-demo-2349". The project IDis sometimes casually called the project name.
• Project number: a unique numeric identifier.

Every monitored-resource type includes aproject_id label, with a stringrepresentation of the project number of the project that owns the resourceand the data about that resource.

In MQL queries, you refer to this label asresource.project_id.Theresource.project_id label has the project number in text form as itsvalue, but MQL converts that value to the project ID in certainsituations.

In the following cases, MQL treats the value of theresource.project_id label as the project ID rather than the project number:

• The legend for a chart displays the project ID rather than the projectnumber for the value of theresource.project_id label.

• Equality comparisons of the value of theresource.project_id toa string literal recognize both the project number and theproject ID. For example, both the following returns true forresources owned by this project:

  • resource.project_id == "monitoring-demo"
  • resource.project_id == "530310927541"

  This case applies for the== and!= operators and for their functionforms,eq() andne().

• A regular-expression match on theresource.project_id label workproperly against either the project number or project ID. For example,both of the following expressions returntrue for resources owned bythis project:

  • resource.project_id =~ "monitoring-.*"
  • resource.project_id =~ ".*27541"

  This case applies for the=~ and!~ operators and for the function form,re_full_match.

For all other cases, the actual value of theresource.project_id label isused. For example,concatenate("project-", resource.project_id) results inthe valueproject-530310927541 and notproject-monitoring-demo.

Ratios and the “edge effect”

In general, it is best to compute ratios based on time series collected fora single metric type, by using label values. A ratio computed over twodifferent metric types is subject to anomalies due to different samplingperiods and alignment windows.

For example, suppose that you have two different metric types, an RPC totalcount and an RPC error count, and you want to compute the ratio of error-countRPCs over total RPCs. The unsuccessful RPCs are counted in the time series ofboth metric types. Therefore, there is a chance that, when you align the timeseries, an unsuccessful RPC doesn't appear in the same alignment interval forboth time series. This differencecan happen for several reasons, including the following:

• Because there are two different time series recording the same event, thereare two underlying counter values implementing the collection, and theyaren't updated atomically.
• The sampling rates might differ. When the time series are aligned to a commonperiod, the counts for a single event might appear in adjacent alignmentintervals in the time series for the different metrics.

The difference in the number of values in corresponding alignment intervals canlead to nonsensicalerror/total ratio values like 1/0 or 2/1.

Ratios of larger numbers are less likely to result in nonsensical values.You can get larger numbers by aggregation, either by using an alignment windowthat islonger than the sampling period, or by grouping data for certainlabels. These techniques minimize the effect of small differences in thenumber of points in a given interval. That is, a two-point disparity is moresignificant when the expected number of points in an interval is 3 than whenthe expected number is 300.

If you are using built-in metric types, then you might have no choice but tocompute ratios across metric types to get the value you need.

If you are designing custom metrics that might count the same thing—likeRPCs returning error status—in two different metrics, consider insteada single metric, which includes each count only once. For example, supposethat you are counting RPCs and you want to track the ratio of unsuccessfulRPCs to all RPCs. To solve this problem,create a single metric type to count RPCs, and use a label to record thestatus of the invocation, including the "OK" status. Then each status value,error or "OK", is recorded by updating a single counter for that case.

MQL date formats

MQL supports a limited number of date formats.In MQL queries, dates are expressed as one of the following:

• d'BASE_STRING'
• D'BASE_STRING'

TheBASE_STRING is a string of the form2010/06/23-19:32:15-07:00. The first dash (-), separating the date and time,can be replaced by a space. In the time component, parts of the clock time(19:32:15) or the timezone specifier (-07:00) can be dropped.

The following examples are valid dates in MQL queries:

• d'2010/06/23-19:32:15-07:00'
• d'2010/06/23 19:32:15-07:00'
• d'2010/06/23 19:32:15'
• D'2010/06/23 19:32'
• d'2010/06/23-19'
• D'2010/06/23 -07:00'

The following table lists the grammar for theBASE_STRING:

Length and complexity of queries

Monitoring Query Language queries can be long and complex, but not without limits.

• A query text, encoded as UTF-8, is limited to 10,000 bytes.
• A query is limited to 2,000 language constructs; that is,the AST^† complexity is limited to 2,000nodes.

^† Anabstract syntax tree,or AST, is a representation of source code—in this case, the MQLquery string—in which nodes in the tree map to syntactic structures inthe code.

MQL macros

This product or feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of theService Specific Terms. Pre-GA products and features are available "as is" and might have limited support. For more information, see thelaunch stage descriptions.

MQL includes a macro-definition utility. You can use the MQLmacros to replace repeated operations, make complex queries more readable,and make query development easier. You can define macros for table operationsand for functions.

Macro definitions start with the keyworddef.

When you convert a query tostrict form, macro invocationsare replaced with their corresponding text and the macro definitions areremoved.

When you save a chart query that includes macros, the query is converted tostrict form, so any macros are not preserved. When you save a query for acondition in an alerting policy, the query is not converted to strict form,so macros are preserved.

Macros for table operations

You can write macros to make new table operations. The general syntax lookslike the following:

defMACRO_NAME [MACRO_PARAMETER[,MACRO_PARAMETER]] =MACRO_BODY ;

To invoke the macro, use the following syntax:

@MACRO_NAME [MACRO_ARG [,MACRO_ARG]]

For example, suppose you are using the following query to fetch CPU utilizationdata:

fetchgce_instance::compute.googleapis.com/instance/cpu/utilization|every1m|group_by[zone],mean(val())

The first line can be replaced with the following macro:

def my_fetch = fetch gce_instance::compute.googleapis.com/instance/cpu/utilization ;

To invoke the macro in the query, replace the originalfetch as follows:

defmy_fetch=fetchgce_instance::compute.googleapis.com/instance/cpu/utilization;@my_fetch|every1m|group_by[zone],mean(val())

You can replace the second and third lines with macros that take arguments.The macro definition lists the parameters to the macro, and in the macro body,refer to the parameters to the macro as$MACRO_PARAMETER. For example, youcan define the following macros:

def my_every time_arg = every $time_arg ;def my_group label, aggr = group_by [$label], $aggr ;

To invoke these macros and provide the arguments, specify the arguments ina comma-delimited list in the macro invocations. The following shows thequery with all the defined macros and their invocations:

defmy_fetch=fetchgce_instance::compute.googleapis.com/instance/cpu/utilization;defmy_everytime_arg=every$time_arg;defmy_grouplabel,aggr=group_by[$label],$aggr;{@my_fetch}|@my_every1m|@my_groupzone,mean(val())

Macros are not preserved when the query is converted tostrict-form. For example, the strict form of the previousquery looks like the following:

fetch gce_instance::compute.googleapis.com/instance/cpu/utilization| align mean_aligner()| every 1m| group_by [resource.zone],           [value_utilization_mean: mean(value.utilization)]

Macros for functions

For MQL functions, you specify any parameters in a comma-delimitedlist in parentheses. The parentheses distinguish a function macro from atable-operation macro. The parenthese must appear in the invocation, even ifthere are no arguments. Macros are not preserved when thequery is converted tostrict-form.

defMACRO_NAME([MACRO_PARAMETER [,MACRO_PARAMETER]]) =MACRO_BODY ;

For example, the following query retrieves tables for two metrics, combinesthe two tables into one with two value columns, and computes the ratio ofreceived bytes to total bytes for a column calledreceived_percent:

{  fetch k8s_pod :: kubernetes.io/pod/network/received_bytes_count ;  fetch k8s_pod :: kubernetes.io/pod/network/sent_bytes_count}| join| value [received_percent:val(0) * 100 / (val(0) + val(1))]

You can replace thereceived_percent computation with a macro like thefollowing example:

def recd_percent(recd, sent) = $recd * 100 / ($recd + $sent) ;

To invoke a function macro, use the following syntax:

@MACRO_NAME([MACRO_ARG[,MACRO_ARG]])

When invoking a function macro with no arguments, you must specify the emptyparentheses to distinguish the invocation from the invocation of atable-operation macro.

The following example shows the previous query with a macro for the ratiocomputation:

defrecd_percent(recd,sent)=$recd*100/($recd+$sent);{fetchk8s_pod::kubernetes.io/pod/network/received_bytes_count;fetchk8s_pod::kubernetes.io/pod/network/sent_bytes_count}|join|value[received_percent:@recd_percent(val(0), val(1))]

Macro capabilities

MQL macros aresyntactic elements, as opposed totextual elementslike the macros used in the C preprocessor. This distinction means that anMQL macro body must always be a syntactically valid expression. It mightnot be semantically valid, which also depends on the macro arguments and on thelocation where the macro is expanded.

Because MQL macros are syntactic, there are very few restrictions on the kindof expression they can expand to. Syntactic macros are just another way tomanipulate theabstract syntax tree. The followingexamples show some of the things you can do with syntactic macros:

#Abbreviatingacolumnname.defmy_col()=instance_name;#Map-valuedmacro.defmy_map(c)=[$c, @my_col()];#Abbreviatingastring.defmy_zone()='us-central.*';#Abbreviatingafilterexpression.defmy_filter(f)=zone=~@my_zone() &&$f;

MQL also supports implicit string-literal concatenation. This featurecan be very useful when writing queries that include long metric names. When astring literal and a macro argument, which also has to be a string literal,appear next to each other in the macro body, macro expansion concatenates theminto a single string literal.

In the following example,gce_instance is aBARE_NAME lexical element.It is automatically promoted to a string literal, which is useful in buildingtable names:

#Buildsatablenameindomain'd'withthesuffix'm'.defmy_table(d,m)=gce_instance::$d'/instance/'$m;#Tablenameunderthegivendomain.defmy_compute_table(m)=@my_table('compute.googleapis.com',$m);

Putting it all together, the following query uses all of the previously definedmacros:

fetch@my_compute_table('cpu/utilization')|filter@my_filter(instance_name=~'gke.*')|group_by@my_map(zone)

Note that macro arguments can also be arbitrary expressions, as long as theyare syntactically correct. For example, the macromy_filter cantake a Boolean expression likeinstance_name =~ 'gke.*' as itsfirst argument.

Abbreviating table operations can be very useful as well, as the followingquery demonstrates:

#Calculatetheratiobetweencomputemetrics'm1'and'm2'.defmy_compute_ratiom1,m2={fetch@my_compute_table($m1);fetch@my_compute_table($m2)}|join|div;#UsethetableopmacrotocalculatetheratiobetweenCPUutilizationand#thenumberofreservedcoresperzone.@my_compute_ratio'cpu/utilization','cpu/reserved_cores'|group_by[zone]

Finally, function macros can behave just like regular functions; that is, theyallow for function promotion where the value column or columns of the inputtable become the first arguments to the macro. The following example shows avariant of the previous query that uses a function macro:

#Simplearithmeticmacro.defmy_add_two(x)=$x+2;#Similartopreviousquery,butnowusingthenewarithmeticmacrowith#functionargumentpromotion.fetch@my_compute_table('cpu/utilization')|filter@my_filter(instance_name=~'gke.*')|group_by@my_map(zone),[.sum.@my_add_two]

Limitations

The MQL macro feature does not support the following:

• Nesting of macro definitions: you can't define a macro in the body of anothermacro.
• Recursively defined macros. No macro body can reference any macro,including itself, that is not yet fully defined.
• Use of macro-defined functions as table operations.
• Use of macro arguments as names of functions or table operations.
• Preservation of macros when the query is converted tostrictform. The macro invocations are replaced with thecorresponding expressions, and the macro definitions are removed.