measurements

Table of contents

By default, CFEngine's monitoring componentcf-monitord records performancedata about the system. These include process counts, service traffic, loadaverage and CPU utilization and temperature when available. It also records athree year trend summary based any 'shift'-averages.

Custommeasurements promises can monitor or log very specific user datathrough a generic interface. The end-result is to either generate a periodictime series, like the above mentioned values, or to log the results tocustom-defined reports.

Promises of typemeasurement are written just like all other promises within abundle destined for the agent concerned, in this casemonitor. However, it isnot necessary to add them to thebundlesequence, becausecf-monitordexecutes all bundles of typemonitor.

code

bundlemonitorself_watch{measurements:# Follow a special process over time# using CFEngine's process cache to avoid resampling# Example content from /var/cfengine/state/cf_rootprocs#USER                             PID         PPID          PGID          %CPU         %MEM        VSZ        NI         RSS    TTY      NLWP STIME     ELAPSED     TIME COMMAND#root                             19103       1             19103         0.2          2.1         71716      0          10676  ?           1 18:09       40:13 00:00:06 /var/cfengine/bin/cf-monitord --no-fork# match_value:#root \s+                         [0-9.]+ \s+ [0-9.]+  \s+  [0-9.]+ \s+   [0-9.]+ \s+  [0-9.]+ \s+ [0-9]+ \s+ [0-9]+ \s+ ([0-9]+) .*""/var/cfengine/state/cf_rootprocs"handle=>"cf_monitord_RSS",stream_type=>"file",data_type=>"int",history_type=>"weekly",units=>"kb",match_value=>proc_value(".*cf-monitord.*","root\s+[0-9.]+\s+[0-9.]+\s+[0-9.]+\s+[0-9.]+\s+[0-9.]+\s+\s+[0-9.]+\s+[0-9.]+\s+([0-9]+).*"),comment=>"The ammount of memory (RSS or Resident Set Size) cf-monitored is consuming";}bodymatch_valueproc_value(x,y){select_line_matching=>"$(x)";select_multiline_policy=>"sum";extraction_regex=>"$(y)";}

code

bundlemonitorwatch_diskspace{measurements:# Discover disk device information"/bin/df"handle=>"free_diskspace_watch",stream_type=>"pipe",data_type=>"slist",history_type=>"static",units=>"device",match_value=>file_systems;}bodymatch_valuefile_systems{select_line_matching=>"/.*";extraction_regex=>"(.*)";}

The general pattern of these promises is to decide whether the source of theinformation is either a file or pipe, determine the data type (integer, stringetc.), specify a pattern to match the result in the file stream and then specifywhat to do with the result afterwards.

History:

Custom measurements open sourced in 3.12.0

Architecture

Information gathered bycf-monitord is stored in${sys.statedir}/env_data toshare with other agents likecf-agent andcf-promises which both provide threevariables per measurement with different prefixes:value_ for the latest measurement,av_ for the running average anddev_ for the standard deviation of the running average.

The standard measurements and their variable names are listed inmon special variables.

The data over time is stored in${sys.statedir}/cf_observations.lmdb which can be examinedmost easily with thecf-check command:

code

cf-check dump /var/cfengine/state/cf_observations.lmdb

By default in theMasterfiles Policy Framework,cf-serverd uses two variables,def.default_data_select_host_monitoring_include anddef.default_data_select_policy_hub_monitoring_include toconfigure which measurements will be included in enterprise reporting.

On the hub side, reports are collected and measurements data is inserted into theMonitoringHGMonitoringMgMeta andMonitoringYrMeta tables of the Enterprise Hub database.

A diagnostic query to run with aCustom Report in Mission Portal.

Select all unique observables in the database:

code

SELECTdistinct(observable)FROMmonitoringmgmeta;

To see which measures are collected for each host, select all data from thistable and then use report result filters to examine a particular host or measure's havingdata or not.

code

SELECT*FROMmonitoringmgmeta;

Measurement data is presented in Mission Portal in theMeasurements App and in theMeasurements section of theHost info page.

When policy is changed in regards to monitor bundles, bothcf-monitordandcf-serverd should be restarted in order to receive the updated policy.

It is possible toconfigure masterfiles to restartcf-monitord when variables which affect it's configuration are changed.

All measurements historical data is stored in${sys.statedir}/cf_observations.lmdb. This is where reporting data is pulled from.

${sys.statedir}/env_data is a current snapshot of measurement values which agents (cf-agent andcf-promises) use to generate thespecial monitor variables.

${sys.statedir}/ts_key is the source for the list of observables/measurements used in the system. This file includes the integer id, name and other meta data in a slightly human readable form, for example:

code

0,users,Users with active processes - including system users,average users per 2.5 mins,0.000,100.000,1

Thets_key file should not be altered.

Note that if a measurement hasalways had a value of zero it will not be reported and so not available in Mission Portal Measurements or Host info pages.

It is important to specify a promisehandle for measurement promises, as thenames defined in the handle are used to determine the name of the log file orvariable to which data will be reported. Log files are created underWORKDIR/state. Data that have no history type are stored in a special variablecontext calledmon, analogous to the system variables in sys. Thus the valuesmay be used in other promises in the form$(mon.handle).

Attributes

stream_type

Description: The datatype being collected.

Default:pipe

CFEngine treats all input using a stream abstraction. The preferred interfaceis files, since they can be read without incurring the cost of a process.However pipes from executed commands may also be invoked.

Type: (menu option)

Allowed input range:

code

pipefile

Example:

code

stream_type=>"pipe";

data_type

Description: The datatype being collected.

When CFEngine observes data, such as the attached partitions in the example above, the datatype determines how that data will be handled. Integer and real values, counters etc., are recorded as time-series if the history type is 'weekly', or as single values otherwise. If multiple items are matched by an observation (e.g. several lines in a file match the given regular expression), then these can be made into a list by choosingslist, else the first matching item will be selected.

Type: (menu option)

Allowed input range:

code

counterintrealstringslist

Example:

code

"/bin/df"handle=>"free_disk_watch",stream_type=>"pipe",data_type=>"slist",history_type=>"static",units=>"device",match_value=>file_systems,action=>sample_min(10,15);

history_type

Description: Whether the data can be seen as a time-series or just anisolated value

Type: (menu option)

Allowed input range:

scalar

A single value, with compressed statistics is retained. The value of thedata is not expected to change much for the lifetime of the daemon (andso will be sampled less often by cf-monitord).

static

A synonym for 'scalar'.

log

The measured value is logged as an infinite time-series in$(sys.workdir)/state/<handle>_measure.log.

weekly

A standard CFEngine two-dimensional time average (over a weekly period)is retained.

Example:

code

"/proc/meminfo"handle=>"free_memory_watch",stream_type=>"file",data_type=>"int",history_type=>"weekly",units=>"kB",match_value=>free_memory;

Notes:

Measurements with history typeweeklyare collected by CFEngine Enterprise reporting.
Measurements with history typestatic arenot collected by CFEngine Enterprise reporting.
Measurements with history typescalar arenot collected by CFEngine Enterprise reporting.
Measurements with history typelog arenot collected by CFEngine Enterprise reporting.

units

Description: The engineering dimensions of this value or a note aboutits intent used in plots

This is an arbitrary string used in documentation only.

Type:string

Allowed input range: (arbitrary string)

Example:

code

"/var/cfengine/state/cf_rootprocs"handle=>"monitor_self_watch",stream_type=>"file",data_type=>"int",history_type=>"weekly",units=>"kB",match_value=>proc_value(".*cf-monitord.*","root\s+[0-9.]+\s+[0-9.]+\s+[0-9.]+\s+[0-9.]+\s+([0-9]+).*");

match_value

Type:body match_value

See also:Common body attributes

select_line_matching

Description: Regular expression for matching line location

The expression isanchored, meaning it must match a whole line, and nota fragment within a line.

This attribute is mutually exclusive ofselect_line_number.

Type:string

Allowed input range:.*

Example:

code

# Editingbodylocationexample{select_line_matching=>"Expression match.* whole line";}# Measurement promisesbodymatch_valueexample{select_line_matching=>"Expression match.* whole line";}

select_line_number

Description: Read from the n-th line of the output (fixed format)

This is mutually exclusive ofselect_line_matching.

Type:int

Allowed input range:0,99999999999

Example:

code

bodymatch_valuefind_line{select_line_number=>"2";}

extraction_regex

Description: Regular expression that should contain a singleback-reference for extracting a value.

A single parenthesized back-reference should be given to lift the value to bemeasured out of the text stream. The regular expression isunanchored, meaningit may match a partial string

Type:string

Allowed input range: (arbitrary string)

Example:

code

bodymatch_valuefree_memory{select_line_matching=>"MemFree:.*";extraction_regex=>"MemFree:\s+([0-9]+).*";}

track_growing_file

Description: If true, CFEngine remembers the position to which is lastread when opening the file, and resets to the start if the file hassince been truncated

This option applies only to file based input streams. If this is true,CFEngine treats the file as if it were a log file, growing continuously.Thus the monitor reads all new entries since the last sampling time oneach invocation. In this way, the monitor does not count lines in thelog file redundantly.

This makes a log pattern promise equivalent to something like tail -flogfile | grep pattern in Unix parlance.

Type:boolean

Example:

code

bundlemonitorwatch{measurements:"/home/mark/tmp/file"handle=>"line_counter",stream_type=>"file",data_type=>"counter",match_value=>scan_log("MYLINE.*"),history_type=>"log",action=>sample_rate("0");}#bodymatch_valuescan_log(x){select_line_matching=>"^$(x)$";track_growing_file=>"true";}#bodyactionsample_rate(x){ifelapsed=>"$(x)";expireafter=>"10";}

select_multiline_policy

Description: Regular expression for matching line location

This option governs how CFEngine handles multiple matching lines in theinput stream. It can average or sum values if they are integer or real,or use first or last representative samples. If non-numerical data typesare used only the first match is used.

Type: (menu option)

Allowed input range:

code

averagesumfirstlast

Example:

code

bodymatch_valuemyvalue(xxx){select_line_matching=>".*$(xxx).*";extraction_regex=>"root\s+\S+\s+\S+\s+\S+\s+\S+\s+\S+\s+\S+\s+\S+\s+(\S+).*";select_multiline_policy=>"sum";}