• Field references
• Sprintf format
• Conditionals

When you need to refer to a field by name, you can use the Logstashfield reference syntax.

The basic syntax to access a field is[fieldname].If you are referring to atop-level field, you can omit the[] and usefieldname. To refer to anested field, specify the full path to that field:[top-level field][nested field].

For example, this event has five top-level fields (agent, ip, request, response, ua) and three nested fields (status, bytes, os).

{  "agent": "Mozilla/5.0 (compatible; MSIE 9.0)",  "ip": "192.168.24.44",  "request": "/index.html"  "response": {    "status": 200,    "bytes": 52353  },  "ua": {    "os": "Windows 7"  }}

To reference theos field, specify[ua][os].To reference a top-level field such asrequest, specify the field name.

You might find situations in which you need to refer to a field or collection of fields by name.You can accomplish this goal using the Logstash field reference syntax.

The syntax to access a field specifies the entire path to the field, with each fragment wrapped in square brackets.When a field name contains square brackets, the brackets must be properly escaped.

Field references can be expressed literally withinconditional statements in your pipeline configurations,as string arguments to your pipeline plugins, or within sprintf statements that will be used by your pipeline plugins:

filter {  #  +----literal----+     +----literal----+  #  |               |     |               |  if [@metadata][date] and [@metadata][time] {    mutate {      add_field {        "[@metadata][timestamp]" => "%{[@metadata][date]} %{[@metadata][time]}"      # |                      |    |  |               |    |               | |      # +----string-argument---+    |  +--field-ref----+    +--field-ref----+ |      #                             +-------- sprintf format string ----------+      }    }  }}

Expand the section below if you would like to explore the formal grammar.

fieldReferenceLiteral: ( pathFragment )+  ;

eventApiFieldReference: fieldReferenceLiteral  | fieldName  | compositeFieldReference  ;

pathFragment: '[' fieldName ']'  ;

fieldName: ( ~( '[' | ']' ) )+  ;

fieldReference = "[path][to][deep nested field]"compositeFieldReference = "[@metadata][#{fieldReference}][size]"# => "[@metadata][[path][to][deep nested field]][size]"

compositeFieldReference: ( pathFragment | embeddedFieldReference )+  ;

embeddedFieldReference: '[' fieldReference ']'  ;

For Logstash to reference a field whose name contains a character that has special meaning in the field reference grammar, the character must be escaped.Logstash can be globally configured to use one of two field reference escape modes:

• none (default): no escape sequence processing is done. Fields containing literal square brackets cannot be referenced by the Event API.
• percent: URI-style percent encoding of UTF-8 bytes. The left square bracket ([) is expressed as%5B, and the right square bracket (]) is expressed as%5D.
• ampersand: HTML-style ampersand encoding (&# + decimal unicode codepoint +;). The left square bracket ([) is expressed as[, and the right square bracket (]) is expressed as].

The field reference format is also used in what Logstash callssprintf format. This format enables you to embed field values in other strings. For example, the statsd output has anincrement setting that enables you to keep a count of apache logs by status code:

output {  statsd {    increment => "apache.%{[response][status]}"  }}

Similarly, you can convert the UTC timestamp in the@timestamp field into a string.

Instead of specifying a field name inside the curly braces, use the%{{FORMAT}} syntax whereFORMAT is ajava time format.

For example, if you want to use the file output to write logs based on the event’s UTC date and hour and thetype field:

output {  file {    path => "/var/log/%{type}.%{{yyyy.MM.dd.HH}}"  }}

You can generate a fresh timestamp by using%{{TIME_NOW}} syntax instead of relying on the value in@timestamp. This is particularly useful when you need to estimate the time span of each plugin.

input {  heartbeat {    add_field => { "heartbeat_time" => "%{{TIME_NOW}}" }  }}filter {  mutate {    add_field => { "mutate_time" => "%{{TIME_NOW}}" }  }}

Sometimes you want to filter or output an event only under certain conditions. For that, you can use a conditional.

Conditionals in Logstash look and act the same way they do in programming languages. Conditionals supportif,else if andelse statements and can be nested.

The conditional syntax is:

if EXPRESSION {  ...} else if EXPRESSION {  ...} else {  ...}

What’s an expression? Comparison tests, boolean logic, and so on!

You can use these comparison operators:

• equality:==,!=,<,>,<=,>=
• regexp:=~,!~ (checks a pattern on the right against a string value on the left)
• inclusion:in,not in

Supported boolean operators are:

• and,or,nand,xor

Supported unary operators are:

• !

Expressions can be long and complex. Expressions can contain other expressions, you can negate expressions with!, and you can group them with parentheses(...).

For example, this conditional uses the mutate filter to remove the fieldsecret if the fieldaction has a value oflogin:

filter {  if [action] == "login" {    mutate { remove_field => "secret" }  }}

If an expression generates an error when it is evaluated, event processing stops and a warning message is written to the log. For example, comparing integer value100 with string value"100" cannot be evaluated with certainty, and so processing stops and the error is logged.

To capture the full content of the message at the time the error occurs, set the log level todebug. Check outLogging for more information about how to configure logging and available log levels.

You can specify multiple expressions in a single condition:

output {  # Send production errors to pagerduty  if [loglevel] == "ERROR" and [deployment] == "production" {    pagerduty {    ...    }  }}

You can use thein operator to test whether a field contains a specific string, key, or list element. Note that the semantic meaning ofin can vary, based on the target type. For example, when applied to a string.in means "is a substring of". When applied to a collection type,in means "collection contains the exact value".

filter {  if [foo] in [foobar] {    mutate { add_tag => "field in field" }  }  if [foo] in "foo" {    mutate { add_tag => "field in string" }  }  if "hello" in [greeting] {    mutate { add_tag => "string in field" }  }  if [foo] in ["hello", "world", "foo"] {    mutate { add_tag => "field in list" }  }  if [missing] in [alsomissing] {    mutate { add_tag => "shouldnotexist" }  }  if !("foo" in ["hello", "world"]) {    mutate { add_tag => "shouldexist" }  }}

You use thenot in conditional the same way. For example, you could usenot in to only route events to Elasticsearch whengrok is successful:

output {  if "_grokparsefailure" not in [tags] {    elasticsearch { ... }  }}

You can check for the existence of a specific field, but there’s currently no way to differentiate between a field that doesn’t exist versus a field that’s simply false.The expressionif [foo] returnsfalse when:

• [foo] doesn’t exist in the event,
• [foo] exists in the event, but is false, or
• [foo] exists in the event, but is null

For more complex examples, seeUsing Conditionals.

In Logstash, there is a special field called@metadata. The contents of@metadata are not part of any of your events at output time, which makes it great to use for conditionals, or extending and building event fields with field reference andsprintf formatting.

This configuration file yields events from STDIN. Whatever you type becomes themessage field in the event. Themutate events in the filter block add a few fields, some nested in the@metadata field.

input { stdin { } }filter {  mutate { add_field => { "show" => "This data will be in the output" } }  mutate { add_field => { "[@metadata][test]" => "Hello" } }  mutate { add_field => { "[@metadata][no_show]" => "This data will not be in the output" } }}output {  if [@metadata][test] == "Hello" {    stdout { codec => rubydebug }  }}

Let’s see what comes out:

$ bin/logstash -f ../test.confPipeline main startedasdf{    "@timestamp" => 2016-06-30T02:42:51.496Z,      "@version" => "1",          "host" => "example.com",          "show" => "This data will be in the output",       "message" => "asdf"}

The "asdf" typed in became themessage field contents, and the conditional successfully evaluated the contents of thetest field nested within the@metadata field. But the output did not show a field called@metadata, or its contents.

Therubydebug codec allows you to reveal the contents of the@metadata field if you add a config flag,metadata => true:

stdout { codec => rubydebug { metadata => true } }

Let’s see what the output looks like with this change:

$ bin/logstash -f ../test.confPipeline main startedasdf{    "@timestamp" => 2016-06-30T02:46:48.565Z,     "@metadata" => {           "test" => "Hello",        "no_show" => "This data will not be in the output"    },      "@version" => "1",          "host" => "example.com",          "show" => "This data will be in the output",       "message" => "asdf"}

Now you can see the@metadata field and its sub-fields.

Make use of the@metadata field any time you need a temporary field but do not want it to be in the final output.

Perhaps one of the most common use cases for this new field is with thedate filter and having a temporary timestamp.

This configuration file has been simplified, but uses the timestamp format common to Apache and Nginx web servers. In the past, you’d have to delete the timestamp field yourself, after using it to overwrite the@timestamp field. With the@metadata field, this is no longer necessary:

input { stdin { } }filter {  grok { match => [ "message", "%{HTTPDATE:[@metadata][timestamp]}" ] }  date { match => [ "[@metadata][timestamp]", "dd/MMM/yyyy:HH:mm:ss Z" ] }}output {  stdout { codec => rubydebug }}

Notice that this configuration puts the extracted date into the[@metadata][timestamp] field in thegrok filter. Let’s feed this configuration a sample date string and see what comes out:

$ bin/logstash -f ../test.confPipeline main started02/Mar/2014:15:36:43 +0100{    "@timestamp" => 2014-03-02T14:36:43.000Z,      "@version" => "1",          "host" => "example.com",       "message" => "02/Mar/2014:15:36:43 +0100"}

That’s it! No extra fields in the output, and a cleaner config file because you do not have to delete a "timestamp" field after conversion in thedate filter.

Another use case is theCouchDB Changes input plugin. This plugin automatically captures CouchDB document field metadata into the@metadata field within the input plugin itself. When the events pass through to be indexed by Elasticsearch, the Elasticsearch output plugin allows you to specify theaction (delete, update, insert, etc.) and thedocument_id, like this:

output {  elasticsearch {    action => "%{[@metadata][action]}"    document_id => "%{[@metadata][_id]}"    hosts => ["example.com"]    index => "index_name"    protocol => "http"  }}

Sprintf date/time format in conditionals is not currently supported, but a workaround is available. Put the date calculation in a field so that you can use the field reference in a conditional.

Example

Using sprintf time format directly to add a field based on ingestion timewill not work:

----------# non-working examplefilter{  if "%{+HH}:%{+mm}" < "16:30" {    mutate {      add_field => { "string_compare" => "%{+HH}:%{+mm} is before 16:30" }    }  }}----------

This workaround gives you the intended results:

filter {  mutate{     add_field => {      "[@metadata][time]" => "%{+HH}:%{+mm}"     }  }  if [@metadata][time] < "16:30" {    mutate {      add_field => {        "string_compare" => "%{+HH}:%{+mm} is before 16:30"      }    }  }}