This document discusses the concept of structured logging and the methods foradding structure to log entry payload fields. When the log payload is formattedas a JSON object and that object is stored in thejsonPayload field, the logentry is called astructured log. For these logs, you can construct queriesthat search specific JSON paths and you can index specific fields in thelog payload. In contrast, when the log payload is formatted as a string andstored in thetextPayload field, the log entry isunstructured.You can search the text field, but you can't index its content.

To create structured log entries, do any of the following:

• Call theentries.write API method and supplya fully formattedLogEntry.
• Use thegcloud logging write command.

• Use a Cloud Logging client library which writes structured logs.

• Use the BindPlane service.

• Use anagent to write logs:

  Note: We recommend that you use an agent to write logs,when that option is available. To write structured logs,configure your application to write serialized JSON objects.For a list of logging frameworks, seeRecommended logging frameworks.

  • Some Google Cloud services contain an integrated logging agent that sendsthe data written tostdout orstderr as logs to Cloud Logging.You can use this approach for Google Cloud services such as Google Kubernetes Engine,App Engine flexible environment, and Cloud Run functions.

  • For Compute Engine virtual machines (VMs), you can install and configurethe Ops Agent or the legacy Logging agent, and then usethe installed agent to send logs to Cloud Logging.

For more information about these approaches, see the following sections.

Write logs by using client libraries or the API

You can write log data by using theCloud Logging client libraries, whichcall the Cloud Logging API, or by directly calling the Cloud Logging API.Client libraries can simplify population of the special JSON fields byautomatically capturing some information and by providing interfaces toappropriately populate the fields. However, for full control over thestructure of your payloads, directly call the Cloud Logging API and pass thefullLogEntry structure to the Cloud Logging API.

For more information, see theentries.write reference.

For code examples, seeWriting structured logs.

Write logs by using the gcloud CLI

You can write log data by using the gcloud CLI. The interfacesupports unstructured logs and structured logs. When you want to writea structured log, provide the command a serialized JSON object.

For a quickstart, seeWrite and query log entries with the Google Cloud CLI.

For code examples, see thegcloud logging write reference.

Write logs by using BindPlane

You can use the BindPlane service to send logs to Logging.For these logs, the payloads are in JSONformat and are structured according to the source system. For information onfinding and viewing logs ingested by using BindPlane, see the BindPlaneQuickstart Guide.

Write logs by using an agent

To get logs from your Compute Engine instances,you can use theOps Agent orthelegacy Cloud Logging agent.Both agents can collect metrics from third-party applications, andboth provide support for structured logging:

• TheOps Agent is the recommended agentfor collectingtelemetry from your Compute Engine instances. This agent combineslogging and metrics into a single agent, provides a YAML-basedconfiguration, and features high-throughput logging.

  For information about how to configure the Ops Agent to supportstructured logging or to customize the form of a structured log, seeConfigure the Ops Agent.

• Thelegacy Cloud Logging agent collectslogs. This agent doesn't collect other forms of telemetry.

The remainder of this section is specific to thelegacy Logging agent.

Logging agent: special JSON fields

Some fields in the JSON object are recognized as special by thelegacy Logging agent and extracted into theLogEntry structure. These special JSON fields can be used to setthe following fields in theLogEntry:

• severity
• spanId
• labels defined by the user
• httpRequest

Because JSON is more precise and versatile than text lines, you can use JSONobjects to write multiline messages and add metadata.

To create structured log entries for your applications using the simplifiedformat, see the following table, which lists the fields and their values in JSON:

{"severity":"ERROR","message":"There was an error in the application.","httpRequest":{"requestMethod":"GET"},"time":"2020-10-12T07:20:50.52Z","logging.googleapis.com/insertId":"42","logging.googleapis.com/labels":{"user_label_1":"value_1","user_label_2":"value_2"},"logging.googleapis.com/operation":{"id":"get_data","producer":"github.com/MyProject/MyApplication","first":"true"},"logging.googleapis.com/sourceLocation":{"file":"get_data.py","line":"142","function":"getData"},"logging.googleapis.com/spanId":"000000000000004a","logging.googleapis.com/trace":"projects/my-projectid/traces/06796866738c859f2f19b7cfb3214824","logging.googleapis.com/trace_sampled":false}

{"insertId":"42","jsonPayload":{"message":"There was an error in the application","time":"2020-10-12T07:20:50.52Z"},"httpRequest":{"requestMethod":"GET"},"resource":{"type":"k8s_container","labels":{"container_name":"hello-app","pod_name":"helloworld-gke-6cfd6f4599-9wff8","project_id":"stackdriver-sandbox-92334288","namespace_name":"default","location":"us-west4","cluster_name":"helloworld-gke"}},"timestamp":"2020-10-12T07:20:50.52Z","severity":"ERROR","labels":{"user_label_2":"value_2","user_label_1":"value_1"},"logName":"projects/stackdriver-sandbox-92334288/logs/stdout","operation":{"id":"get_data","producer":"github.com/MyProject/MyApplication","first":true},"trace":"projects/my-projectid/traces/06796866738c859f2f19b7cfb3214824","sourceLocation":{"file":"get_data.py","line":"142","function":"getData"},"receiveTimestamp":"2020-10-12T07:20:57.52Z","spanId":"000000000000004a"}

The legacy Logging agent,google-fluentd, is aCloud Logging-specific packaging of theFluentd log data collector.The Logging agent comes with the default Fluentd configurationand uses Fluentd input plugins to pull event logs from external sourcessuch as files on disk, or to parse incoming log records.

Fluentd has a list ofsupported parsersthat extract logs and convert them into structured (JSON) payloads.

By configuring a log source withformat [PARSER_NAME], you can build on thebuilt-in parsers provided by Fluentd. For information about configuring thelegacy Logging agent, seeConfigure the Logging agent.

The following code samples show the Fluentd configuration, the input logrecord, and the output structured payload, which is part of a Cloud Logginglog entry:

• Fluentd configuration:

  <source>@typetailformatsyslog# This uses a predefined log format regex named# `syslog`. See details at https://docs.fluentd.org/parser/syslog.path/var/log/syslogpos_file/var/lib/google-fluentd/pos/syslog.posread_from_headtruetagsyslog</source>

• Log record (input):

  <6>Feb2812:00:00192.168.0.1fluentd[11111]:[error]Syslogtest

• Structured payload (output):

  jsonPayload:{"pri":"6","host":"192.168.0.1","ident":"fluentd","pid":"11111","message":"[error] Syslog test"}

For more information about how thesyslog parser works, see the detailedFluentd documentation.

Logging agent: standard parsers enabled by default

The following table includes the standard parsers that are included in theagent if you enable structured logging:

For instructions on enabling structured logging when installing thelegacy Logging agent, see theInstallation section.

Logging agent: installation

To enable structured logging, you must change the default configuration ofthe legacy Logging agent when installing or reinstalling it.Enabling structured logging replaces the previously listed configuration filesbut doesn't change the operation of the agent itself.

When you enable structured logging, the listed logs are converted tolog entries with different formats than they had before you enabled structuredlogs. If the logs are being routed to destinations outside ofLogging, the change could affect any post-processing applications.For example, if routing logs to BigQuery, BigQueryrejects the new log entries for the remainder of the day as having anincorrect schema.

For instructions on installing the legacy Logging agent andenabling structured logging, refer toInstalling the Loggingagent.

You can find the legacy Logging agent configuration files at/etc/google-fluentd/config.d/, which should now include theStandard parsers enabled by default.

Logging agent: configure Apache access log format

By default, the legacy Logging agent stores Apache accesslog data in thejsonPayload field. For example:

{"logName":...,"resource":...,"httpRequest":...,"jsonPayload":{"user":"some-user","method":"GET","code":200,"size":777,"host":"192.168.0.1","path":"/some-path","referer":"some-referer","agent":"Opera/12.0"},...}

Alternatively, you can configure the legacy Logging agent toextract certain fields to thehttpRequest field. For example:

{"logName":...,"resource":...,"httpRequest":{"requestMethod":"GET","requestUrl":"/some-path","requestSize":"777","status":"200","userAgent":"Opera/12.0","serverIp":"192.168.0.1","referrer":"some-referrer",},"jsonPayload":{"user":"some-user"},...}

Configuring thehttpRequest field, as shown in the prior sample, assiststracing: the Google Cloud console presents all logs for a given HTTP requestin a parent-child hierarchy.

To configure this extraction, add the following to the end of your/etc/google-fluentd/config.d/apache.conf:

<filterapache-access>@typerecord_transformerenable_rubytrue<record>httpRequest${ {"requestMethod" => record['method'],"requestUrl" => record['path'],"requestSize" => record['size'],"status" => record['code'],"userAgent" => record['agent'],"serverIp" => record['host'],"referer" => record['referer']}}</record>remove_keysmethod,path,size,code,agent,host,referer</filter>

For more details on how to configure your log entries, seeModifying log records.

Logging agent: configure nginx access log format

By default, the legacy Logging agent stores nginx accesslog data in thejsonPayload field. For example:

{"logName":...,"resource":...,"httpRequest":...,"jsonPayload":{"remote":"127.0.0.1","host":"192.168.0.1","user":"some-user","method":"GET","path":"/some-path","code":"200","size":"777","referrer":"some-referrer","agent":"Opera/12.0","http_x_forwarded_for":"192.168.3.3"},...}

Alternatively, you can configure the legacy Logging agent toextract certain fields to thehttpRequest field. For example:

{"logName":...,"resource":...,"httpRequest":{"requestMethod":"GET","requestUrl":"/some-path","requestSize":"777","status":"200","userAgent":"Opera/12.0","remoteIp":"127.0.0.1","serverIp":"192.168.0.1","referrer":"some-referrer",},"jsonPayload":{"user":"some-user","http_x_forwarded_for":"192.168.3.3"},...}

Configuring thehttpRequest field, as shown in the prior sample, assiststracing: the Google Cloud console presents all logs for a given HTTP requestin a parent-child hierarchy.

To configure this extraction, add the following to the end of your/etc/google-fluentd/config.d/nginx.conf:

<filternginx-access>@typerecord_transformerenable_rubytrue<record>httpRequest${ {"requestMethod" => record['method'],"requestUrl" => record['path'],"requestSize" => record['size'],"status" => record['code'],"userAgent" => record['agent'],"remoteIp" => record['remote'],"serverIp" => record['host'],"referer" => record['referer']}}</record>remove_keysmethod,path,size,code,agent,remote,host,referer</filter>

For more details on how to configure your log entries, seeModifying log records.

Write your own parser

If your logs aren't supported by the standard parsers, you can write your ownparser. Parsers consist of a regular expression that is used to match logrecords and apply labels to the pieces.

The following code examples show a log line in the log record, a configurationwith a regular expression that indicates the format of the log line, and thestored log entry:

• A log line in the log record:

  REPAIRCAR$500

• A configuration with a regular expression that indicates the format ofthe log line:

  $sudovim/etc/google-fluentd/config.d/test-structured-log.conf$cat/etc/google-fluentd/config.d/test-structured-log.conf<source>@typetail# Format indicates the log should be translated from text to# structured (JSON) with three fields, "action", "thing" and "cost",# using the following regex:format/(?<action>\w+)(?<thing>\w+)\$(?<cost>\d+)/# The path of the log file.path/tmp/test-structured-log.log# The path of the position file that records where in the log file# we have processed already. This is useful when the agent# restarts.pos_file/var/lib/google-fluentd/pos/test-structured-log.posread_from_headtrue# The log tag for this log input.tagstructured-log</source>

• The resulting log entry:

  {insertId:"eps2n7g1hq99qp"jsonPayload:{"action":"REPAIR""thing":"CAR""cost":"500"}labels:{compute.googleapis.com/resource_name:"add-structured-log-resource"}logName:"projects/my-sample-project-12345/logs/structured-log"receiveTimestamp:"2023-03-21T01:47:11.475065313Z"resource:{labels:{instance_id:"3914079432219560274"project_id:"my-sample-project-12345"zone:"us-central1-c"}type:"gce_instance"}timestamp:"2023-03-21T01:47:05.051902169Z"}

Troubleshoot issues

To troubleshoot common issues found with installing or interacting with thelegacy Logging agent, seeTroubleshooting the agent.

What's next

• To query and view log entries, seeView logs by using the Logs Explorer.

• To read log entries using the Google Cloud CLI, seeReading log entries.

• To read log entries using the Logging API, see theentries.list method.