This document explains how you can find log entries that you routed fromCloud Logging toCloud Storage buckets.

Log entries are saved to Cloud Storage buckets in hourly batches. It mighttake from 2 to3 hours before the first entries begin toappear.

Before you begin

For a conceptual discussion of sinks, seeOverview of routing and storage models: Sinks.

For instructions on how to route your logs, seeRoute logs to supported destinations.

View logs

To view your logs routed to Cloud Storage, do the following:

1. In the Google Cloud console, go to theBuckets page:

   Go toBuckets

   If you use the search bar to find this page, then select the result whose subheading isCloud Storage.

2. Select the Cloud Storage bucket you are using as your routingdestination.

Logs organization

When you route logs to a Cloud Storage bucket, Loggingwrites a set of files to the bucket.

The files are organized in directory hierarchies by log type and date. The logtype, referred to as[LOG_ID] in theLogEntry reference,can be a simple name likesyslog or a compound name likeappengine.googleapis.com/request_log. If these logs were stored in a bucketnamedmy-gcs-bucket,then the directories would be named as in the following example:

my-gcs-bucket/syslog/YYYY/MM/DD/my-gcs-bucket/appengine.googleapis.com/request_log/YYYY/MM/DD/

A single Cloud Storage bucket can contain logs from multiple resource types.The maximum file size is 3.5 GiB.

Logging doesn't guarantee deduplication of log entries fromsinks containing identical or overlapping queries; log entries from those sinksmight be written multiple times to a Cloud Storage bucket.

The leaf directories (DD/) contain multiple files, each of which holds therouted log entries for a time period specified in the file name. The filesaresharded and their names end in a shard number,Sn orAn (n=0, 1, 2,...). For example, here are two files that might be stored within the directorymy-gcs-bucket/syslog/2015/01/13/:

08:00:00_08:59:59_S0.json08:00:00_08:59:59_S1.json

These two files together contain thesysloglog entries for all instances during the hour beginning 08:00:00 UTC and ending08:59:59 UTC. The log entrytimestamps are expressed in UTC (Coordinated Universal Time).

Log entries that arrive with areceiveTimestampwithin the 60-minute aligned window of theirtimestamp are written tomain shard files. For example, a log entry with atimestamp of 08:00:00and areceiveTimestamp of 08:10:00 is stored in the main shard file.

These files include a numbered main shard in the suffix:_Sn.json.

Log entries that arrive with atimestamp in a different 60-minutealigned window than theirreceiveTimestamp are written toaddendum shardfiles. For example, A log entry with atimestamp of 08:00:00 andareceiveTimestamp of 09:10:00 is stored in an addendum shard file.

These files include a numbered addendum shard withthe suffix:_An:Unix_timestamp.json.

For example, a log entry that has atimestamp between 08:00:00 and08:59:59 but areceiveTimestamp in a different 60-minute aligned windowis written to a file with the_An:Unix_timestamp.jsonsuffix, where the Unix timestamp identifies the time the file was routed toCloud Storage. If a log entry had atimestamp of08:50:00 and areceiveTimestamp of 09:10:00, and was routed at09:15:00 on March 25, 2021, the addendum file would be written as follows:

08:00:00_08:59:59_A0:1616681700.json

To get all the log entries, you must read all the shards for each timeperiod—in this case, file shards 0 and 1. The number of file shardswritten can change for each time period.

Within the individual sharded files, log entries are stored as a listofLogEntry objects. For an example of asyslog entry, seeLog entries organization.

Note that sort order of log entries within the files is neither uniform norotherwise guaranteed.

Late-arriving log entries

Routed log entries are saved to Cloud Storage buckets in hourly batches.It might take from 2 to3 hours before the first entries begin toappear. Routed log file shards with the suffixAn("Append") hold log entries that arrived late.

If the destination experiences an outage, then Cloud Logging buffersthe data until the outage is over.

If there aren't any logs in your sink's destination, check theexport system metrics.The export system metrics indicate how many log entries are routed and howmany are dropped due to errors. If the export system metrics indicate that nolog entries were routed to the destination, check your [filter][export-query] toverify that log entries matching your filter have recently arrived inLogging.

In the Google Cloud console, go to theLog Router page:

Go toLog Router

If you use the search bar to find this page, then select the result whose subheading isLogging.

App Engine log entries

App Engine combines multiple sub-entries of typegoogle.appengine.logging.v1.LogLine (also called AppLog orAppLogLine) under a primary log entry of typegoogle.appengine.logging.v1.RequestLog for the requestthat causes the log activity. The log lines each have a "request ID" thatidentifies the primary entry. The Logs Explorer displays the log lines withthe request log entry. Logging attempts to put all the loglines into the batch with the original request, even if their timestamps wouldplace them in the next batch. If that isn't possible, the request log entrymight be missing some log lines, and there might be "orphan" log lines without arequest in the next batch. If this possibility is important to you, be preparedto reconnect the pieces of the request when you process your logs.

Troubleshooting

If logs seem to be missing from your sink's destination or you otherwise suspectthat your sink isn't properly routing logs, then seeTroubleshoot routing logs.

Pricing

For pricing information, seeGoogle Cloud Observability pricing page. If you route log data toother Google Cloud services, then see the following documents:

• Cloud Storage pricing
• BigQuery pricing
• Pub/Sub pricing