This document explains how you can find log entries that you routed fromCloud Logging toPub/Sub topics, which occursin near real-time.We recommend using Pub/Sub for integratingCloud Logging logs withthird-party software.

When you route log entries to a Pub/Sub topic,Loggingpublishes each log entry as a Pub/Sub message as soon asLogging receives that log entry. Routed log entries aregenerally available within seconds of their arrival to Logging,with 99% of log entries available in less than 60 seconds.

Before you begin

For a conceptual discussion of sinks, seeOverview of routing and storage models: Sinks.

For instructions on how to route your log entries, seeRoute logs to supported destinations.

View logs

To view your logs as they are streamed through Pub/Sub,do the following:

1. In the Google Cloud console, go to theTopics page:

   Go toTopics

   If you use the search bar to find this page, then select the result whose subheading isPub/Sub.

2. Find or create a subscription to the topic used in the log sink, and pull alog entry from it. You might have to wait for a new log entry to bepublished.

Logs organization

Thedata field of each message is a base64-encodedLogEntryobject.As an example, a Pub/Sub subscriber might pull the followingobject from a topic that is receiving log entries.The object shown contains a list with a single message, althoughPub/Sub might return several messages if several log entries areavailable.Thedata value (about 600 characters) and theackId value(about 200 characters) have been shortened to make the example easier to read:

{ "receivedMessages": [  {   "ackId": "dR1JHlAbEGEIBERNK0EPKVgUWQYyODM...QlVWBwY9HFELH3cOAjYYFlcGICIjIg",   "message": {    "data": "eyJtZXRhZGF0YSI6eyJzZXZ0eSI6Il...Dk0OTU2G9nIjoiaGVsbG93b3JsZC5sb2cifQ==",    "attributes": {     "compute.googleapis.com/resource_type": "instance",     "compute.googleapis.com/resource_id": "123456"    },    "messageId": "43913662360"   }  } ]}

If you decode thedata field and format it, you get the followingLogEntry object:

{  "log": "helloworld.log",  "insertId": "2015-04-15|11:41:00.577447-07|10.52.166.198|-1694494956",  "textPayload": "Wed Apr 15 20:40:51 CEST 2015 Hello, world!",  "timestamp": "2015-04-15T18:40:56Z",  "labels": {    "compute.googleapis.com\/resource_type": "instance",    "compute.googleapis.com\/resource_id": "123456"  },  "severity": "WARNING"  }}

Third-party integration with Pub/Sub

You route your log entries to a Pub/Sub topic. The thirdparty receives your log entries by subscribing to the same topic.Logging supports logging integration with third parties, suchas Splunk or Datadog. For a current list of integrations,seePartners for Google Cloud Observability integrations.

To perform the integration, expect to do something like the following:

1. In your project where your log entries originate, createyour Pub/Sub topic with adefault subscription:

   1. Enable the Pub/Sub API.

      Roles required to enable APIs

      To enable APIs, you need the Service Usage Admin IAM role (roles/serviceusage.serviceUsageAdmin), which contains theserviceusage.services.enable permission.Learn how to grant roles.

      Enable the API

   2. In the Google Cloud console, go to theTopics page:

      Go toTopics

      If you use the search bar to find this page, then select the result whose subheading isPub/Sub.

   3. ClickCreate topic.

   4. In theTopic ID field, enter an ID for your topic. For example,projects/my-project-id/topics/my-pubsub-topic.

      Each message sent to the topic include the timestamp of the routedlog entry in the Pub/Sub messageattributes; forexample:

      "attributes": {  "logging.googleapis.com/timestamp": "2024-07-01T00:00:00Z"}

   5. Retain the optionAdd a default subscription. Don't select any otheroption.

   6. ClickCreate topic.

2. In your project where your log entries originate, configureLogging to route log entries to your topic:

   1. In the Google Cloud console, go to theLog Router page:

      Go toLog Router

      If you use the search bar to find this page, then select the result whose subheading isLogging.

   2. ClickCreate Sink,enter a name and description for the sink, and then clickNext.
   3. In theSink Service menu, selectCloud Pub/Sub topic,select the Pub/Sub topic, and then clickNext.
   4. Select the log entries to include in the sink and then clickNext.
   5. Optional: Select the log entries to exclude.

   6. ClickCreate Sink.

      A dialog with the messageSink created appears. Thismessage indicates that your sink was successfully created withpermissions to route future matching log entries to the destinationyou selected.

   7. Grant the principal specified in the sink's writer identitythe following IAM roles:

      • Logs Writer (roles/logging.logWriter)

      • Pub/Sub Publisher (roles/pubsub.publisher)

      For more information about obtaining the writer identity and granting arole, seeSet destination permissions.

   Cloud Logging is now sending log entries to your Pub/Subtopic.

3. Create the subscription.

   For example, if you use Dataflow topull the data from your Pub/Sub topic and send it toDatadog, then you need to perform two steps:

   1. Create, or obtain, a service account, and then grant it theIAM roles necessary to subscribe to your topic. At aminimum, the service account requires the following roles:

      • Pub/Sub Subscriber (roles/pubsub.subscriber)
      • Dataflow Admin (roles/dataflow.admin)
      • Dataflow Worker (roles/dataflow.worker)

      For more information, see the following documents:

      • Create a service account.
      • Grant the service account the necessary roles.

   2. Create a job from a template, and then run that job.For this example, you would use thePub/Sub to Datadog template.

Your third party should begin receiving the log entries right away.

For an exploration of common logs routing scenarios usingPub/Sub, seeScenarios for exporting Cloud Logging data: Splunk.

Troubleshooting

If log entries seem to be missing from your sink's destination or you otherwisesuspect that your sink isn't properly routing log entries, then seeTroubleshoot routing logs.

Pricing

For pricing information, seeGoogle Cloud Observability pricing page. If you route log data toother Google Cloud services, then see the following documents:

• Cloud Storage pricing
• BigQuery pricing
• Pub/Sub pricing