This page describes how to manually copy log entries that are already storedin Cloud Logging log buckets to Cloud Storage buckets. A copy operationdoesn't require that the log bucket and Cloud Storage bucket be in thesame Google Cloud project.

You might want to copy log entries from log buckets to Cloud Storagebuckets for the following reasons:

• In case you forgot to route log entries to Cloud Storage prior totheir storage in Logging.
• To share log entries with auditors outside of Logging.
• To analyze log entries with scripts in Cloud Storage.

When you copy log entries to Cloud Storage, the log entries also remainin the log bucket they were copied from.

Copy operations don't replacesinks, which automatically send allincoming log entries that match inclusion and exclusion filters to apre-selected supported storage destination, including Cloud Storagebuckets. You can't use sinks to retroactively route logs. Use sinks when youknow in advance that you want to store log entries in a Cloud Storagebucket.

To copy logs and then manage and monitor the operation, you must use theGoogle Cloud CLI.

Limitations

The following limitations apply when copying log entries:

• You can copy logs to Cloud Storage buckets only; other destinations aren'tavailable.

• You can't copy logs from log buckets that haveCMEK configured.

Before you begin

Before you get started with copying logs, do the following:

• Verify that you're using the latest version of thegcloud CLI. For instructions about how to installor update the gcloud CLI, seeManaging Google Cloud CLI components.

• To get the permissions that you need to copy log entries, ask your administrator to grant you the following IAM roles on the project that contains the log bucket you are copying from:

  • To run a copy operation:Logging Admin (roles/logging.admin)
  • To view and manage the status of a copy operation:Logs Configuration Writer (roles/logging.configWriter)

• To get the permissions that you need to write log entries to Cloud Storage, ask your administrator to grant you theStorage Object Creator (roles/storage.objectCreator) IAM role on the project that contains the Cloud Storage bucket.

Copy log entries

Logging only copies log entries that are stored in the log bucketwhen the copy operation starts. Log entries that are stored in log bucketsafter the copy operation starts don't get copied to Cloud Storage.

To copy log entries to Cloud Storage, you need to know the followinginformation:

• The ID and location of the log bucket you're copying from.To retrieve the log bucket ID and its location, use thegcloud CLI commandgcloud logging buckets list.
• The name of the Cloud Storage bucket you're copying to. For informationabout how to retrieve the Cloud Storage bucket name, seeGetting Cloud Storage bucket information.
• Afilter for the log entries youwant to copy.

To copy log entries, run thegcloud logging copy command:

gcloudloggingcopyBUCKET_IDstorage.googleapis.com/CLOUD_STORAGE_BUCKET_NAME\--location=LOCATION--log-filter='FILTER'--project=PROJECT_ID

Before you run the previous command, do the following:

• ReplaceBUCKET_ID with the name of your log bucket.
• ReplaceCLOUD_STORAGE_BUCKET_NAME with the name of yourCloud Storage bucket.
• ReplaceLOCATION with the location of the log bucket.

• (Optional): ReplaceFILTER with the filter that defines which logs arecopied.

  If you omit the--log-filter flag, then all log entries in thelog bucket are copied to the Cloud Storage bucket.

  Note: The copy command cannot copy log entries whose retention has expired.

• ReplacePROJECT_ID with your Google Cloud project ID. You can omit this flagwhen the active gcloud CLIconfiguration is set to the correct Google Cloud project.

Example command:

gcloud logging copy my-log-bucket storage.googleapis.com/my-gcs-bucket \--location=global --log-filter='timestamp > "2024-07-18T10:00:00.0Z"' \--project=my-project

This command creates along-running operationto run in the background and returns the name of the copy operation and thelocation of the log bucket:

name: projects/PROJECT_ID/locations/LOCATION/operations/OPERATION_ID

The location of the copy operation is the same as the location of the logbucket that you are copying from.

View and manage copy operations

You can view and manage your copy operations by using thegcloud logging operationscommands, which let you list, view and cancel operations.

The following commands require that you specify the location of the operation.Use the location of your log bucket. For information about how to find thelocation of your log bucket, seeView a bucket's details.

List copy operations

You can list recent copy operations, including scheduled, running, completed,failed, and cancelled operations. Recent copy operations appear in the resultsfor up to 30 days after the end time.

To list copy operations, run the following command:

gcloudloggingoperationslist--location=LOCATION\--operation-filter=request_type=CopyLogEntries\--project=PROJECT_ID

Before you run the previous command, do the following:

• ReplaceLOCATION with the location of the log bucket that you are copyingfrom.
• ReplacePROJECT_ID with your Google Cloud project ID.

The command returns information about the long-running operation, includingthe operation ID:

projects/PROJECT_ID/locations/LOCATION/operations/OPERATION_ID

View the status of a copy operation

You can retrieve the status and other metadata about copy operations,including the following:

• startTime: The timestamp indicating the creation of the operation.
• endTime: The timestamp indicating the completion of the operation.
• state: The status of the operation (scheduled, running, cancelled, failed,or succeeded).
• cancellation: Whether the user has requested to cancel the operation.
• progress: Estimated progress of the operation (0-100%).
• destination: The name of the Cloud Storage bucket to which theoperation is copying logs.
• filter: The filter specifying which log entries to copy.
• name: The name of the log bucket from which the operation is copying logs.
• logEntriesCopiedCount: The number of log entries successfully copied tothe Cloud Storage bucket by the operation.

Note that not all of the listed metadata fields apply to every copy operation.For example, if a copy operation is still running, theendtime metadatawouldn't apply to the operation. As another example, if the--log-filter=FILTER flag wasn't used when running thegcloud logging copy command, then thefilter metadata wouldn't applyto the operation.

To retrieve information about a copy operation, run the following command:

gcloudloggingoperationsdescribeOPERATION_ID\--location=LOCATION--project=PROJECT_ID

Before you run the previous command, do the following:

• ReplaceOPERATION_ID with the ID of the operation.
• ReplaceLOCATION with the location of the log bucket that you are copyingfrom.
• ReplacePROJECT_ID with your Google Cloud project ID.

The command returns metadata about the copy operation. For example, here isan output for an operation that is in progress:

done: falsemetadata:  `@type`: type.googleapis.com/google.logging.v2.CopyLogEntriesMetadata  progress: 75  destination: storage.googleapis.com/my-storage-bucket-1  source: projects/my-test-project/locations/us-central1/buckets/my-logging-bucket-2  verb: copy  startTime: `2024-05-23T10:52:40.039751Z`  state: OPERATION_STATE_RUNNINGname: projects/my-test-project/locations/us-central1/buckets/my-logging-bucket-2   </pre>

Cancel a copy operation

You can cancel an in-progress copy operation. If you cancel a copy operation,all log entries that were copied before the operation was cancelled remainin the Cloud Storage bucket.

After cancelling a copy operation, Logging completes all ongoingprocesses before it completes the cancellation. This might result in some logentries still being copied to Cloud Storage after you cancel theoperation.

To cancel a copy operation, run the following command:

gcloudloggingoperationscancelOPERATION_ID\--location=LOCATION--project=PROJECT_ID

Before you run the previous command, do the following:

• ReplaceOPERATION_ID with the ID of the operation.
• ReplaceLOCATION with the location of the log bucket that you are copyingfrom.
• ReplacePROJECT_ID with your Google Cloud project ID.

View logs in Cloud Storage

To view and understand the logs that you copied to Cloud Storage, seeView logs routed to Cloud Storage.

Quotas and limits

All copy operations take at least an hour to complete, no matter the amount ofdata that is being copied.

To copy a large volume—for example, petabytes—split the copyingacross multiple copy operations by using thetimestamp field in the--filter flag.

The copy command cannot copy log entries whose retention has expired.

Pricing

For pricing information, seeGoogle Cloud Observability pricing page. If you route log data toother Google Cloud services, then see the following documents:

• Cloud Storage pricing
• BigQuery pricing
• Pub/Sub pricing