File system transfer logs
This document describes the current file system transfer log format.
Note: The log format described on this page is subject to change. We recommendthat you avoid writing scripts, programs, or design processes that depend on theordering, naming, or presence of columns or values in this log. Refer to thispage andStorage Transfer Service release notes for updates tothe log format.View transfer logs
File system transfers produce detailed transfer logs that you can use to verifythe results of your transfer job. Each job produces a collection of transferlogs that are stored in the transer's Cloud Storage bucket: the sourcebucket, destination bucket, or intermediate bucket, depending on the transferdirection.
Logs are produced while the transfer job is running. The complete logs aretypically available within 15 minutes of job completion.
Important: Anyone that has access to the Cloud Storage bucket hasaccess to the transfer logs.View logs in the Cloud Storage bucket
Transfer logs are stored in the transfer's Cloud Storage bucket at thefollowing path:
bucket-name/storage-transfer/logs/transferJobs/job-name/transferOperations/operation-name
where the path components are as follows:
bucket-nameis the name of the Cloud Storagebucket involved in this transfer.job-nameis the job name, as displayed in thejob list.operation-nameis the name of the individualtransfer operation, comprised as the IS08601 timestamp and generated ID.
Logs are aggregated and stored as objects. Each batch of logs is named by itscreation time. For example:
mybucket/storage-transfer/logs/transferOperations/job1/2019-10-19T10_52_56.519081644-07_00.log
Run BigQuery queries on transfer logs
To run BigQuery queries on your transfer logs:
Example queries
Display the number of files that attempted transfer and whether they failedor succeeded
select ActionStatus, count(*) as num_filesfrombig-query-tablewhere Action="TRANSFER"group by 1;
Wherebig-query-table is the name of theBigQuery table that contains the transfer log.
Display all files that failed to transfer
select Src_File_Pathfrombig-query-tablewhere Action="TRANSFER" and ActionStatus="FAILED";
Wherebig-query-table is the name of theBigQuery table that contains the transfer log.
Display checksum and timestamp for each file that successfully transferred
select Timestamp, Action, ActionStatus, Src_File_Path, Src_File_Size,Src_File_Crc32C, Dst_Gcs_BucketName, Dst_Gcs_ObjectName, Dst_Gcs_Size,Dst_Gcs_Crc32C, Dst_Gcs_Md5frombig-query-tablewhere Action="TRANSFER" and ActionStatus="SUCCEEDED";
Wherebig-query-table is the name of theBigQuery table that contains the transfer log.
Display all error information for directories that failed to transfer
select FailureDetails_ErrorType, FailureDetails_GrpcCode, FailureDetails_Messagefrombig-query-tablewhere Action="FIND" and ActionStatus="FAILED";
Wherebig-query-table is the name of theBigQuery table that contains the transfer log.
Format description
File system transfer logs are saved in tab-separated values (TSV) format, andcontain a header row with field names. The order that the fields are displayedare generally stable, but aren't guaranteed, and must be inferred from theheader row.
Each line represents a single log record, that is a status update on part of theoverall transfer operation.
Note: Storage Transfer Service doesn't record log entries for skipped files.The following table describes the fields in the log file:
| Log field | Description |
|---|---|
| Timestamp | ISO 8601-compliant timestamp when we recorded the event. |
| Operation Name | The fully-qualified operation name. |
| Action | Describes the action of this particular task. One of the following:
|
| ActionStatus | The high-level status of the action. One of the following:
|
| FailureDetails.ErrorType | A string that represents the type of error encountered. For example,FILE_NOT_FOUND. Populated only if an error was encountered. |
| FailureDetails.GrpcCode | A string value of the Google RPC code. For example,FAILED_PRECONDITION. Populated only if an error was encountered. |
| FailureDetails.Message | A human-readable error message for the failure. Populated only if an error was encountered. |
| Src.Type | String that describes the source's storage system type. This is alwaysON_PREM. |
| Src.File.Path | Path of the transferred on-premises file. |
| Src.File.LastModified | The POSIX modified time (mtime) of the on-premises file. |
| Src.FileSize | The file size in bytes. |
| Src.File.Crc32C | The CRC32C checksum of the file's contents. |
| Dst.Type | String that describes the destination storage system type. This is alwaysGCS. |
| Dst.Gcs.BucketName | The destination Cloud Storage bucket name. |
| Dst.Gcs.ObjectName | The object prefix for the destination object. |
| Dst.Gcs.LastModified | The POSIX modified time (mtime) of the destination object. |
| Dst.Gcs.Size | The object size in bytes. |
| Dst.Gcs.Crc23C | The CRC32C checksum of the object's contents. |
| Dst.Md5 | The MD5 checksum of the final object in Cloud Storage. |
Example log entry
The following is an example of two lines of log output, the header and a dataline:
Timestamp OperationName Action ActionStatus FailureDetails.ErrorType FailureDetails.GrpcCode FailureDetails.Message Src.Type Src.File.Path Src.File.LastModified Src.File.Size Src.File.Crc32C Dst.Type Dst.Gcs.BucketName Dst.Gcs.ObjectName Dst.Gcs.LastModified Dst.Gcs.Size Dst.Gcs.Crc32C Dst.Gcs.Md52019-10-18T21:06:17Z transferOperations/agent-manual_transferJobs/OPI1494275376193533620_0000000001571432448 TRANSFER SUCCEEDED ON_PREM_FILE /mnt/disks/disk-1/directory1/file.txt 1561409931 406 3089075447 GCS_OBJECT destination-gcs-bucket file.txt 1571432776 406 3089075447 FDjya7dWbd0OrgnZ7g9ZJw==
Except as otherwise noted, the content of this page is licensed under theCreative Commons Attribution 4.0 License, and code samples are licensed under theApache 2.0 License. For details, see theGoogle Developers Site Policies. Java is a registered trademark of Oracle and/or its affiliates.
Last updated 2026-02-19 UTC.