This document describes how to create a Cloud Storage subscription.You can use the Google Cloud console, the Google Cloud CLI, the client library,or the Pub/Sub API to create a Cloud Storage subscription.

Before you begin

Before reading this document, ensure that you're familiar with the following:

• How aCloud Storage subscription works.
• HowCloud Storage works and how to create andmanageCloud Storage buckets.
• How to configure adead letter topic to handle message failures.

Required roles and permissions

The following is a list of guidelines regarding roles and permissions:

• To create a subscription, you must configure access control at the projectlevel.

• You also need resource-level permissions if your subscriptions and topicsare in different projects, as discussed later in this section.

• To create a Cloud Storage subscription, either thePub/Sub service agent or a custom service account must havepermission to write to the specific Cloud Storage bucket and to read thebucket metadata. For more information about how to grant these permissions, seethe next section of this document.

To get the permissions that you need to create Cloud Storage subscriptions, ask your administrator to grant you thePub/Sub Editor (roles/pubsub.editor) IAM role on the project. For more information about granting roles, seeManage access to projects, folders, and organizations.

This predefined role contains the permissions required to create Cloud Storage subscriptions. To see the exact permissions that are required, expand theRequired permissions section:

You might also be able to get these permissions withcustom roles or otherpredefined roles.

To let a principal in one project create a Cloud Storage subscription inanother project, you must grant that principal the Pub/Sub Editor(roles/pubsub.editor) role in both projects. This provides the necessarypermissions to create the new Google Cloud subscription and attachit to the original topic. The Pub/Sub Editor (roles/pubsub.editor) role on thetopic also helps you attach Google Cloud subscriptions in a differentproject to the topic.

Assign roles to service accounts

Some Google Cloud services have Google Cloud-managed service accounts that let theservices access your resources. These service accounts areknown as service agents. Pub/Sub creates and maintains aservice agent for each project in the formatservice-project-number@gcp-sa-pubsub.iam.gserviceaccount.com.

You can choose between letting the Pub/Sub service agent or a custom service account permission to write to the Cloud Storage bucket.

Granting permission to the Pub/Sub service agent means that any user who has permission to create a subscription in your project can write to the Cloud Storage bucket. If you want to provide more granular permission for writing to the Cloud Storage bucket, configure a custom service account instead.

For more information about Cloud Storage IAM,seeCloud Storage Identity and Access Management.

Assign Cloud Storage roles to the Pub/Sub service agent

If you want to create a Cloud Storage subscription using thePub/Sub service agent, then it must have permission to writeto the specific Cloud Storage bucket and to read the bucket metadata.

Grant the Storage Object Creator (roles/storage.objectCreator) and StorageLegacy Bucket Reader (roles/storage.legacyBucketReader) roles to thePub/Sub service agent. Grant the permission on the individualbucket.

Assign Cloud Storage roles to a custom service account

If you want to use a custom service account for writing to aCloud Storage bucket, then you must set the following permissions:

• The custom service account must have permission to write to the specificCloud Storage bucket and to read the bucket metadata.
• The Pub/Sub service agent must haveiam.serviceAccounts.getAccessToken permission on the custom service account.
• The user creating the subscription must haveiam.serviceAccounts.actAspermission on the custom service account.

Create the service account and grant permissions with the following steps:

1. Create the custom service account. Theservice account must be in the same project as the subscription.

2. Grant the Storage Object Creator (roles/storage.objectCreator) and StorageLegacy Bucket Reader (roles/storage.legacyBucketReader) roles to the customservice account.

   You can grant the service account permission on a single table inthe project or onall tables in the project. To do so, see the appropriate section inAssign Google Cloud roles to the Pub/Sub service agent. In the procedure, replace the Pub/Sub service agent email address with the custom service account email address.

3. Give the Pub/Sub service agenttheiam.serviceAccounts.getAccessToken permission on the custom serviceaccount or onall service accounts in the project.You can grant this permission by giving theroles/iam.serviceAccountTokenCreatorrole to the Pub/Sub service agent.

   Choose the appropriate method based on your requirements.

If you created the custom service account, you should already have the necessaryiam.serviceAccounts.actAs permission. If you need to grant someone else thepermission on the service account:

1. In the Google Cloud console, go to theService accounts page.

   Go to Service accounts

2. Enter the name of the custom service account in theFilter.

3. Select the service account from the list.

4. ClickPrincipals with access.

5. ClickGrant access.

6. In theAdd principals section, enter the name the account to which youwant to grant access.

7. In theSelect a role drop-down, enterService Account, and select theService Account User role.

8. ClickSave.

Cloud Storage subscription properties

When you configure a Cloud Storage subscription, you must specifythe properties common to all subscription types and some additionalCloud Storage subscription-specific properties.

Common subscription properties

Learn about thecommon subscription propertiesthat you can set across all subscriptions.

Bucket name

A Cloud Storage bucket must already exist before you create aCloud Storage subscription.

The messages are sent as batches and stored in the Cloud Storage bucket.A single batch or file is stored as anobjectin the bucket.

The Cloud Storage bucket must haveRequester Pays disabled.

To create a Cloud Storage bucket, seeCreate buckets.

Filename prefix, suffix, and datetime

The output Cloud Storage files generated by the Cloud Storagesubscription are stored as objects in the Cloud Storage bucket. The nameof the object stored in the Cloud Storage bucket is of the followingformat:<file-prefix><UTC-date-time>_<uuid><file-suffix>.

The following list includes details of the file format and the fields that youcan customize:

• <file-prefix> is the custom filename prefix. This is an optional field.

• <UTC-date-time> is a customizable auto-generated string based on the timethe object is created.

• <uuid> is an auto-generated random string for the object.

• <file-suffix> is the custom filename suffix. This is an optional field. Thefilename suffix cannot end in "/".

• You can change the filename prefix and suffix:

  • For example, if the value of the filename prefix isprod_ and the value ofthe filename suffix is_archive, a sample object name isprod_2023-09-25T04:10:00+00:00_uN1QuE_archive.

  • If you don't specify the filename prefix and suffix, the object name storedin the Cloud Storage bucket is of the format:<UTC-date-time>_<uuid>.

  • Cloud Storage object naming requirements also apply to the filenameprefix and suffix. For more information, seeAbout Cloud Storage objects.

• You can change how the date and time are displayed in the filename:

  • Required datetime matchers that you can use only once: year (YYYY orYY), month (MM), day (DD), hour (hh), minute (mm), second (ss).For example,YY-YYYY orMMM is invalid.

  • Optional matchers that you can use only once: datetime separator (T) andand timezone offset (Z or+00:00).

  • Optional elements that you can use multiple times: hyphen (-),underscore (_), colon (:), and forward slash (/).

  • For example, if the value of the filename datetime format isYYYY-MM-DD/hh_mm_ssZ, a sample object name isprod_2023-09-25/04_10_00Z_uNiQuE_archive.

  • If the filename datetime format ends in a character which is not a matcher,that character will replace the separator between<UTC-date-time> and<uuid>. For example, if the value of the filename datetime format isYYYY-MM-DDThh_mm_ss-, a sample object name isprod_2023-09-25T04_10_00-uNiQuE_archive.

File batching

Cloud Storage subscriptions let you decide when you want to createa new output file that is stored as an object in the Cloud Storagebucket. Pub/Sub writes an output file when one of thespecified batching conditions are met. The following are theCloud Storage batching conditions:

• Storage batch max duration. This is a required setting.Pub/Sub writes a new output file if the specified value for maxduration is exceeded. The duration is measured from when Pub/Subbegins writing to a new file to the time the file is finalized. For example,if you set the max duration to 5 minutes, Pub/Sub finalizes thefile at most 5 minutes after Pub/Sub began writing to the file.A new file can be created before the max duration has passed. If you don'tspecify the value, a default value of 5 minutes is applied. The following arethe applicable values for max duration:

  • Minimum value = 1 minute
  • Default value = 5 minutes
  • Maximum value = 10 minutes

• Storage batch max bytes. This is an optional setting. TheCloud Storage subscription writes a new output file if thespecified value of max bytes is exceeded. The following are the applicablevalues for max bytes:

  • Minimum value = 1 KB
  • Maximum value = 10 GiB

• Storage batch max messages. This is an optional setting. TheCloud Storage subscription writes a new output file if thespecified number of max messages is exceeded. The following are the applicablevalues for max messages:

  • Minimum value = 1000

For example, you can configure max duration as 6 minutes andmax bytes as 2 GB. If at the 4th minute, the output file reaches afile size of 2 GB, Pub/Sub finalizes the previous file and startswriting to a new file.

A Cloud Storage subscription might write to multiple files in aCloud Storage bucket simultaneously. If you have configured yoursubscription to create a new file every 6th minute, you might observemultiple Cloud Storage files being created every 6 minutes.

In some situations, Pub/Sub might start writing to a newfile earlier than the time configured by the file batching conditions.A file might also exceed the Max bytes value if the subscriptionreceives messages larger than the Max bytes value.

File format

When you create a Cloud Storage subscription, you can specifythe format of the output files that are to be stored in aCloud Storage bucket asText orAvro.

• Text: The messages are stored as plain text. A newline characterseparates a message from the previous message in the file. Only messagepayloads are stored, not attributes or other metadata.

• Avro: The messages are stored inApache Avro binary format. When you selectAvro, you can enable the following additional properties:

  • Write metadata: This option lets you store the message metadata along with the message. Metadata such assubscription_name,message_id,publish_time, andattributes fields are written to top-level fields in the output Avro object while all other message properties other than data (for example, an ordering_key, if present) are added as entries in theattributes map.

    Ifwrite metadata is disabled, only the message payload is written to the output Avro object. Here is the Avro schema for the output messages withwrite metadata disabled:

    {"type":"record","namespace":"com.google.pubsub","name":"PubsubMessage","fields":[{"name":"data","type":"bytes"}]}

    Here is the Avro schema for the output messages withwrite metadata enabled:

    {"type":"record","namespace":"com.google.pubsub","name":"PubsubMessageWithMetadata","fields":[{"name":"subscription_name","type":"string"},{"name":"message_id","type":"string"},{"name":"publish_time","type":{"type":"long","logicalType":"timestamp-micros"}},{"name":"attributes","type":{"type":"map","values":"string"}},{"name":"data","type":"bytes"}]}

  • Use topic schema: This option lets Pub/Sub use theschema of the Pub/Sub topic to whichthe subscription is attached when writing Avro files.

    When you use this option, remember to check the following additional requirements:

    • The topic schema must be inApache Avro format.

    • If bothuse topic schema andwrite metadata are enabled, the topic schema must have aRecord object at its root. Pub/Sub will expand the Record's list of fields to include the metadata fields. As a result, the Record cannot contain any fields with the same name as the metadata fields (subscription_name,message_id,publish_time, orattributes).

Service account

You have the following options to write messages to a BigQuery table or Cloud Storage bucket:

• Configure acustom service account so that only users who have theiam.serviceAccounts.actAs permission on the service account can create a subscription that writes to the table or bucket. An example role that includes theiam.serviceAccounts.actAs permission is theService Account User (roles/iam.serviceAccountUser) role.

• Use the default Pub/Sub service agent that lets any user with the ability to create subscriptions in the project to create a subscription that writes to the table or bucket. The Pub/Sub service agent is the default setting when you don't specify a custom service account.

Create a Cloud Storage subscription

namespacepubsub=::google::cloud::pubsub;namespacepubsub_admin=::google::cloud::pubsub_admin;[](pubsub_admin::SubscriptionAdminClientclient,std::stringconst&project_id,std::stringconst&topic_id,std::stringconst&subscription_id,std::stringconst&bucket){google::pubsub::v1::Subscriptionrequest;request.set_name(pubsub::Subscription(project_id,subscription_id).FullName());request.set_topic(pubsub::Topic(project_id,topic_id).FullName());request.mutable_cloud_storage_config()->set_bucket(bucket);autosub=client.CreateSubscription(request);if(!sub){if(sub.status().code()==google::cloud::StatusCode::kAlreadyExists){std::cout <<"The subscription already exists\n";return;}throwstd::move(sub).status();}std::cout <<"The subscription was successfully created: "            <<sub->DebugString() <<"\n";}

usingGoogle.Cloud.PubSub.V1;usingGoogle.Protobuf.WellKnownTypes;usingSystem;publicclassCreateCloudStorageSubscriptionSample{publicSubscriptionCreateCloudStorageSubscription(stringprojectId,stringtopicId,stringsubscriptionId,stringbucket,stringfilenamePrefix,stringfilenameSuffix,TimeSpanmaxDuration){SubscriberServiceApiClientsubscriber=SubscriberServiceApiClient.Create();TopicNametopicName=TopicName.FromProjectTopic(projectId,topicId);SubscriptionNamesubscriptionName=SubscriptionName.FromProjectSubscription(projectId,subscriptionId);varsubscriptionRequest=newSubscription{SubscriptionName=subscriptionName,TopicAsTopicName=topicName,CloudStorageConfig=newCloudStorageConfig{Bucket=bucket,FilenamePrefix=filenamePrefix,FilenameSuffix=filenameSuffix,MaxDuration=Duration.FromTimeSpan(maxDuration)}};varsubscription=subscriber.CreateSubscription(subscriptionRequest);returnsubscription;}}

import("context""fmt""io""time""cloud.google.com/go/pubsub/v2""cloud.google.com/go/pubsub/v2/apiv1/pubsubpb""google.golang.org/protobuf/types/known/durationpb")// createCloudStorageSubscription creates a Pub/Sub subscription that exports messages to Cloud Storage.funccreateCloudStorageSubscription(wio.Writer,projectID,topic,subscription,bucketstring)error{// projectID := "my-project-id"// topic := "projects/my-project-id/topics/my-topic"// subscription := "projects/my-project/subscriptions/my-sub"// bucket := "my-bucket" // bucket must not have the gs:// prefixctx:=context.Background()client,err:=pubsub.NewClient(ctx,projectID)iferr!=nil{returnfmt.Errorf("pubsub.NewClient: %w",err)}deferclient.Close()sub,err:=client.SubscriptionAdminClient.CreateSubscription(ctx,&pubsubpb.Subscription{Name:subscription,Topic:topic,CloudStorageConfig:&pubsubpb.CloudStorageConfig{Bucket:bucket,FilenamePrefix:"log_events_",FilenameSuffix:".avro",OutputFormat:&pubsubpb.CloudStorageConfig_AvroConfig_{AvroConfig:&pubsubpb.CloudStorageConfig_AvroConfig{WriteMetadata:true,},},MaxDuration:durationpb.New(1*time.Minute),MaxBytes:1e8,},})iferr!=nil{returnfmt.Errorf("failed to create cloud storage sub: %w",err)}fmt.Fprintf(w,"Created Cloud Storage subscription: %v\n",sub)returnnil}

importcom.google.cloud.pubsub.v1.SubscriptionAdminClient;importcom.google.protobuf.Duration;importcom.google.pubsub.v1.CloudStorageConfig;importcom.google.pubsub.v1.ProjectSubscriptionName;importcom.google.pubsub.v1.ProjectTopicName;importcom.google.pubsub.v1.Subscription;importjava.io.IOException;publicclassCreateCloudStorageSubscriptionExample{publicstaticvoidmain(String...args)throwsException{// TODO(developer): Replace these variables before running the sample.StringprojectId="your-project-id";StringtopicId="your-topic-id";StringsubscriptionId="your-subscription-id";Stringbucket="your-bucket";StringfilenamePrefix="log_events_";StringfilenameSuffix=".text";DurationmaxDuration=Duration.newBuilder().setSeconds(300).build();createCloudStorageSubscription(projectId,topicId,subscriptionId,bucket,filenamePrefix,filenameSuffix,maxDuration);}publicstaticvoidcreateCloudStorageSubscription(StringprojectId,StringtopicId,StringsubscriptionId,Stringbucket,StringfilenamePrefix,StringfilenameSuffix,DurationmaxDuration)throwsIOException{try(SubscriptionAdminClientsubscriptionAdminClient=SubscriptionAdminClient.create()){ProjectTopicNametopicName=ProjectTopicName.of(projectId,topicId);ProjectSubscriptionNamesubscriptionName=ProjectSubscriptionName.of(projectId,subscriptionId);CloudStorageConfigcloudStorageConfig=CloudStorageConfig.newBuilder().setBucket(bucket).setFilenamePrefix(filenamePrefix).setFilenameSuffix(filenameSuffix).setMaxDuration(maxDuration).build();Subscriptionsubscription=subscriptionAdminClient.createSubscription(Subscription.newBuilder().setName(subscriptionName.toString()).setTopic(topicName.toString()).setCloudStorageConfig(cloudStorageConfig).build());System.out.println("Created a CloudStorage subscription: "+subscription.getAllFields());}}}

/***TODO(developer):Uncommentthesevariablesbeforerunningthesample.*///consttopicName='YOUR_TOPIC_NAME';//constsubscriptionName='YOUR_SUBSCRIPTION_NAME';//constbucket='YOUR_BUCKET_ID';//constfilenamePrefix='YOUR_FILENAME_PREFIX';//constfilenameSuffix='YOUR_FILENAME_SUFFIX';//constmaxDuration=60;//ImportstheGoogleCloudclientlibraryconst{PubSub}=require('@google-cloud/pubsub');//Createsaclient;cachethisforfurtheruseconstpubSubClient=newPubSub();asyncfunctioncreateCloudStorageSubscription(topicName,subscriptionName,bucket,filenamePrefix,filenameSuffix,maxDuration,){constoptions={cloudStorageConfig:{bucket,filenamePrefix,filenameSuffix,maxDuration:{seconds:maxDuration,},},};awaitpubSubClient.topic(topicName).createSubscription(subscriptionName,options);console.log(`Createdsubscription${subscriptionName}withacloudstorageconfiguration.`,);}

/***TODO(developer):Uncommentthesevariablesbeforerunningthesample.*///consttopicName='YOUR_TOPIC_NAME';//constsubscriptionName='YOUR_SUBSCRIPTION_NAME';//constbucket='YOUR_BUCKET_ID';//constfilenamePrefix='YOUR_FILENAME_PREFIX';//constfilenameSuffix='YOUR_FILENAME_SUFFIX';//constmaxDuration=60;//ImportstheGoogleCloudclientlibraryimport{CreateSubscriptionOptions,PubSub}from'@google-cloud/pubsub';//Createsaclient;cachethisforfurtheruseconstpubSubClient=newPubSub();asyncfunctioncreateCloudStorageSubscription(topicName:string,subscriptionName:string,bucket:string,filenamePrefix:string,filenameSuffix:string,maxDuration:number,){constoptions:CreateSubscriptionOptions={cloudStorageConfig:{bucket,filenamePrefix,filenameSuffix,maxDuration:{seconds:maxDuration,},},};awaitpubSubClient.topic(topicName).createSubscription(subscriptionName,options);console.log(`Createdsubscription${subscriptionName}withacloudstorageconfiguration.`,);}

use Google\Cloud\PubSub\PubSubClient;/** * Creates a Pub/Sub GCS subscription. * * @param string $projectId  The Google project ID. * @param string $topicName  The Pub/Sub topic name. * @param string $subscriptionName  The Pub/Sub subscription name. * @param string $bucket The Cloud Storage bucket name without any prefix like "gs://". */function create_cloud_storage_subscription($projectId, $topicName, $subscriptionName, $bucket){    $pubsub = new PubSubClient([        'projectId' => $projectId,    ]);    $topic = $pubsub->topic($topicName);    $subscription = $topic->subscription($subscriptionName);    $config = ['bucket' => $bucket];    $subscription->create([        'cloudStorageConfig' => $config    ]);    printf('Subscription created: %s' . PHP_EOL, $subscription->name());}

fromgoogle.cloudimportpubsub_v1fromgoogle.protobufimportduration_pb2# TODO(developer)# project_id = "your-project-id"# topic_id = "your-topic-id"# subscription_id = "your-subscription-id"# bucket = "my-bucket"filename_prefix="log_events_"filename_suffix=".avro"# Either CloudStorageConfig.AvroConfig or CloudStorageConfig.TextConfig# defaults to TextConfigavro_config=pubsub_v1.types.CloudStorageConfig.AvroConfig(write_metadata=True)publisher=pubsub_v1.PublisherClient()subscriber=pubsub_v1.SubscriberClient()topic_path=publisher.topic_path(project_id,topic_id)subscription_path=subscriber.subscription_path(project_id,subscription_id)max_duration=duration_pb2.Duration()max_duration.FromSeconds(300)cloudstorage_config=pubsub_v1.types.CloudStorageConfig(bucket=bucket,filename_prefix=filename_prefix,filename_suffix=filename_suffix,avro_config=avro_config,# Min 1 minutes, max 10 minutesmax_duration=max_duration,# Min 1 KB, max 10 GiBmax_bytes=10000000,)# Wrap the subscriber in a 'with' block to automatically call close() to# close the underlying gRPC channel when done.withsubscriber:subscription=subscriber.create_subscription(request={"name":subscription_path,"topic":topic_path,"cloud_storage_config":cloudstorage_config,})print(f"CloudStorage subscription created:{subscription}.")print(f"Bucket for subscription is:{bucket}")print(f"Prefix is:{filename_prefix}")print(f"Suffix is:{filename_suffix}")

Monitor a Cloud Storage subscription

Cloud Monitoring provides a number of metrics tomonitor subscriptions.

For a list of all the available metrics related to Pub/Suband their descriptions, see theMonitoring documentation for Pub/Sub.

You can also monitor subscriptions fromwithin Pub/Sub.

What's next

• Troubleshoot aCloud Storage subscription.

• Read aboutCloud Storage.

• Review pricing for Pub/Sub, includingCloud Storage subscription.