You can migrate your Pub/Sub Lite service toGoogle Cloud Managed Service for Apache Kafka orPub/Sub.

This document describes how to set up automatic exporting ofPub/Sub Lite messages to Pub/Sub.

Here are some scenarios where you might use this feature:

• Interoperate between workloads that use a mix of Pub/Sub Liteand Pub/Sub.
• Migrate a Pub/Sub Lite workload to Pub/Sub.
• Use advanced Pub/Sub features, such as push subscriptions andfiltering, from an existing application that is based onPub/Sub Lite.
• Consolidate multiple data pipelines.

Overview

To export messages from Pub/Sub Lite to Pub/Sub,you create a special type of subscription called anexport subscription. Anexport subscription receives messages from a Lite topic, converts them toPub/Sub messages, and sends the converted messages to adestination Pub/Sub topic.

A Lite topic can have a combination of both export subscriptions and standardsubscriptions. The two subscription types are identical in terms ofquota usage andreservation throughput. An export subscriptionconsumes Lite subscription throughput capacity and is charged forPub/Sub publishing throughput.

An export subscription connects a Lite topic to exactly onePub/Sub topic. However, a Lite topic can have multiple exportsubscriptions that connect to different Pub/Sub topics(fan-out architecture). You can also export from multiple Lite topics to thesame Pub/Sub topic (fan-in architecture).

Authentication

An export subscription accesses both Pub/Sub Lite andPub/Sub resources. To create an export subscription, you need thefollowing permissions:

• pubsublite.subscriptions.create. The following predefined roles contain thispermission:

  • roles/pubsublite.admin
  • roles/pubsublite.editor

  SeeAccess Control for Pub/Sub Lite.

• pubsub.topics.get. The following predefined roles contain this permission:

  • roles/pubsub.admin
  • roles/pubsub.editor
  • roles/pubsub.viewer

  SeeAccess Control for Pub/Sub.

Service agents

An export subscription publishes to a Pub/Sub topic on yourbehalf. To do this, it uses aservice agent.

After you create the first export subscription in a project, aPub/Sub Lite service agent is automatically created. If youcreate additional export subscriptions in the same project, they use the sameservice agent. The service agent has the following naming scheme:service-<your_project_number>@gcp-sa-pubsublite.iam.gserviceaccount.com.

The service agent is created with permissions to publish to allPub/Sub and Pub/Sub Lite topics within the sameproject as the export subscription. If your destination Pub/Subtopic is in a different project than the export subscription, you must grant theservice agent additional permissions by adding thePub/Sub Publisher role(roles/pubsub.publisher). You can grant permissions for an entire project oran individual topic. We recommend granting permissions at the topic level,following the principle of least privilege.

For more information, seeControlling access through the Google Cloud console.You can also use thegcloud projects add-iam-policy-bindingcommand to add IAM roles:

gcloudpubsubtopicsadd-iam-policy-bindingTOPIC_NAME\--member=serviceAccount:service-PROJECT_NUMBER@gcp-sa-pubsublite.iam.gserviceaccount.com--role=roles/pubsub.publisher

Replace the following:

• TOPIC_NAME: The name of the destination Pub/Subtopic to add the IAM policy binding.
• PROJECT_NUMBER: The project number of thePub/Sub Lite export subscription's project.

Create an export subscription

You can create a Lite export subscription with the Google Cloud console, theGoogle Cloud CLI, or the Pub/Sub Lite API.

A Lite export subscription must be in the same project and location as the Litetopic to which it is attached. To create the Lite topic, seeCreate and manage Lite topics.

If you attach an export subscription to a Lite topic, make sure that allmessages published to the Lite topic are compatible with Pub/Sub.For more information, seeMessage compatibility.

Once it has been created, you cannot change an export subscription to astandard subscription, or vice versa.

gcloudpubsublite-subscriptionscreateSUBSCRIPTION_ID\--location=LOCATION\--topic=TOPIC_ID\--export-pubsub-topic=PUBSUB_TOPIC_NAME\--export-dead-letter-topic=DEAD_LETTER_TOPIC_ID\--export-desired-state=DESIRED_STATE

Created[SUBSCRIPTION_ID].

POST https://REGION-pubsublite.googleapis.com/v1/admin/projects/PROJECT_NUMBER/locations/LOCATION/subscriptions/SUBSCRIPTION_IDAuthorization: Bearer $(gcloud auth print-access-token)

{"topic":"projects/PROJECT_NUMBER/locations/LOCATION/topics/TOPIC_ID","deliveryConfig":{"deliveryRequirement":"DELIVERY_REQUIREMENT",},"exportConfig":{"desiredState":"DESIRED_STATE","deadLetterTopic":"projects/PROJECT_NUMBER/locations/LOCATION/topics/DEAD_LETTER_TOPIC_ID","pubsubConfig":{"topic":"PUBSUB_TOPIC_NAME"}}}

{  "deliveryConfig": {      "deliveryRequirement": "DELIVERY_REQUIREMENT",  },  "exportConfig": {      "desiredState": "DESIRED_STATE",      "deadLetterTopic": "projects/PROJECT_NUMBER/locations/LOCATION/topics/DEAD_LETTER_TOPIC_ID",      "pubsubConfig": {          "topic": "PUBSUB_TOPIC_NAME"      },  "name": "projects/PROJECT_NUMBER/locations/LOCATION/subscriptions/SUBSCRIPTION_ID",  "topic": "projects/PROJECT_NUMBER/locations/LOCATION/topics/TOPIC_ID",}

import("context""fmt""io""cloud.google.com/go/pubsublite")funccreatePubsubExportSubscription(wio.Writer,projectID,region,location,topicID,subID,pubsubTopicIDstring)error{// projectID := "my-project-id"// region := "us-central1"// NOTE: location can be either a region ("us-central1") or a zone ("us-central1-a")// For a list of valid locations, see https://cloud.google.com/pubsub/lite/docs/locations.// location := "us-central1"// NOTE: topic and subscription must be in the same region/zone (e.g. "us-central1-a")// topicID := "my-topic"// subID := "my-subscription"// pubsubTopicID := "destination-topic-id"ctx:=context.Background()client,err:=pubsublite.NewAdminClient(ctx,region)iferr!=nil{returnfmt.Errorf("pubsublite.NewAdminClient: %w",err)}deferclient.Close()// Initialize the subscription to the oldest retained messages for each// partition.targetLocation:=pubsublite.AtTargetLocation(pubsublite.Beginning)sub,err:=client.CreateSubscription(ctx,pubsublite.SubscriptionConfig{Name:fmt.Sprintf("projects/%s/locations/%s/subscriptions/%s",projectID,location,subID),Topic:fmt.Sprintf("projects/%s/locations/%s/topics/%s",projectID,location,topicID),DeliveryRequirement:pubsublite.DeliverImmediately,// Can also be DeliverAfterStored.// Configures an export subscription that writes messages to a Pub/Sub topic.ExportConfig:&pubsublite.ExportConfig{DesiredState:pubsublite.ExportActive,// Can also be ExportPaused.Destination:&pubsublite.PubSubDestinationConfig{Topic:fmt.Sprintf("projects/%s/topics/%s",projectID,pubsubTopicID),},},},targetLocation)iferr!=nil{returnfmt.Errorf("client.CreateSubscription got err: %w",err)}fmt.Fprintf(w,"Created export subscription: %s\n",sub.Name)returnnil}

importcom.google.api.gax.rpc.AlreadyExistsException;importcom.google.cloud.pubsublite.AdminClient;importcom.google.cloud.pubsublite.AdminClientSettings;importcom.google.cloud.pubsublite.BacklogLocation;importcom.google.cloud.pubsublite.CloudRegion;importcom.google.cloud.pubsublite.CloudRegionOrZone;importcom.google.cloud.pubsublite.CloudZone;importcom.google.cloud.pubsublite.ProjectNumber;importcom.google.cloud.pubsublite.SeekTarget;importcom.google.cloud.pubsublite.SubscriptionName;importcom.google.cloud.pubsublite.SubscriptionPath;importcom.google.cloud.pubsublite.TopicName;importcom.google.cloud.pubsublite.TopicPath;importcom.google.cloud.pubsublite.proto.ExportConfig;importcom.google.cloud.pubsublite.proto.ExportConfig.PubSubConfig;importcom.google.cloud.pubsublite.proto.ExportConfig.State;importcom.google.cloud.pubsublite.proto.Subscription;importcom.google.cloud.pubsublite.proto.Subscription.DeliveryConfig;importcom.google.cloud.pubsublite.proto.Subscription.DeliveryConfig.DeliveryRequirement;importjava.util.concurrent.ExecutionException;publicclassCreatePubsubExportSubscriptionExample{publicstaticvoidmain(String...args)throwsException{// TODO(developer): Replace these variables before running the sample.StringcloudRegion="your-cloud-region";charzoneId='b';StringtopicId="your-topic-id";StringsubscriptionId="your-subscription-id";StringpubsubTopicId="destination-topic-id";longprojectNumber=Long.parseLong("123456789");// True if using a regional location. False if using a zonal location.// https://cloud.google.com/pubsub/lite/docs/topicsbooleanregional=false;createPubsubExportSubscriptionExample(cloudRegion,zoneId,projectNumber,topicId,subscriptionId,pubsubTopicId,regional);}publicstaticvoidcreatePubsubExportSubscriptionExample(StringcloudRegion,charzoneId,longprojectNumber,StringtopicId,StringsubscriptionId,StringpubsubTopicId,booleanregional)throwsException{CloudRegionOrZonelocation;if(regional){location=CloudRegionOrZone.of(CloudRegion.of(cloudRegion));}else{location=CloudRegionOrZone.of(CloudZone.of(CloudRegion.of(cloudRegion),zoneId));}TopicPathtopicPath=TopicPath.newBuilder().setProject(ProjectNumber.of(projectNumber)).setLocation(location).setName(TopicName.of(topicId)).build();SubscriptionPathsubscriptionPath=SubscriptionPath.newBuilder().setLocation(location).setProject(ProjectNumber.of(projectNumber)).setName(SubscriptionName.of(subscriptionId)).build();com.google.pubsub.v1.TopicNamepubsubTopicName=com.google.pubsub.v1.TopicName.of(String.valueOf(projectNumber),pubsubTopicId);Subscriptionsubscription=Subscription.newBuilder().setDeliveryConfig(// Possible values for DeliveryRequirement:// - `DELIVER_IMMEDIATELY`// - `DELIVER_AFTER_STORED`// You may choose whether to wait for a published message to be successfully written// to storage before the server delivers it to subscribers. `DELIVER_IMMEDIATELY` is// suitable for applications that need higher throughput.DeliveryConfig.newBuilder().setDeliveryRequirement(DeliveryRequirement.DELIVER_IMMEDIATELY)).setExportConfig(// Configures an export subscription that writes messages to a Pub/Sub topic.ExportConfig.newBuilder()// Possible values for State:// - `ACTIVE`: enable message processing.// - `PAUSED`: suspend message processing..setDesiredState(State.ACTIVE).setPubsubConfig(PubSubConfig.newBuilder().setTopic(pubsubTopicName.toString()))).setName(subscriptionPath.toString()).setTopic(topicPath.toString()).build();// Target location within the message backlog that the subscription should be initialized to.SeekTargetinitialLocation=SeekTarget.of(BacklogLocation.BEGINNING);AdminClientSettingsadminClientSettings=AdminClientSettings.newBuilder().setRegion(location.extractRegion()).build();// Initialize client that will be used to send requests. This client only needs to be created// once, and can be reused for multiple requests. After completing all of your requests, call// the "close" method on the client to safely clean up any remaining background resources, or// use "try-with-close" statement to do this automatically.try(AdminClientadminClient=AdminClient.create(adminClientSettings)){Subscriptionresponse=adminClient.createSubscription(subscription,initialLocation).get();System.out.println(response.getAllFields()+" created successfully.");}catch(ExecutionExceptione){try{throwe.getCause();}catch(AlreadyExistsExceptionalreadyExists){System.out.println("This subscription already exists.");}catch(Throwablethrowable){throwable.printStackTrace();}}}}

fromgoogle.api_core.exceptionsimportAlreadyExistsfromgoogle.cloud.pubsub_v1importPublisherClientfromgoogle.cloud.pubsubliteimportAdminClient,Subscription,ExportConfigfromgoogle.cloud.pubsublite.typesimport(BacklogLocation,CloudRegion,CloudZone,SubscriptionPath,TopicPath,)defcreate_lite_pubsub_export_subscription(project_number,cloud_region="us-central1",zone_id="a",topic_id="my-topic-id",subscription_id="my-subscription-id",pubsub_topic_id="destination-topic-id",regional=True,target_location=BacklogLocation.BEGINNING,):ifregional:location=CloudRegion(cloud_region)else:location=CloudZone(CloudRegion(cloud_region),zone_id)topic_path=TopicPath(project_number,location,topic_id)subscription_path=SubscriptionPath(project_number,location,subscription_id)destination_topic_path=PublisherClient.topic_path(project_number,pubsub_topic_id)subscription=Subscription(name=str(subscription_path),topic=str(topic_path),delivery_config=Subscription.DeliveryConfig(# Possible values for delivery_requirement:# - `DELIVER_IMMEDIATELY`# - `DELIVER_AFTER_STORED`# You may choose whether to wait for a published message to be successfully written# to storage before the server delivers it to subscribers. `DELIVER_IMMEDIATELY` is# suitable for applications that need higher throughput.delivery_requirement=Subscription.DeliveryConfig.DeliveryRequirement.DELIVER_IMMEDIATELY,),# Configures an export subscription that writes messages to a Pub/Sub topic.export_config=ExportConfig(# Possible values for desired_state:# - `ACTIVE`: enable message processing.# - `PAUSED`: suspend message processing.desired_state=ExportConfig.State.ACTIVE,pubsub_config=ExportConfig.PubSubConfig(topic=destination_topic_path,),),)# Initialize client that will be used to send requests across threads. This# client only needs to be created once, and can be reused for multiple requests.client=AdminClient(cloud_region)try:response=client.create_subscription(subscription,target_location)print(f"{response.name} created successfully.")exceptAlreadyExists:print(f"{subscription_path} already exists.")

Update an export subscription

You can update Lite subscriptions with the Google Cloud console, theGoogle Cloud CLI, or the Pub/Sub Lite API. It can take up to 30seconds for the new settings to be applied.

gcloudpubsublite-subscriptionsupdateSUBSCRIPTION_ID\--location=LOCATION\--delivery-requirement=DELIVERY_REQUIREMENT\--export-pubsub-topic=PUBSUB_TOPIC_NAME\--export-dead-letter-topic=DEAD_LETTER_TOPIC_ID\--export-desired-state=DESIRED_STATE

Updatedsubscription[SUBSCRIPTION_ID].deliveryConfig:deliveryRequirement:DELIVERY_REQUIREMENTexportConfig:currentState:DESIRED_STATEdeadLetterTopic:projects/PROJECT_NUMBER/locations/LOCATION/topics/DEAD_LETTER_TOPIC_IDdesiredState:DESIRED_STATEpubsubConfig:topic:PUBSUB_TOPIC_NAMEname:projects/PROJECT_NUMBER/locations/LOCATION/subscriptions/SUBSCRIPTION_IDtopic:projects/PROJECT_NUMBER/locations/LOCATION/topics/TOPIC_ID

PATCH https://REGION-pubsublite.googleapis.com/v1/admin/projects/PROJECT_NUMBER/locations/LOCATION/subscriptions/SUBSCRIPTION_ID?updateMask=deliveryConfig.deliveryRequirement,exportConfigAuthorization: Bearer $(gcloud auth print-access-token)

{"deliveryConfig":{"deliveryRequirement":"DELIVERY_REQUIREMENT",},"exportConfig":{"desiredState":"DESIRED_STATE","deadLetterTopic":"projects/PROJECT_NUMBER/locations/LOCATION/topics/DEAD_LETTER_TOPIC_ID","pubsubConfig":{"topic":"PUBSUB_TOPIC_NAME"}}}

{"deliveryConfig": {  "deliveryRequirement": "DELIVERY_REQUIREMENT",},"exportConfig": {  "desiredState": "DESIRED_STATE",  "deadLetterTopic": "projects/PROJECT_NUMBER/locations/LOCATION/topics/DEAD_LETTER_TOPIC_ID",  "pubsubConfig": { "topic": "PUBSUB_TOPIC_NAME" }},"name": "projects/PROJECT_NUMBER/locations/LOCATION/subscriptions/SUBSCRIPTION_ID","topic": "projects/PROJECT_NUMBER/locations/LOCATION/topics/TOPIC_ID",}

Pause or start an export subscription

Export subscriptions have a setting calleddesired state, which has one oftwo values:

• Active: The subscription exports Lite messages to Pub/Sub.
• Paused: Exporting of Lite messages is suspended.

To change the desired state in the Google Cloud console:

1. Go to theLite Subscriptions page.

   Go to Lite Subscriptions

2. Click the Lite subscription ID.

3. In theLite subscription details page, clickPause orStart.

You can also update the desired state by using the Google Cloud CLI orthe Pub/Sub Lite API. SeeUpdate an export subscription.

Best practices

This section describes some best practices when using export subscriptions.

Reservations

We recommend using an export subscription with areservation, rather than explicitly settingthe throughput capacity of the subscription.

Message compatibility

If a Pub/Sub Lite message is not compatible withPub/Sub, the export subscription does not publish the message toPub/Sub. Instead, it places the message in the dead-letter topic,if one was assigned. If no dead-letter topic was assigned, incompatiblemessages are simply dropped.

When publishing messages to the Lite topic, be aware of the followingcompatibility issues:

• Keys. Pub/Sub Lite keys have typebytes, whilePub/Sub ordering keys are typestring. To be compatible, thePub/Sub Lite key must contain only UTF-8 characters.

• Attributes. Message attributes have the following requirements:

  • In order to be compatible, all Pub/Sub Lite message attributesmust have a single value. Pub/Sub Lite supports messageattributes with multiple values, but Pub/Sub only supportssingle-value attributes.
  • The message attributes must not exceed thePub/Sub message limits,including maximum attributes per message, and maximum key and value size perattribute.

Dead-letter topic

To retain and handle incompatible messages, we recommend using a dead-lettertopic. You can assign a dead-letter topic when you create the exportsubscription, or you can update an existing export subscription to use adead-letter topic. If the subscription receives a message that is incompatiblewith Pub/Sub, it publishes the message to the dead-letter topic.

A dead-letter topic is a regular Pub/Sub Lite topic. It must be inthe same location and project as the export subscription, and it must be adifferent topic from the source topic.

Typically, a dead-letter topic has low throughput utilization. Therefore, werecommend assigning a reservation to the dead-letter topic, rather thanallocate throughput to the topic.

Delivery errors

An export subscription attempts to deliver all compatible messages to thedestination Pub/Sub topic. If message delivery fails, the exportsubscription is suspended. To find the category of error, check thesubscription/export_status metric. The following values indicate an error:

• PERMISSION_DENIED: Insufficient permissions to export messages.
• NOT_FOUND: One or more resources was not found; for example, the destinationtopic does not exist.

For more information on troubleshooting, seeTroubleshoot export subscriptions.

After you resolve the error, the export subscription is automatically restarteddue to periodic retries.

Pricing

You are charged for the Pub/Sub Lite and Pub/Subresources that the export subscription consumes. Specifically, you are chargedfor the allocated subscription throughput and storage on thePub/Sub Lite subscription, which are configured for thePub/Sub Lite topic. You are also charged for publishing to thedestination Pub/Sub topic. SeePub/Sub pricing.

There are no additional charges for using the export feature, and there is noprice difference between Pub/Sub Lite export subscriptions andstandard subscriptions.

Troubleshoot export subscriptions

This section describes some troubleshooting tips for export subscriptions.

The export subscription is paused

If the subscription is paused, no messages are exported.

To detect this issue:

• Google Cloud console: View thesubscription details. If the subscriptionis paused,Desired state andCurrent state arePaused.

• Metrics: Thesubscription/export_status metric isPAUSED.

To resolve this issue,start the subscription.

The destination topic or dead-letter topic was deleted

If you delete the Pub/Sub topic attached to an exportsubscription, or delete the dead-letter topic, an error occurs.

To detect this issue:

• Google Cloud console: View thesubscription details. If the topic wasdeleted, theCurrent state isNot found.

• Metrics: Thesubscription/export_status metric. If the topic wasdeleted, the value isNOT_FOUND.

To resolve this issue, check the destination Pub/Sub topic and thedead-letter topic (if one was configured).

If the destination Pub/Sub was deleted, recreate the topic withthe same name. The export subscription resumes publishing, assuming thepermissions did not change.

If the dead-letter topic was deleted, create a new dead-letter topic and updatethe export subscription to reference it.

Incompatible messages

If messages are incompatible with Pub/Sub, they are not exported.

To detect this issue:

• Metrics: Thesubscription/unexportable_message_count metric shows the countof incompatible messages that could not be exported.

To resolve this issue, use adead-letter topic to retainthe incompatible messages. Examine the messages to determine the cause, thentransform and republish them if necessary. SeeMessage compatibility.

Export is throttled

To detect this issue:

• Metrics: Thesubscription/flow_control_status metric shows a flow-controlreason ofNO_CLIENT_TOKENS, which indicates that the per-partition limit ofoutstanding bytes or messages has been reached. Until the issue is resolved,the backlog will increase for associated export subscriptions.

This error has several possible root causes. Most of the possible causes occuron the back end, but check for the following:

• Ensure that you publish Lite messages sharing the same key at a rate of lessthan 1 MiB/s per key. The export subscription writes Lite message keys asPub/Sub ordering keys, and Pub/Sub has a 1 MiB/slimit on each ordering key. Exceeding thislimit can cause throttling.

User not authorized to perform this action

ThePub/Sub Lite service agent must havepermissions to publish to the destination Pub/Sub topic.

To detect this issue:

• Google Cloud console: View thesubscription details. If there arepermission errors, theCurrent state isPermission denied.

• Metrics: Thesubscription/export_status metric isPERMISSION_DENIED.

For example, the following situations can cause this error:

• The Pub/Sub Lite service agent is missing the correct permissionor role to publish messages to the destination Pub/Sub topic ina different project.
• The service agent was removed from the IAM policy of the exportsubscription's parent project.
• The Pub/Sub Lite service agent is still being set up. A serviceagent is automatically created when you create the first export subscriptionin a project. The permission error should be automatically resolved within 10minutes.

To resolve the issue, check if the service agent was granted the correctpermission or role. SeeService agents.

What's next

Choose between Pub/Sub or Pub/Sub Lite.