You can migrate your Pub/Sub Lite service toGoogle Cloud Managed Service for Apache Kafka orPub/Sub.

This feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of theService Specific Terms. Pre-GA features are available "as is" and might have limited support. For more information, see thelaunch stage descriptions.

This document is useful if you're considering migrating from self-managedApache Kafka to Pub/Sub Lite.

Pub/Sub Lite is a high-volume messaging service built forlow cost of operation. Pub/Sub Lite offers Zonal and RegionalStorage along with pre-provisioned capacity. Within Pub/Sub Lite,you can choose zonal or regional Lite topics. Regional Lite topics offerthe same availability SLA as Pub/Sub topics. However, there arereliability differences between Pub/Sub andPub/Sub Lite in terms ofmessage replication.

To learn more about Pub/Sub and Pub/Sub Lite,seeWhat is Pub/Sub.

To learn more about Lite-supported regions and zones, seePub/Sub Lite locations.

Terminology in Pub/Sub Lite

The following are some key terms for Pub/Sub Lite.

• Message. Data that moves through the Pub/Sub Lite service.

• Topic. A named resource that represents a feed of messages. WithinPub/Sub Lite, you can choose to create a zonal or regionalLite topic. Pub/Sub Lite regional topics store data intwo zones of a single region. Pub/Sub Lite zonal topicsreplicate data within just one zone.

• Reservation. A named pool of throughput capacity shared by multipleLite topics in a region.

• Subscription A named resource that represents an interest inreceiving messages from a particular Lite topic. A subscription is similarto a consumer group in Kafka that only connects to a single topic.

• Subscriber. A client of Pub/Sub Lite that receives messagesfrom a Lite topic and on a specified subscription. A subscriptioncan have multiple subscriber clients. In such a case, the messages areload-balanced across the subscriber clients. In Kafka, a subscriberis called a consumer.

• Publisher. An application that creates messages and sends (publishes)them to a specific Lite topic. A topic can have multiple publishers.In Kafka, a publisher is called a producer.

Differences between Kafka and Pub/Sub Lite

While Pub/Sub Lite is conceptually similar to Kafka,it's a different system with a narrower API that is more focused ondata ingestion. While the differences are immaterial forstream ingestion and processing, there are some specific use cases wherethese differences are important.

Kafka as a database

Unlike Kafka, Pub/Sub Lite doesn't currently supporttransactional publishing or log compaction, although idempotence is supported.These Kafka features are more useful when you use Kafka as a database than as amessaging system. If you use Kafka primarily as a database, consider runningyour own Kafka cluster or using a managed Kafka solution such asConfluent Cloud.If neither of these solutions are an option, you can also consider usinga horizontally scalable database such asCloud Spanner.

Kafka streams

Kafka streams is a data processing system built on top of Kafka. While it does allowinjection of consumer clients, it requires access to all administratoroperations. Kafka Streams also uses the transactional database propertiesof Kafka for storing internal metadata. So, Pub/Sub Litecannot currently be used for Kafka Streams applications.

Apache Beam is a similar streaming data processing system which is integrated with Kafka,Pub/Sub, and Pub/Sub Lite.You can run Beam pipelines in a fully-managed way withDataflow, or on your preexistingApache Flink andApache Spark clusters.

Monitor

Kafka clients can read server-side metrics. In Pub/Sub Lite,metrics relevant to publisher and subscriber behavior are managedthroughCloud Monitoringwith no additional configuration.

Capacity management

The capacity of a Kafka topic is determined by the capacity of the cluster.Replication, key compaction, and batch settings determine the capacity requiredto service any given topic on the Kafka cluster. The throughput of a Kafkatopic is limited by the capacity of the machines on which the brokersare running. By contrast, you must define both storage and throughputcapacity for a Pub/Sub Lite topic.Pub/Sub Lite storage capacity is aconfigurable property of the topic. Throughput capacity is based on thecapacity of the configuredreservation,and inherent or configured per-partition limits.

Authentication and security

Apache Kafka supports several open authentication and encryption mechanisms.With Pub/Sub Lite, authentication is based on theIAM system. Security is assured through encryption at restand in transit. Read more about Pub/Sub Lite authentication in theMigration workflow section, later in this document.

Map Kafka properties to Pub/Sub Lite properties

Kafka has many configuration options that control topic structure,limits, and broker properties. Some common ones useful for data ingestionare discussed in this section, with their equivalents inPub/Sub Lite. As Pub/Sub Lite is amanaged system, you don't have to considermany broker properties.

Topic configuration properties

To learn more about the Lite topic properties,seeProperties of a Lite topic.

Producer configuration properties

Pub/Sub Lite supports theProducer wire protocol.Some properties change the behavior of the producer Cloud Client Libraries;some common ones are discussed in the following table.

Consumer configuration properties

Pub/Sub Lite supports theConsumer wire protocol.Some properties change the behavior of the consumer Cloud Client Libraries;some common ones are discussed in the following table.

Compare Kafka and Pub/Sub Lite features

The following table compares Apache Kafka features with Pub/Sub Lite features:

The following table compares what functionality is self-hosted with Kafka and what functionality is managed by Google by using Pub/Sub Lite:

Kafka and Pub/Sub Lite cost comparison

The way you estimate and manage costs in Pub/Sub Liteis different than in Kafka. The costs for a Kafka cluster on-premises or incloud include the cost of machines, disk, networking, inbound messages, and outboundmessages. It also includes overhead costs for managing and maintaining thesesystems and their related infrastructure. When managing a Kafka cluster,you must manually upgrade the machines, plan cluster capacity, andimplement disaster recovery that includes extensive planning and testing.You must aggregate all these various costs to determine your truetotal cost of ownership (TCO).

Pub/Sub Lite pricing includes the reservation cost(published bytes, subscribed bytes,bytes handled by the Kafka proxy) andthe cost of provisioned storage. You pay for exactly the resources that youreserve in addition to outbound message charges. You can use thepricing calculator to provide an estimate of your costs.

Migration workflow

To migrate a topic from a Kafka cluster toPub/Sub Lite, use the following instructions.

Configure Pub/Sub Lite resources

1. Create a Pub/Sub Litereservation for the expected throughputfor all the topics that you're migrating.

   Use the Pub/Sub Litepricing calculator to calculate the aggregatethroughput metrics of your existing Kafka topics. For more informationabout how to create reservations,seeCreate and manage Lite reservations.

2. Create one Pub/Sub Lite topic for each corresponding topicin Kafka.

   For more information about how to create Lite topics,seeCreate and manage Lite topics.

3. Create one Pub/Sub Lite subscription for each correspondingconsumer group and topic pair in the Kafka cluster.

   For example, for a consumer group namedconsumers that consumes fromtopic-a andtopic-b, you must create a subscriptionconsumers-aattached totopic-a and a subscriptionconsumers-b attached totopic-b.For more information about how to create subscriptions,seeCreate and manage Lite subscriptions.

Authenticate to Pub/Sub Lite

Based on the type of your Kafka client, choose one of the following methods:

• Java-based Kafka clients running version 3.1.0 or later with rebuilding

• Java-based Kafka clients running version 3.1.0 or later without rebuilding

• All other clients without rebuilding

Java-based Kafka clients version 3.1.0 or later with rebuilding

For Java-based Kafka clients of version 3.1.0 or later that can be rebuilton the instance where you're running the Kafka client:

1. Install thecom.google.cloud:pubsublite-kafka-auth package.

2. Obtain the necessary parameters for authenticating toPub/Sub Lite with the help ofcom.google.cloud.pubsublite.kafka.ClientParameters.getParams.

   ThegetParams() method (see acode sample ) initializes the followingJAAS andSASL configurations as parameters for authenticating toPub/Sub Lite:

   security.protocol=SASL_SSLsasl.mechanism=OAUTHBEARERsasl.oauthbearer.token.endpoint.url=http://localhost:14293sasl.jaas.config=org.apache.kafka.common.security.oauthbearer.secured.OAuthBearerLoginCallbackHandler

Java-based Kafka clients running version 3.1.0 or later without rebuilding

For Kafka clients that supportKIP-768, we supportconfiguration-only OAUTHBEARER authentication that uses a Python sidecar script.These versions include theJanuary 2022 Java version 3.1.0 or later.

Perform the following steps on the instance where you're running your Kafka client:

1. Install Python 3.6 or higher.

   SeeInstalling Python.

2. Install the Google authentication package:pip install google-auth

   This library simplifies the various server-to-server authenticationmechanisms to access Google APIs. See thegoogle-auth page.

3. Run thekafka_gcp_credentials.py script.

   This script starts a local HTTP server and fetches the default Google Cloud credentialsin the environment usinggoogle.auth.default().

   Theprincipal in the fetched credentialsmust have thepubsublite.locations.openKafkaStreampermission for the Google Cloud project you are using and the location to whichyou are connecting. Pub/Sub Lite Publisher(roles/pubsublite.publisher) and Pub/Sub Lite Subscriber(roles/pubsublite.subscriber) roles have thisrequired permission. Add these roles to yourprincipal.

   The credentials are used in theSASL/OAUTHBEARER authenticationfor the Kafka client.

   The following parameters are required in your propertiesto authenticate to Pub/Sub Lite from the Kafka client:

   security.protocol=SASL_SSLsasl.mechanism=OAUTHBEARERsasl.oauthbearer.token.endpoint.url=localhost:14293sasl.login.callback.handler.class=org.apache.kafka.common.security.oauthbearer.secured.OAuthBearerLoginCallbackHandlersasl.jaas.config=org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule\requiredclientId="unused"clientSecret="unused"\extension_pubsubProject="PROJECT_ID";

   ReplacePROJECT_ID with the ID of your project running Pub/Sub Lite.

All other clients without rebuilding

For all other clients, perform the following steps:

1. Download aservice account key JSON filefor the service account that you intend to use for your client.

2. Encode the service account file by using base64-encode to use as yourauthentication string.

   On Linux or macOS systems, you can use thebase64 command(often installed by default) as follows:

   base64 <my_service_account.json >password.txt

   You can use the contents of the password file for authenticationwith the following parameters.

   Java

   security.protocol=SASL_SSLsasl.mechanism=PLAINsasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \ username="PROJECT_ID" \ password="contents of base64 encoded password file";

   ReplacePROJECT_ID with the ID of your project running Pub/Sub.

   librdkafka

   security.protocol=SASL_SSLsasl.mechanism=PLAINsasl.username=PROJECT_IDsasl.password=contents of base64 encoded password file

   ReplacePROJECT_ID with the ID of your project running Pub/Sub..

Clone data using Kafka Connect

The Pub/Sub Lite team maintains an implementation of aKafka Connect Sink. You can configure this implementation to copy datafrom a Kafka topic to a Pub/Sub Lite topic using aKafka Connect cluster.

To configure the connector to perform the data copy, seePub/Sub Group Kafka Connector.

If you want to ensure that partition affinity is unaffected by the migrationprocess, ensure that the kafka topic and Pub/Sub Lite topic have thesame number of partitions, and that thepubsublite.ordering.modeproperty is set toKAFKA. This causes the connector to route messages tothe Pub/Sub Lite partition with the same index as the kafka partitionthey were originally published to.

Migrate consumers

Pub/Sub Lite's resource model is different than Kafka's. Most notably,unlike a consumer group, a subscription is an explicit resource and isassociated with exactly one topic. Because of this difference, any place in theKafka Consumer API that requires atopic to be passed, the full subscriptionpath must be passed instead.

In addition to the SASL configurations for the Kafka client,the following settings are also required when using the Kafka Consumer API tointeract with Pub/Sub Lite.

bootstrap.servers=REGION-kafka-pubsub.googleapis.com:443group.id=unused

ReplaceREGION with theregion where your Pub/Sub Lite subscription exists.

Before starting the first Pub/Sub Lite consumer job for a givensubscription, you can initiate (but don't wait on) anadmin seek operation to set the initial location foryour consumer.

When you start your consumers, they reconnect to the current offset inthe message backlog. Run both the old andnew clients in parallel as long as it takes to verify their behavior,then turn down the old consumer clients.

Migrate producers

In addition to the SASL configurations for the Kafka client,the following is also required as a producer param when using theKafka Producer API to interact with Pub/Sub Lite.

bootstrap.servers=REGION-kafka-pubsub.googleapis.com:443

ReplaceREGION with theregion where your Pub/Sub Lite topic exists.

After you migrateall the consumers of the topic to readfrom Pub/Sub Lite, move your producer traffic to write toPub/Sub Lite directly.

Gradually migrate the producer clients to write to thePub/Sub Lite topic instead of the Kafka topic.

Restart the producer clients to pick up new configurations.

Turn down Kafka Connect

After you migrateall the producers to write toPub/Sub Lite directly, the connector does not copy data anymore.

You can turn down the Kafka Connect instance.

Troubleshoot Kafka connections

Since Kafka clients communicate through a bespoke wire protocol,we cannot provide error messages for failures in all requests.Rely on the error codes sent as part of the message.

You can see more details about errors that occur in the client bysetting the logging level for theorg.apache.kafka prefix toFINEST.

Low throughput and increasing backlog

There are multiple reasons why you might be seeing low throughputand an increasing backlog. One reason might be insufficient capacity.

You can configure throughput capacity at the topic level or byusing reservations. If insufficient throughput capacity for subscribe andpublish is configured, the corresponding throughput forsubscribe and publish is throttled.

This throughput error is signaled by thetopic/flow_control_status metricfor publishers, and thesubscription/flow_control_statusmetric for subscribers. The metric provides the following states:

• NO_PARTITION_CAPACITY: This message indicates that the per-partitionthroughput limit is reached.

• NO_RESERVATION_CAPACITY: This message indicates that the per-reservationthroughput limit is reached.

You can view the utilization graphs for the topic or reservationpublish and subscribe quota and check whether utilization is at or near 100%.

To resolve this issue, increase thethroughput capacity of thetopic or reservation.

Topic authorization failed error message

Publishing by using the Kafka API requires the Lite service agent to have theright permissions to publish to the Pub/Sub Lite topic.

You get the errorTOPIC_AUTHORIZATION_FAILED in your client in the eventthat you don't have the correct permissions to publish to thePub/Sub Lite topic.

To resolve the issue, check if the Lite service agent for the project passedin the auth configuration.

Invalid topic error message

Subscribing by using the Kafka API requires passing the full subscription pathall places where atopic is expected in the Kafka Consumer API.

You get the errorINVALID_TOPIC_EXCEPTION in your Consumer client if you don'tpass a well formatted subscription path.

Invalid request when not using reservations

Using kafka wire protocol support requires that all topics have an associatedreservation in order to charge for usage.