This document provides some common troubleshooting tips when publishing messagesto a standard Pub/Sub topic.

Learn more about how toPublish messages to topics and the different features.

High publish latency

Publish latency is the amount of time it takes to complete a publish request that is issued by a publisher client. Publish latency is distinct fromend-to-end delivery latency, which is the amount of time it takes for a message that is published to Pub/Sub to be delivered to a subscriber client. You might observe high publish latency or end-to-end latency, even when the value of the other latency type is low. High publish latency can be incurred at the Pub/Sub publisher client, in transit between the client and the Pub/Sub backend, or in the Pub/Sub backend. You can inspect the publish latency incurred in the Pub/Sub backend using thetopic/send_request_latencies metric. High backend publish latency could be related to the following factors:

• Pub/Sub is designed for low-latency, high-throughput delivery. If the topic has low throughput, the resources associated with the topic could take longer to initialize.

• If you are using a message storage policy, it could affect the overall latency of the requests to the topic and subscription. Check thePerformance and availability implications of using this configuration.

If your publisher client is observing publish latency significantly higher than what is reflected in the metric, it could be a sign of one of these client-side factors:

• Ensure you are not creating a new publisher for every publish. It is recommended to use a single publisher client per topic per application to publish all messages. Spinning up new publisher objects and adding new threads has a latency cost. If you are using Cloud Run functions to publish messages, note that invocations can also affect publish latency.

• If you find that the default retry settings are not sufficient for your use case, make the corresponding adjustments. However, verify that the new values are not too high. See how to configure theRetry requests.

Note that high publish latency can lead toDEADLINE_EXCEEDED errors, which are discussed in the next section.

Publish operations fail with DEADLINE_EXCEEDED

ADEADLINE_EXCEEDED error during a publish request indicates that the request failed to complete within the time allocated. This could be due to various factors, such as the requests not reaching the Pub/Sub service or performance issues affecting the requests.

To verify that publish requests are reaching the Pub/Sub service, monitor the topic using thetopic/send_request_count metric, grouped byresponse_code. This metric helps you determine if requests fail before reaching the Pub/Sub topic. If the metric is empty, there is an external factor preventing the messages from reaching the Pub/Sub service. Additionally, to rule out the possibility of an intermittent issue, check the error rate using thetopic/send_request_count metric graph mentioned earlier, or theAPIs & Services page in the Google Cloud console. If the error rate is very low, this could be an intermittent issue.

If requests are reaching Pub/Sub, these are some possible causes of publish operations failing with aDEADLINE_EXCEEDED error:

Client-side bottleneck

Publish failures are likely caused by a client-side bottleneck, such as insufficient memory, CPU pressure, bad thread health, or network congestion in the VM hosting the publisher client. If aPublish call returnsDEADLINE_EXCEEDED, it could be that asynchronousPublish calls are being enqueued faster than they are sent to the service, which progressively increases the request latency. Additionally, check if any of the following help to determine a possible bottleneck in your system:

• Check whether you are publishing messages faster than the client can send them. Usually each asynchronousPublish call returns aFuture object. To track the number of messages waiting to be sent, store the number of messages to be sent with eachPublish call and delete it only in the callback of theFuture object.

• Ensure that you have sufficient upload bandwidth between the machine where the publisher is running and Google Cloud. Development Wi-Fi networks typically have bandwidth of 1-10 MBps, or 1000-10000 typical messages per second. Publishing messages in a loop without any rate limiting could create a short burst of high bandwidth over a short time period. To avoid this, you can run the publisher on a machine within Google Cloud or reduce the rate at which you publish the messages to match your available bandwidth.

• Check whether you see very high latency between your host and Google Cloud for any of the reasons like startup network congestion or firewalls.Calculating network throughput has pointers on finding out your bandwidth and latency for different scenarios.

• Ultimately, there are limits to how much data a single machine can publish. You may need to try to scale horizontally or run multiple instances of the publisher client on several machines.Testing Cloud Pub/Sub clients to maximize streaming performance demonstrates how Pub/Sub scales on a single Google Cloud VM with increasing number of CPUs. For example, you can achieve 500 MBps to 700 MBps for 1KB messages on a 16 core Compute Engine instance.

Inadequate publish timeout duration

To reduce the timeout rate for publish calls, ensure you have a long enough timeout defined in the publisher client's retry settings. For the retry settings, set the initial deadline to 10 seconds and the total timeout to 600 seconds. Even if you are not accumulating a large backlog of unsent messages, occasional spikes in request latency can cause publish calls to timeout. However, if your issues are caused by a persistent bottleneck, rather than occasional timeouts, retrying more times could lead to more errors.

Client library issues

You could be running a version of the client library with a known issue. The following list includes the issue trackers for all client libraries. If you find a known issue affecting the client library version that you are using, upgrade to the latest version of the client library. This ensures that you have picked up any relevant updates, including fixes and performance improvements.

Schema issues

If your publishes start to returnINVALID_ARGUMENT, it is possible that someone hasupdated the topic to change the associated schema,deleted the schema, ordeleted the schema revisions associated with the topic. In this case, update the topic's schema settings to a schema and set of revisions that match the messages being published, or adjust the message format to match the current schema.

Message encryption issues

If you have configured your Pub/Sub topic to encrypt published messages using a customer-managed encryption key, publishing could fail with aFAILED_PRECONDITION error. This error might occur if the Cloud KMS key is disabled or if externally managed keys through Cloud EKM are no longer accessible. To resume publishing,restore access to the key.

Single Message Transform (SMT) issues

If you have configuredSMTs on yourPub/Sub topic, publishing might failwithINVALID_ARGUMENT errors when transformations fail to be applied to messages.If any message in a publish batch fails transformation, the entire batchfails to be published. The error returned indicates the failure reason, forexample:

INVALID_ARGUMENT: Pub/Sub failed to apply a message transformation to one ormore messages in the publish request. Error: Failed to execute JavaScript UDF:`my_function`. Return value is not an object.

Monitor SMTs

To understand the performance and impact of SMTs on a topic,use the following monitoring metrics:

Thetopic/message_transform_latenciesmetric measures how long it takes for SMTs to be applied to a message. Themetric measures only the SMT latency and does not include other parts of themessage delivery time.

The metric provides two key labels:

• status: reports whether the transformation is successful or encountered an issue.

• filtered: indicates if the SMT caused the message to be filtered out.When an SMT filters a message on a topic, Pub/Sub drops themessage, and the message is never sent to subscribers. Thisfiltered labelis true only when an SMT performs the filtering. Messages filtered usingPub/Sub'sbuilt-in filtering capabilitiesare not reflected in this specific metric.

Thetopic/byte_cost metricis used to identify messages that are filtered by SMTs or where SMTsfailed. Look for these specific values:

• When an SMT filters a message, the operation_type issmt_publish_filter_drop.

• If an SMT fails to transform a message, you see aresponse_code that is notOK.

What's next

ExploreOpenTelemetry tracing to help you debug your publish latency.