This page describes the retry model used by the C++ client libraries.

The client libraries issue RPCs (Remote Procedure Calls) on your behalf. TheseRPCs can fail due to transient errors. Servers restart, load balancersclosing overloaded or idle connections, and rate limits can take effect, andthese are only some examples of transient failures.

The libraries could return these errors to the application. However, manyof these errors are easy to handle in the library, which makes the applicationcode simpler.

Retryable Errors and Retryable Operations

Only transient errors are retryable. For example,kUnavailable indicates thatthe client could not connect, or lost its connection to a service while arequest was in progress. This is almost always a transient condition, though itmay take a long time to recover. These errors are always retryable (assumingthe operation itself is safe to retry). In contract,kPermissionDenied errorsrequire additional intervention (usually by a human) to be resolved. Such errorsare not considered "transient", or at least not transient in the timescalesconsidered by the retry loops in the client library.

Likewise, some operations are not safe to retry, regardless of the nature of theerror. This includes any operations that make incremental changes. For example,it is not safe to retry an operation to remove "the latest version of X" wherethere may be multiple versions of a resource named "X". This is because thecaller probably intended to remove a single version, and retrying such a requestcan result in removing all the versions.

Configure retry loops

The client libraries accept three different configuration parameters tocontrol the retry loops:

• The*IdempotencyPolicy determines if a particular request is idempotent.Only such requests are retried.
• The*RetryPolicy determines (a) if an error should be consider a transientfailure, and (b) how long (or how many times) the client library retries arequest.
• The*BackoffPolicy determines how long the client library waits beforereissuing the request.

Default Idempotency Policy

In general, an operation isidempotent if successfullycalling the function multiple times leaves the system in the same state assuccessfully calling the function once. Only idempotent operations aresafe to retry. Examples of idempotent operations include, but are not limitedto, all read-only operations, and operations that can only succeed once.

By default, the client library only treat RPCs that are implemented viaGET orPUT verbs as idempotent. This may be too conservative, in some services evensomePOST requests are idempotent. You can always override the defaultidempotency policy to better fit your needs.

Some operations are only idempotent if they include pre-conditions. For example,"remove the latest version if the latest version is Y" is idempotent, as itcan only succeed once.

From time to time, the client libraries receive improvements to treat moreoperations as idempotent. We consider these improvements bug fixes, and thereforenon-breaking even if they change the client library behavior.

Note that while it may be safe to retry an operation, this does not mean theoperation produces the same result on the second attempt vs. the firstsuccessful attempt. For example, creating a uniquely identified resource may besafe to retry, as the second and successive attempts fail and leave thesystem in the same state. However, the client may receive an "already exists"error on the retry attempts.

Default Retry Policy

Following the guidelines outlined inaip/194, most C++ client librariesonly retryUNAVAILABLE gRPC-errors. These are mapped toStatusCode::kUnavailable. The default policy is to retry requests for30 minutes.

Note thatkUnavailable errors donot indicate that the server failed toreceive the request. This error code is used when the request cannot be sent,but it is also used if the request is successfully sent, received by theservice, and the connection is lost before the response is received by theclient. Moreover, if you could determine if the request was successfullyreceived, you could solve theTwo General's problem,a well-known impossibility result in distributed systems.

Therefore, it is not safe to retry all operations that fail withkUnavailable. The idempotency of the operation matters too.

Default Backoff Policy

By default, most libraries use a truncated exponential backoff strategy, withjitter. The initial backoff is 1 second, the maximum backoff is 5 minutes,and the backoff doubles after each retry.

Change default retry and backoff policies

Each library defines an*Option struct to configure these policies. You canprovide these options when you create the*Client class, or even on eachrequest.

For example, this shows how to change the retry and backoff policies for aCloud Pub/Sub client:

namespacepubsub=::google::cloud::pubsub;using::google::cloud::future;using::google::cloud::Options;using::google::cloud::StatusOr;[](std::stringproject_id,std::stringtopic_id){autotopic=pubsub::Topic(std::move(project_id),std::move(topic_id));// By default a publisher will retry for 60 seconds, with an initial backoff// of 100ms, a maximum backoff of 60 seconds, and the backoff will grow by// 30% after each attempt. This changes those defaults.autopublisher=pubsub::Publisher(pubsub::MakePublisherConnection(std::move(topic),Options{}.set<pubsub::RetryPolicyOption>(pubsub::LimitedTimeRetryPolicy(/*maximum_duration=*/std::chrono::minutes(10)).clone()).set<pubsub::BackoffPolicyOption>(pubsub::ExponentialBackoffPolicy(/*initial_delay=*/std::chrono::milliseconds(200),/*maximum_delay=*/std::chrono::seconds(45),/*scaling=*/2.0).clone())));std::vector<future<bool>>done;for(charconst*data:{"1","2","3","go!"}){done.push_back(publisher.Publish(pubsub::MessageBuilder().SetData(data).Build()).then([](future<StatusOr<std::string>>f){returnf.get().ok();}));}publisher.Flush();intcount=0;for(auto&f:done){if(f.get())++count;}std::cout <<count <<" messages sent successfully\n";}

Consult the documentation of each library to find the specific names andexamples for that library.

Next Steps

• SeeClient Library Configuration to learn more about common library configuration options.