This page describes best practices for retrying failed requests to theIdentity and Access Management (IAM) API.

For requests that are safe to retry, we recommend using truncated exponential backoff with introduced jitter.

Overview of truncated exponential backoff

Each request to the IAM API can succeed or fail. If your application retries failed requests without waiting, it might send a large number of retries to IAM in a short period of time. As a result, you might exceedquotas and limits that apply to every IAM resource in your Google Cloud project.

To avoid triggering this issue, we strongly recommend that you usetruncated exponential backoff with introducedjitter, which is a standard error-handling strategy for network applications. In this approach, a client periodically retries a failed request with exponentially increasing delays between retries. A small, random delay, known as jitter, is also added between retries. This random delay helps prevent a synchronized wave of retries from multiple clients, also known as thethundering herd problem.

Exponential backoff algorithm

The following algorithm implements truncated exponential backoff with jitter:

1. Send a request to IAM.
2. If the request fails, wait 1 +random-fraction seconds, then retry the request.
3. If the request fails, wait 2 +random-fraction seconds, then retry the request.
4. If the request fails, wait 4 +random-fraction seconds, then retry the request.
5. Continue this pattern, waiting 2ⁿ +random-fraction seconds after each retry, up to amaximum-backoff time.
6. Afterdeadline seconds, stop retrying the request.

Use the following values as you implement the algorithm:

• Before each retry, the wait time ismin((2n + random-fraction), maximum-backoff), withn starting at 0 and incremented by 1 for each retry.
• Replacerandom-fraction with a random fractional value less than or equal to 1. Use a different value for each retry. Adding this random value prevents clients from becoming synchronized and sending large numbers of retries at the same time.
• Replacemaximum-backoff with the maximum amount of time, in seconds, to wait between retries. Typical values are 32 or 64 (2⁵ or 2⁶) seconds. Choose the value that works best for your use case.
• Replacedeadline with the maximum number of seconds to keep sending retries. Choose a value that reflects your use case. For example, in a continuous integration/continuous deployment (CI/CD) pipeline that is not highly time-sensitive, you might setdeadline to 300 seconds (5 minutes).

Types of errors to retry

Use this retry strategy for all requests to the IAM API thatreturn the error codes500,502,503, or504.

Optionally, you can use this retry strategy for requests to theIAM API that return the error code404.IAM reads are eventually consistent; as aresult, resources might not be visible immediately after you create them, whichcan lead to404 errors.

In addition, use a modified version of this retry strategy for all requests tothe IAM API that return the error code409 and the statusABORTED. This type of error indicates a concurrency issue; for example, youmight be trying to update anallow policy that another client hasalready overwritten. For this type of error, you should always retry the entireread-modify-write series of requests, usingtruncated exponential backoff with introduced jitter. If you retry only the write operation, the request willcontinue to fail.

What's next

• Learnhow concurrency issues are managed in allowpolicies.
• Understand how toimplement the read-modify-write patternfor updating allow policies.