Retry¶

Retry implementation for Google API client libraries.

classgoogle.api_core.retry.AsyncRetry(predicate:typing.Callable[[Exception],bool]=<functionif_exception_type.<locals>.if_exception_type_predicate>,initial:float=1.0,maximum:float=60.0,multiplier:float=2.0,timeout:typing.Optional[float]=120.0,on_error:typing.Optional[typing.Callable[[Exception],typing.Any]]=None,**kwargs:typing.Any)[source]¶

Bases:google.api_core.retry.retry_base._BaseRetry

Exponential retry decorator for async coroutines.

This class is a decorator used to add exponential back-off retry behaviorto an RPC call.

Although the default behavior is to retry transient API errors, adifferent predicate can be provided to retry other exceptions.

Parameters

• predicate (Callable[Exception]) – A callable that should returnTrueif the given exception is retryable.

• initial (float) – The minimum amount of time to delay in seconds. Thismust be greater than 0.

• maximum (float) – The maximum amount of time to delay in seconds.

• multiplier (float) – The multiplier applied to the delay.

• timeout (Optional[float]) – How long to keep retrying in seconds.Note: timeout is only checked before initiating a retry, so the target mayrun past the timeout value as long as it is healthy.

• on_error (Optional[Callable[Exception]]) – A function to call while processinga retryable exception. Any error raised by this function willnot be caught.

• deadline (float) – DEPRECATED usetimeout instead. If set it will

• parameter. (override timeout) –

__call__(func:Callable[...,Awaitable[_R]],on_error:Callable[[Exception],Any]|None=None)→Callable[_P,Awaitable[_R]][source]¶

Wrap a callable with retry behavior.

Parameters

• func (Callable) – The callable or stream to add retry behavior to.

• on_error (Optional[Callable[Exception]]) – If given, theon_error callback will be called with each retryable exceptionraised by the wrapped function. Any error raised by thisfunction willnot be caught. If on_error was specified in theconstructor, this value will be ignored.

Returns

A callable that will invokefunc with retry

behavior.

Return type

Callable

classgoogle.api_core.retry.AsyncStreamingRetry(predicate:typing.Callable[[Exception],bool]=<functionif_exception_type.<locals>.if_exception_type_predicate>,initial:float=1.0,maximum:float=60.0,multiplier:float=2.0,timeout:typing.Optional[float]=120.0,on_error:typing.Optional[typing.Callable[[Exception],typing.Any]]=None,**kwargs:typing.Any)[source]¶

Bases:google.api_core.retry.retry_base._BaseRetry

Exponential retry decorator for async streaming rpcs.

This class returns an AsyncGenerator when called, which wraps the targetstream in retry logic. If any exception is raised by the target, theentire stream will be retried within the wrapper.

Although the default behavior is to retry transient API errors, adifferent predicate can be provided to retry other exceptions.

Important Note: when a stream is encounters a retryable error, it willsilently construct a fresh iterator instance in the backgroundand continue yielding (likely duplicate) values as if no error occurred.This is the most general way to retry a stream, but it often is not thedesired behavior. Example: iter([1, 2, 1/0]) -> [1, 2, 1, 2, …]

There are two ways to build more advanced retry logic for streams:

1. Wrap the target

   Use atarget that maintains state between retries, and creates adifferent generator on each retry call. For example, you can wrap agrpc call in a function that modifies the request based on what hasalready been returned:

   asyncdefattempt_with_modified_request(target,request,seen_items=[]):# remove seen items from request on each attemptnew_request=modify_request(request,seen_items)new_generator=awaittarget(new_request)asyncforiteminnew_generator:yielditemseen_items.append(item)retry_wrapped=AsyncRetry(is_stream=True,...)(attempt_with_modified_request,target,request,[])

   2. Wrap the retry generator

      Alternatively, you can wrap the retryable generator itself beforepassing it to the end-user to add a filter on the stream. Forexample, you can keep track of the items that were successfully yieldedin previous retry attempts, and only yield new items when thenew attempt surpasses the previous ones:

      asyncdefretryable_with_filter(target):stream_idx=0# reset stream_idx when the stream is retrieddefon_error(e):nonlocalstream_idxstream_idx=0# build retryableretryable_gen=AsyncRetry(is_stream=True,...)(target)# keep track of what has been yielded out of filterseen_items=[]asyncforiteminretryable_gen:ifstream_idx>=len(seen_items):yielditemseen_items.append(item)elifitem!=previous_stream[stream_idx]:raiseValueError("Stream differs from last attempt")"stream_idx+=1filter_retry_wrapped=retryable_with_filter(target)

Parameters

• predicate (Callable[Exception]) – A callable that should returnTrueif the given exception is retryable.

• initial (float) – The minimum amount of time to delay in seconds. Thismust be greater than 0.

• maximum (float) – The maximum amount of time to delay in seconds.

• multiplier (float) – The multiplier applied to the delay.

• timeout (Optional[float]) – How long to keep retrying in seconds.Note: timeout is only checked before initiating a retry, so the target mayrun past the timeout value as long as it is healthy.

• on_error (Optional[Callable[Exception]]) – A function to call while processinga retryable exception. Any error raised by this function willnot be caught.

• is_stream (bool) – Indicates whether the input functionshould be treated as a stream function (i.e. an AsyncGenerator,or function or coroutine that returns an AsyncIterable).If True, the iterable will be wrapped with retry logic, and anyfailed outputs will restart the stream. If False, only the inputfunction call itself will be retried. Defaults to False.To avoid duplicate values, retryable streams should typically bewrapped in additional filter logic before use.

• deadline (float) – DEPRECATED usetimeout instead. If set it will

• parameter. (override timeout) –

__call__(func:Callable[...,AsyncIterable[_Y]|Awaitable[AsyncIterable[_Y]]],on_error:Callable[[Exception],Any]|None=None)→Callable[_P,Awaitable[AsyncGenerator[_Y,None]]][source]¶

Wrap a callable with retry behavior.

Parameters

• func (Callable) – The callable or stream to add retry behavior to.

• on_error (Optional[Callable[Exception]]) – If given, theon_error callback will be called with each retryable exceptionraised by the wrapped function. Any error raised by thisfunction willnot be caught. If on_error was specified in theconstructor, this value will be ignored.

Returns

A callable that will invokefunc with retry

behavior.

Return type

Callable

classgoogle.api_core.retry.Retry(predicate:typing.Callable[[Exception],bool]=<functionif_exception_type.<locals>.if_exception_type_predicate>,initial:float=1.0,maximum:float=60.0,multiplier:float=2.0,timeout:typing.Optional[float]=120.0,on_error:typing.Optional[typing.Callable[[Exception],typing.Any]]=None,**kwargs:typing.Any)[source]¶

Bases:google.api_core.retry.retry_base._BaseRetry

Exponential retry decorator for unary synchronous RPCs.

This class is a decorator used to add retry or polling behavior to an RPCcall.

Although the default behavior is to retry transient API errors, adifferent predicate can be provided to retry other exceptions.

There are two important concepts that retry/polling behavior may operate on,Deadline and Timeout, which need to be properly defined for the correctusage of this class and the rest of the library.

Deadline: a fixed point in time by which a certain operation mustterminate. For example, if a certain operation has a deadline“2022-10-18T23:30:52.123Z” it must terminate (successfully or with anerror) by that time, regardless of when it was started or whether itwas started at all.

Timeout: the maximum duration of time after which a certain operationmust terminate (successfully or with an error). The countdown begins rightafter an operation was started. For example, if an operation was started at09:24:00 with timeout of 75 seconds, it must terminate no later than09:25:15.

Unfortunately, in the past this class (and the api-core library as a whole) has notbeen properly distinguishing the concepts of “timeout” and “deadline”, and thedeadline parameter has meanttimeout. That is whydeadline has been deprecated andtimeout should be used instead. If thedeadline parameter is set, it will override thetimeout parameter.In other words,retry.deadline should be treated as just a deprecated alias forretry.timeout.

Said another way, it is safe to assume that this class and the rest of thislibrary operate in terms of timeouts (not deadlines) unless explicitlynoted the usage of deadline semantics.

It is also important tounderstand the three most common applications of the Timeout concept in thecontext of this library.

Usually the generic Timeout term may stand for one of the following actualtimeouts: RPC Timeout, Retry Timeout, or Polling Timeout.

RPC Timeout: a value supplied by the client to the server sothat the server side knows the maximum amount of time it is expected tospend handling that specific RPC. For example, in the case of gRPC transport,RPC Timeout is represented by setting “grpc-timeout” header in the HTTP2request. Thetimeout property of this class normally never represents theRPC Timeout as it is handled separately by thegoogle.api_core.timeoutmodule of this library.

Retry Timeout: this is the most common meaning of thetimeout propertyof this class, and defines how long a certain RPC may be retried in casethe server returns an error.

Polling Timeout: defines how long theclient side is allowed to call the polling RPC repeatedly to check a status of along-running operation. Each polling RPC isexpected to succeed (its errors are supposed to be handled by the retrylogic). The decision as to whether a new polling attempt needs to be made is basednot on the RPC status code but on the status of the returnedstatus of an operation. In other words: we will poll a long-running operation untilthe operation is done or the polling timeout expires. Each poll will inform us ofthe status of the operation. The poll consists of an RPC to the server that mayitself be retried as per the poll-specific retry settings in case of errors. Theoperation-level retry settings do NOT apply to polling-RPC retries.

With the actual timeout types being defined above, the client librariesoften refer to just Timeout without clarifying which type specificallythat is. In that case the actual timeout type (sometimes also referred to asLogical Timeout) can be determined from the context. If it is a unary rpccall (i.e. a regular one) Timeout usually stands for the RPC Timeout (ifprovided directly as a standalone value) or Retry Timeout (if provided asretry.timeout property of the unary RPC’s retry config). ForOperation orPollingFuture in general Timeout stands forPolling Timeout.

Parameters

• predicate (Callable[Exception]) – A callable that should returnTrueif the given exception is retryable.

• initial (float) – The minimum amount of time to delay in seconds. Thismust be greater than 0.

• maximum (float) – The maximum amount of time to delay in seconds.

• multiplier (float) – The multiplier applied to the delay.

• timeout (Optional[float]) – How long to keep retrying, in seconds.Note: timeout is only checked before initiating a retry, so the target mayrun past the timeout value as long as it is healthy.

• on_error (Callable[Exception]) – A function to call while processinga retryable exception. Any error raised by this function willnot be caught.

• deadline (float) – DEPRECATED: usetimeout instead. For backwardcompatibility, if specified it will override thetimeout parameter.

__call__(func:Callable[_P,_R],on_error:Callable[[Exception],Any]|None=None)→Callable[_P,_R][source]¶

Wrap a callable with retry behavior.

Parameters

• func (Callable) – The callable to add retry behavior to.

• on_error (Optional[Callable[Exception]]) – If given, theon_error callback will be called with each retryable exceptionraised by the wrapped function. Any error raised by thisfunction willnot be caught. If on_error was specified in theconstructor, this value will be ignored.

Returns

A callable that will invokefunc with retry

behavior.

Return type

Callable

classgoogle.api_core.retry.RetryFailureReason(value)[source]¶

Bases:enum.Enum

The cause of a failed retry, used when building exceptions

classgoogle.api_core.retry.StreamingRetry(predicate:typing.Callable[[Exception],bool]=<functionif_exception_type.<locals>.if_exception_type_predicate>,initial:float=1.0,maximum:float=60.0,multiplier:float=2.0,timeout:typing.Optional[float]=120.0,on_error:typing.Optional[typing.Callable[[Exception],typing.Any]]=None,**kwargs:typing.Any)[source]¶

Bases:google.api_core.retry.retry_base._BaseRetry

Exponential retry decorator for streaming synchronous RPCs.

This class returns a Generator when called, which wraps the targetstream in retry logic. If any exception is raised by the target, theentire stream will be retried within the wrapper.

Although the default behavior is to retry transient API errors, adifferent predicate can be provided to retry other exceptions.

Important Note: when a stream encounters a retryable error, it willsilently construct a fresh iterator instance in the backgroundand continue yielding (likely duplicate) values as if no error occurred.This is the most general way to retry a stream, but it often is not thedesired behavior. Example: iter([1, 2, 1/0]) -> [1, 2, 1, 2, …]

There are two ways to build more advanced retry logic for streams:

1. Wrap the target

   Use atarget that maintains state between retries, and creates adifferent generator on each retry call. For example, you can wrap anetwork call in a function that modifies the request based on what hasalready been returned:

   defattempt_with_modified_request(target,request,seen_items=[]):# remove seen items from request on each attemptnew_request=modify_request(request,seen_items)new_generator=target(new_request)foriteminnew_generator:yielditemseen_items.append(item)retry_wrapped_fn=StreamingRetry()(attempt_with_modified_request)retryable_generator=retry_wrapped_fn(target,request)

2. Wrap the retry generator

   Alternatively, you can wrap the retryable generator itself beforepassing it to the end-user to add a filter on the stream. Forexample, you can keep track of the items that were successfully yieldedin previous retry attempts, and only yield new items when thenew attempt surpasses the previous ones:

   defretryable_with_filter(target):stream_idx=0# reset stream_idx when the stream is retrieddefon_error(e):nonlocalstream_idxstream_idx=0# build retryableretryable_gen=StreamingRetry(...)(target)# keep track of what has been yielded out of filterseen_items=[]foriteminretryable_gen():ifstream_idx>=len(seen_items):seen_items.append(item)yielditemelifitem!=seen_items[stream_idx]:raiseValueError("Stream differs from last attempt")stream_idx+=1filter_retry_wrapped=retryable_with_filter(target)

Parameters

• predicate (Callable[Exception]) – A callable that should returnTrueif the given exception is retryable.

• initial (float) – The minimum amount of time to delay in seconds. Thismust be greater than 0.

• maximum (float) – The maximum amount of time to delay in seconds.

• multiplier (float) – The multiplier applied to the delay.

• timeout (float) – How long to keep retrying, in seconds.Note: timeout is only checked before initiating a retry, so the target mayrun past the timeout value as long as it is healthy.

• on_error (Callable[Exception]) – A function to call while processinga retryable exception. Any error raised by this function willnot be caught.

• deadline (float) – DEPRECATED: usetimeout instead. For backwardcompatibility, if specified it will override thetimeout parameter.

__call__(func:Callable[_P,Iterable[_Y]],on_error:Callable[[Exception],Any]|None=None)→Callable[_P,Generator[_Y,Any,None]][source]¶

Wrap a callable with retry behavior.

Parameters

• func (Callable) – The callable to add retry behavior to.

• on_error (Optional[Callable[Exception]]) – If given, theon_error callback will be called with each retryable exceptionraised by the wrapped function. Any error raised by thisfunction willnot be caught. If on_error was specified in theconstructor, this value will be ignored.

Returns

A callable that will invokefunc with retry

behavior.

Return type

Callable

google.api_core.retry.build_retry_error(exc_list:list[Exception],reason:google.api_core.retry.retry_base.RetryFailureReason,timeout_val:float|None,**kwargs:Any)→tuple[Exception,Exception|None][source]¶

Default exception_factory implementation.

Returns a RetryError if the failure is due to a timeout, otherwisereturns the last exception encountered.

Parameters

• exc_list (-) – list of exceptions that occurred during the retry

• reason (-) – reason for the retry failure.Can be TIMEOUT or NON_RETRYABLE_ERROR

• timeout_val (-) – the original timeout value for the retry (in seconds), for use in the exception message

Returns

a tuple of the exception to be raised, and the cause exception if any

Return type

• tuple

google.api_core.retry.exponential_sleep_generator(initial:float,maximum:float,multiplier:float=2.0)[source]¶

Generates sleep intervals based on the exponential back-off algorithm.

This implements theTruncated Exponential Back-off algorithm.

Parameters

• initial (float) – The minimum amount of time to delay. This mustbe greater than 0.

• maximum (float) – The maximum amount of time to delay.

• multiplier (float) – The multiplier applied to the delay.

Yields

float – successive sleep intervals.

google.api_core.retry.if_exception_type(*exception_types:type[Exception])→Callable[[Exception],bool][source]¶

Creates a predicate to check if the exception is of a given type.

Parameters

exception_types (Sequence[type()]) – The exception types to checkfor.

Returns

A predicate that returns True if the provided

exception is of the given type(s).

Return type

Callable[Exception]

google.api_core.retry.if_transient_error(exception:Exception)→bool¶

Bound predicate for checking an exception type.

google.api_core.retry.retry_target(target:Callable[[],_R],predicate:Callable[[Exception],bool],sleep_generator:Iterable[float],timeout:float|None=None,on_error:Callable[[Exception],None]|None=None,exception_factory:Callable[[list[Exception],RetryFailureReason,float|None],tuple[Exception,Exception|None]]=<functionbuild_retry_error>,**kwargs)[source]¶

Call a function and retry if it fails.

This is the lowest-level retry helper. Generally, you’ll use thehigher-level retry helperRetry.

Parameters

• target (Callable) – The function to call and retry. This must be anullary function - apply arguments withfunctools.partial.

• predicate (Callable[Exception]) – A callable used to determine if anexception raised by the target should be considered retryable.It should return True to retry or False otherwise.

• sleep_generator (Iterable[float]) – An infinite iterator that determineshow long to sleep between retries.

• timeout (Optional[float]) – How long to keep retrying the target.Note: timeout is only checked before initiating a retry, so the target mayrun past the timeout value as long as it is healthy.

• on_error (Optional[Callable[Exception]]) – If given, the on_errorcallback will be called with each retryable exception raised by thetarget. Any error raised by this function willnot be caught.

• exception_factory – A function that is called when the retryable reachesa terminal failure state, used to construct an exception to be raised.It takes a list of all exceptions encountered, a retry.RetryFailureReasonenum indicating the failure cause, and the original timeout valueas arguments. It should return a tuple of the exception to be raised,along with the cause exception if any. The default implementation will raisea RetryError on timeout, or the last exception encountered otherwise.

• deadline (float) – DEPRECATED: usetimeout instead. For backwardcompatibility, if specified it will overridetimeout parameter.

Returns

the return value of the target function.

Return type

Any

Raises

• ValueError – If the sleep generator stops yielding values.

• Exception – a custom exception specified by the exception_factory if provided. If no exception_factory is provided: google.api_core.RetryError: If the timeout is exceeded while retrying. Exception: If the target raises an error that isn’t retryable.

asyncgoogle.api_core.retry.retry_target_async(target:Callable[[],Awaitable[_R]],predicate:Callable[[Exception],bool],sleep_generator:Iterable[float],timeout:float|None=None,on_error:Callable[[Exception],None]|None=None,exception_factory:Callable[[list[Exception],RetryFailureReason,float|None],tuple[Exception,Exception|None]]=<functionbuild_retry_error>,**kwargs)¶

Await a coroutine and retry if it fails.

This is the lowest-level retry helper. Generally, you’ll use thehigher-level retry helperRetry.

Parameters

• target (Callable[[],Any]) – The function to call and retry. This must be anullary function - apply arguments withfunctools.partial.

• predicate (Callable[Exception]) – A callable used to determine if anexception raised by the target should be considered retryable.It should return True to retry or False otherwise.

• sleep_generator (Iterable[float]) – An infinite iterator that determineshow long to sleep between retries.

• timeout (Optional[float]) – How long to keep retrying the target, in seconds.Note: timeout is only checked before initiating a retry, so the target mayrun past the timeout value as long as it is healthy.

• on_error (Optional[Callable[Exception]]) – If given, the on_errorcallback will be called with each retryable exception raised by thetarget. Any error raised by this function willnot be caught.

• exception_factory – A function that is called when the retryable reachesa terminal failure state, used to construct an exception to be raised.It takes a list of all exceptions encountered, a retry.RetryFailureReasonenum indicating the failure cause, and the original timeout valueas arguments. It should return a tuple of the exception to be raised,along with the cause exception if any. The default implementation will raisea RetryError on timeout, or the last exception encountered otherwise.

• deadline (float) – DEPRECATED usetimeout instead. For backwardcompatibility, if set it will override thetimeout parameter.

Returns

the return value of the target function.

Return type

Any

Raises

• ValueError – If the sleep generator stops yielding values.

• Exception – a custom exception specified by the exception_factory if provided. If no exception_factory is provided: google.api_core.RetryError: If the timeout is exceeded while retrying. Exception: If the target raises an error that isn’t retryable.

google.api_core.retry.retry_target_stream(target:Callable[_P,Iterable[_Y]],predicate:Callable[[Exception],bool],sleep_generator:Iterable[float],timeout:Optional[float]=None,on_error:Optional[Callable[[Exception],None]]=None,exception_factory:Callable[[List[Exception],RetryFailureReason,Optional[float]],Tuple[Exception,Optional[Exception]]]=<functionbuild_retry_error>,init_args:tuple=(),init_kwargs:dict={},**kwargs)→Generator[_Y,Any,None][source]¶

Create a generator wrapper that retries the wrapped stream if it fails.

This is the lowest-level retry helper. Generally, you’ll use thehigher-level retry helperRetry.

Parameters

• target – The generator function to call and retry.

• predicate – A callable used to determine if anexception raised by the target should be considered retryable.It should return True to retry or False otherwise.

• sleep_generator – An infinite iterator that determineshow long to sleep between retries.

• timeout – How long to keep retrying the target.Note: timeout is only checked before initiating a retry, so the target mayrun past the timeout value as long as it is healthy.

• on_error – If given, the on_error callback will be called with eachretryable exception raised by the target. Any error raised by thisfunction willnot be caught.

• exception_factory – A function that is called when the retryable reachesa terminal failure state, used to construct an exception to be raised.It takes a list of all exceptions encountered, a retry.RetryFailureReasonenum indicating the failure cause, and the original timeout valueas arguments. It should return a tuple of the exception to be raised,along with the cause exception if any. The default implementation will raisea RetryError on timeout, or the last exception encountered otherwise.

• init_args – Positional arguments to pass to the target function.

• init_kwargs – Keyword arguments to pass to the target function.

Returns

A retryable generator that wraps the target generator function.

Return type

Generator

Raises

• ValueError – If the sleep generator stops yielding values.

• Exception – a custom exception specified by the exception_factory if provided. If no exception_factory is provided: google.api_core.RetryError: If the timeout is exceeded while retrying. Exception: If the target raises an error that isn’t retryable.

asyncgoogle.api_core.retry.retry_target_stream_async(target:Callable[_P,AsyncIterable[_Y]|Awaitable[AsyncIterable[_Y]]],predicate:Callable[[Exception],bool],sleep_generator:Iterable[float],timeout:float|None=None,on_error:Callable[[Exception],None]|None=None,exception_factory:Callable[[list[Exception],RetryFailureReason,float|None],tuple[Exception,Exception|None]]=<functionbuild_retry_error>,init_args:tuple=(),init_kwargs:dict={},**kwargs)→AsyncGenerator[_Y,None]¶

Create a generator wrapper that retries the wrapped stream if it fails.

This is the lowest-level retry helper. Generally, you’ll use thehigher-level retry helperAsyncRetry.

Parameters

• target – The generator function to call and retry.

• predicate – A callable used to determine if anexception raised by the target should be considered retryable.It should return True to retry or False otherwise.

• sleep_generator – An infinite iterator that determineshow long to sleep between retries.

• timeout – How long to keep retrying the target.Note: timeout is only checked before initiating a retry, so the target mayrun past the timeout value as long as it is healthy.

• on_error – If given, the on_error callback will be called with eachretryable exception raised by the target. Any error raised by thisfunction willnot be caught.

• exception_factory – A function that is called when the retryable reachesa terminal failure state, used to construct an exception to be raised.It takes a list of all exceptions encountered, a retry.RetryFailureReasonenum indicating the failure cause, and the original timeout valueas arguments. It should return a tuple of the exception to be raised,along with the cause exception if any. The default implementation will raisea RetryError on timeout, or the last exception encountered otherwise.

• init_args – Positional arguments to pass to the target function.

• init_kwargs – Keyword arguments to pass to the target function.

Returns

A retryable generator that wraps the target generator function.

Return type

AsyncGenerator

Raises

• ValueError – If the sleep generator stops yielding values.

• Exception – a custom exception specified by the exception_factory if provided. If no exception_factory is provided: google.api_core.RetryError: If the timeout is exceeded while retrying. Exception: If the target raises an error that isn’t retryable.