- Notifications
You must be signed in to change notification settings - Fork90
Description
Is your feature request related to a problem? Please describe.
In BigQuery, there are some operations like "start a query" that can actually take multiple API requests. It's usually justjobs.insert
, but if there's a conflict, then we also make calls tojobs.get
. In addition, these have different predicates, such as wanting to retryjobs.get
on 404 in this context but not others.
Describe the solution you'd like
Ideally, I could take a user-provided generic retry and tweak if for the given scenario. For example:
get_job_retry=retry.with_predicate(lambdaexc:isinstance(exc,core_exceptions.NotFound)# Reference the original retry to avoid recursion.orretry._predicate(exc))
or
defdo_query_predicate(exc)->bool:ifisinstance(exc,core_exceptions.RetryError):exc=exc.cause# Per https://github.com/googleapis/python-bigquery/issues/2134, sometimes# we get a 404 error. In this case, if we get this far, assume that the job# doesn't actually exist and try again. We can't add 404 to the default# job_retry because that happens for errors like "this table does not# exist", which probably won't resolve with a retry.ifisinstance(exc,core_exceptions.NotFound):message=exc.message# Don't try to retry table/dataset not found, just job not found.# The URL contains jobs, so use whitespace to disambiguate.returnmessageisnotNoneand" job"inmessage.lower()# Reference the original job_retry to avoid recursion.returnjob_retry._predicate(exc))
Describe alternatives you've considered
- Just as I already have two retry objects in the parameters for the
query()
API, I could update it to have two more for these specific situations. - We could also add a method
with_additional_predicate
that adds anor
clause + this function to the previous predicate.
Additional context
googleapis/python-bigquery#2135 is an issue where I will likely add more "retry" parameters for finer-grained control, but would have preferred taking a user's retry object and amending the predicate.