- Notifications
You must be signed in to change notification settings - Fork95
Description
Is your feature request related to a problem? Please describe.
In BigQuery, there are some operations like "start a query" that can actually take multiple API requests. It's usually justjobs.insert, but if there's a conflict, then we also make calls tojobs.get. In addition, these have different predicates, such as wanting to retryjobs.get on 404 in this context but not others.
Describe the solution you'd like
Ideally, I could take a user-provided generic retry and tweak if for the given scenario. For example:
get_job_retry=retry.with_predicate(lambdaexc:isinstance(exc,core_exceptions.NotFound)# Reference the original retry to avoid recursion.orretry._predicate(exc))
or
defdo_query_predicate(exc)->bool:ifisinstance(exc,core_exceptions.RetryError):exc=exc.cause# Per https://github.com/googleapis/python-bigquery/issues/2134, sometimes# we get a 404 error. In this case, if we get this far, assume that the job# doesn't actually exist and try again. We can't add 404 to the default# job_retry because that happens for errors like "this table does not# exist", which probably won't resolve with a retry.ifisinstance(exc,core_exceptions.NotFound):message=exc.message# Don't try to retry table/dataset not found, just job not found.# The URL contains jobs, so use whitespace to disambiguate.returnmessageisnotNoneand" job"inmessage.lower()# Reference the original job_retry to avoid recursion.returnjob_retry._predicate(exc))
Describe alternatives you've considered
- Just as I already have two retry objects in the parameters for the
query()API, I could update it to have two more for these specific situations. - We could also add a method
with_additional_predicatethat adds anorclause + this function to the previous predicate.
Additional context
googleapis/python-bigquery#2135 is an issue where I will likely add more "retry" parameters for finer-grained control, but would have preferred taking a user's retry object and amending the predicate.