I have a piece of code in Python that seems to cause an error probabilistically because it is accessing a server and sometimes that server has a 500 internal server error. I want to keep trying until I do not get the error. My solution was:
while True: try: #code with possible error except: continue else: #the rest of the code breakThis seems like a hack to me. Is there a more Pythonic way to do this?
- 8Err... what happens when the remote server dies? Will this sit there consuming 100% of a CPU core?user9876– user98762011-01-05 17:15:50 +00:00CommentedJan 5, 2011 at 17:15
- continue should be in else and break in except. Is it a typo?Ankit Jaiswal– Ankit Jaiswal2011-01-05 17:17:03 +00:00CommentedJan 5, 2011 at 17:17
- 3@aand: No. If an exception occurs, he wants to try again (read:
continue), but if no exception occurs, he wants to to some things (sketched out by a comment) and get outta that weird abuse of a loop. (elseexecutes if no exception occurs, is that the missing piece?)user395760– user3957602011-01-05 17:24:58 +00:00CommentedJan 5, 2011 at 17:24 - 2Word of warning: you should add some degree of protection against persistent failures, such as a sleep or a maximum attempt limit. Otherwise, your process will consume CPU indefinitely, as @user9876 noted.Brad Koch– Brad Koch2014-05-30 18:06:34 +00:00CommentedMay 30, 2014 at 18:06
- 2for what it's worth there are less scary examples of the same requirement. I'm trying to create a directory with a new name, so I will append a number to a base and increment that number until I succeed in making the the directory. I will only catch the AlreadyExists error. Unless my folder has infinite directories this will clearly succeed sooner or later.Clumsy cat– Clumsy cat2019-06-12 17:32:24 +00:00CommentedJun 12, 2019 at 17:32
12 Answers12
It won't get much cleaner. This is not a very clean thing to do. At best (which would be more readable anyway, since the condition for thebreak is up there with thewhile), you could create a variableresult = None and loop while itis None. You should also adjust the variables and you can replacecontinue with the semantically perhaps correctpass (you don't care if an error occurs, you just want to ignore it) and drop thebreak - this also gets the rest of the code, which only executes once, out of the loop. Also note that bareexcept: clauses are evil for reasonsgiven in the documentation.
Example incorporating all of the above:
result = Nonewhile result is None: try: # connect result = get_data(...) except: pass# other code that uses result but is not involved in getting it5 Comments
Here is one that hard fails after 4 attempts, and waits 2 seconds between attempts. Change as you wish to get what you want form this one:
from time import sleepfor x in range(0, 4): # try 4 times try: # msg.send() # put your logic here str_error = None except Exception as str_error: pass if str_error: sleep(2) # wait for 2 seconds before trying to fetch the data again else: breakHere is an example with backoff:
from time import sleepsleep_time = 2num_retries = 4for x in range(0, num_retries): try: # put your logic here str_error = None except Exception as e: str_error = str(e) if str_error: sleep(sleep_time) # wait before trying to fetch the data again sleep_time *= 2 # Implement your backoff algorithm here i.e. exponential backoff else: break3 Comments
IndentationError forif str_errorUnboundLocalError: local variable 'error_str' referenced before assignment and modified the except lineexcept Exception as e: and addedstr_error=e beforepass to fix thisMaybe something like this:
connected = Falsewhile not connected: try: try_connect() connected = True except ...: passWhen retrying due to error, you should always:
- implement a retry limit, or you may get blocked on an infinite loop
- implement a delay, or you'll hammer resources too hard, such as your CPU or the already distressed remote server
A simple generic way to solve this problem while covering those concerns would be to use thebackoff library. A basic example:
import backoff@backoff.on_exception( backoff.expo, MyException, max_tries=5)def make_request(self, data): # do the requestThis code wraps make_request with a decorator which implements the retry logic. We retry whenever our specific errorMyException occurs, with a limit of 5 retries.Exponential backoff is a good idea in this context to help minimize the additional burden our retries place on the remote server.
Comments
Theitertools.iter_except recipes encapsulates this idea of "calling a function repeatedly until an exception is raised". It is similar to the accepted answer, but the recipe gives an iterator instead.
From the recipes:
def iter_except(func, exception, first=None): """ Call a function repeatedly until an exception is raised.""" try: if first is not None: yield first() # For database APIs needing an initial cast to db.first() while True: yield func() except exception: passYou can certainly implement the latter code directly. For convenience, I use a separate library,more_itertools, that implements this recipe for us (optional).
Code
import more_itertools as mitlist(mit.iter_except([0, 1, 2].pop, IndexError))# [2, 1, 0]Details
Here thepop method (or given function) is called for every iteration of the list object until anIndexError is raised.
For your case, given someconnect_function and expected error, you can make an iterator that calls the function repeatedly until an exception is raised, e.g.
mit.iter_except(connect_function, ConnectionError)At this point, treat it as any other iterator by looping over it or callingnext().
1 Comment
Maybe decorator based?You can pass as decorator arguments list of exceptions on which we want to retry and/or number of tries.
def retry(exceptions=None, tries=None): if exceptions: exceptions = tuple(exceptions) def wrapper(fun): def retry_calls(*args, **kwargs): if tries: for _ in xrange(tries): try: fun(*args, **kwargs) except exceptions: pass else: break else: while True: try: fun(*args, **kwargs) except exceptions: pass else: break return retry_calls return wrapperfrom random import randint@retry([NameError, ValueError])def foo(): if randint(0, 1): raise NameError('FAIL!') print 'Success'@retry([ValueError], 2)def bar(): if randint(0, 1): raise ValueError('FAIL!') print 'Success'@retry([ValueError], 2)def baz(): while True: raise ValueError('FAIL!')foo()bar()baz()of course the 'try' part should be moved to another funcion becouse we using it in both loops but it's just example;)
1 Comment
Here's an utility function that I wrote to wrap the retry until success into a neater package. It uses the same basic structure, but prevents repetition. It could be modified to catch and rethrow the exception on the final try relatively easily.
def try_until(func, max_tries, sleep_time): for _ in range(0,max_tries): try: return func() except: sleep(sleep_time) raise WellNamedException() #could be 'return sensibleDefaultValue'Can then be called like this
result = try_until(my_function, 100, 1000)If you need to pass arguments tomy_function, you can either do this by havingtry_until forward the arguments, or by wrapping it in a no argument lambda:
result = try_until(lambda : my_function(x,y,z), 100, 1000)Comments
Like most of the others, I'd recommend trying a finite number of times and sleeping between attempts. This way, you don't find yourself in an infinite loop in case something were to actually happen to the remote server.
I'd also recommend continuing only when you get the specific exception you're expecting. This way, you can still handle exceptions you might not expect.
from urllib.error import HTTPErrorimport tracebackfrom time import sleepattempts = 10while attempts > 0: try: #code with possible error except HTTPError: attempts -= 1 sleep(1) continue except: print(traceback.format_exc()) #the rest of the code breakAlso, you don't need an else block. Because of the continue in the except block, you skip the rest of the loop until the try block works, the while condition gets satisfied, or an exception other than HTTPError comes up.
Comments
what about theretrying library on pypi?I have been using it for a while and it does exactly what I want and more (retry on error, retry when None, retry with timeout). Below is example from their website:
import randomfrom retrying import retry@retrydef do_something_unreliable(): if random.randint(0, 10) > 1: raise IOError("Broken sauce, everything is hosed!!!111one") else: return "Awesome sauce!"print do_something_unreliable()Comments
e = ''while e == '': try: response = ur.urlopen('https://https://raw.githubusercontent.com/MrMe42/Joe-Bot-Home-Assistant/mac/Joe.py') e = ' ' except: print('Connection refused. Retrying...') time.sleep(1)This should work. It sets e to '' and the while loop checks to see if it is still ''. If there is an error caught be the try statement, it prints that the connection was refused, waits 1 second and then starts over. It will keep going until there is no error in try, which then sets e to ' ', which kills the while loop.
Comments
Im attempting this now, this is what i came up with;
placeholder = 1 while placeholder is not None: try: #Code placeholder = None except Exception as e: print(str(datetime.time(datetime.now()))[:8] + str(e)) #To log the errors placeholder = e time.sleep(0.5) continueComments
Here is a short piece of code I use to capture the error as a string. Will retry till it succeeds. This catches all exceptions but you can change this as you wish.
start = 0str_error = "Not executed yet."while str_error: try: # replace line below with your logic , i.e. time out, max attempts start = raw_input("enter a number, 0 for fail, last was {0}: ".format(start)) new_val = 5/int(start) str_error=None except Exception as str_error: passWARNING: This code will be stuck in a forever loop until no exception occurs. This is just a simple example and MIGHT require you to break out of the loop sooner or sleep between retries.
Comments
Explore related questions
See similar questions with these tags.








