136

I have a piece of code in Python that seems to cause an error probabilistically because it is accessing a server and sometimes that server has a 500 internal server error. I want to keep trying until I do not get the error. My solution was:

while True:    try:        #code with possible error    except:         continue    else:         #the rest of the code         break

This seems like a hack to me. Is there a more Pythonic way to do this?

askedJan 5, 2011 at 17:08
murgatroid99's user avatar
7
  • 8
    Err... what happens when the remote server dies? Will this sit there consuming 100% of a CPU core?CommentedJan 5, 2011 at 17:15
  • continue should be in else and break in except. Is it a typo?CommentedJan 5, 2011 at 17:17
  • 3
    @aand: No. If an exception occurs, he wants to try again (read:continue), but if no exception occurs, he wants to to some things (sketched out by a comment) and get outta that weird abuse of a loop. (else executes if no exception occurs, is that the missing piece?)CommentedJan 5, 2011 at 17:24
  • 2
    Word of warning: you should add some degree of protection against persistent failures, such as a sleep or a maximum attempt limit. Otherwise, your process will consume CPU indefinitely, as @user9876 noted.CommentedMay 30, 2014 at 18:06
  • 2
    for what it's worth there are less scary examples of the same requirement. I'm trying to create a directory with a new name, so I will append a number to a base and increment that number until I succeed in making the the directory. I will only catch the AlreadyExists error. Unless my folder has infinite directories this will clearly succeed sooner or later.CommentedJun 12, 2019 at 17:32

12 Answers12

134

It won't get much cleaner. This is not a very clean thing to do. At best (which would be more readable anyway, since the condition for thebreak is up there with thewhile), you could create a variableresult = None and loop while itis None. You should also adjust the variables and you can replacecontinue with the semantically perhaps correctpass (you don't care if an error occurs, you just want to ignore it) and drop thebreak - this also gets the rest of the code, which only executes once, out of the loop. Also note that bareexcept: clauses are evil for reasonsgiven in the documentation.

Example incorporating all of the above:

result = Nonewhile result is None:    try:        # connect        result = get_data(...)    except:         pass# other code that uses result but is not involved in getting it
ReinstateMonica3167040's user avatar
ReinstateMonica3167040
81011 silver badges31 bronze badges
answeredJan 5, 2011 at 17:12
Sign up to request clarification or add additional context in comments.

5 Comments

If there is a persistent reason that the connection is not succeeding, this solution will be grinding away in an infinite loop.
@BradKoch Of course. That is inherent to the question, and furthermore any fix (such as an overall timeout or a limited number of tries) is relatively orthogonal to the changes I describe.
But any proposed answer should be safe, or at least note the pitfalls. This does not offer protection against 100% CPU consumption and endangers future readers.
@BradKoch what would you suggest to stop against 100% cpu usage, try perhaps 5-10 times before ending the loop?
@Datanovice Yes, a simple retry limit with delay is a good solution. Many of the other answers demonstrate how to do this. For complicated scenarios like retrying HTTP requests, I'd personallyrecommend using a library.
61

Here is one that hard fails after 4 attempts, and waits 2 seconds between attempts. Change as you wish to get what you want form this one:

from time import sleepfor x in range(0, 4):  # try 4 times    try:        # msg.send()        # put your logic here        str_error = None    except Exception as str_error:        pass    if str_error:        sleep(2)  # wait for 2 seconds before trying to fetch the data again    else:        break

Here is an example with backoff:

from time import sleepsleep_time = 2num_retries = 4for x in range(0, num_retries):      try:        # put your logic here        str_error = None    except Exception as e:        str_error = str(e)    if str_error:        sleep(sleep_time)  # wait before trying to fetch the data again        sleep_time *= 2  # Implement your backoff algorithm here i.e. exponential backoff    else:        break
answeredMay 30, 2014 at 18:33
radtek's user avatar

3 Comments

I like this answer better than others because this one is "nice" to other processes due to the sleep function and also has limited tries.
I'm gettingIndentationError forif str_error
I receivedUnboundLocalError: local variable 'error_str' referenced before assignment and modified the except lineexcept Exception as e: and addedstr_error=e beforepass to fix this
33

Maybe something like this:

connected = Falsewhile not connected:    try:        try_connect()        connected = True    except ...:        pass
answeredJan 5, 2011 at 17:12
mouad's user avatar

2 Comments

Any proposed answer should be safe, or at least note the pitfalls. This does not offer protection against 100% CPU consumption and endangers future readers.
This is a fantastic solution, you need to kill the terminal at least one time, get ready :D
12

When retrying due to error, you should always:

  • implement a retry limit, or you may get blocked on an infinite loop
  • implement a delay, or you'll hammer resources too hard, such as your CPU or the already distressed remote server

A simple generic way to solve this problem while covering those concerns would be to use thebackoff library. A basic example:

import backoff@backoff.on_exception(    backoff.expo,    MyException,    max_tries=5)def make_request(self, data):    # do the request

This code wraps make_request with a decorator which implements the retry logic. We retry whenever our specific errorMyException occurs, with a limit of 5 retries.Exponential backoff is a good idea in this context to help minimize the additional burden our retries place on the remote server.

answeredJun 9, 2019 at 19:22
Brad Koch's user avatar

Comments

5

Theitertools.iter_except recipes encapsulates this idea of "calling a function repeatedly until an exception is raised". It is similar to the accepted answer, but the recipe gives an iterator instead.

From the recipes:

def iter_except(func, exception, first=None):    """ Call a function repeatedly until an exception is raised."""    try:        if first is not None:            yield first()            # For database APIs needing an initial cast to db.first()        while True:            yield func()    except exception:        pass

You can certainly implement the latter code directly. For convenience, I use a separate library,more_itertools, that implements this recipe for us (optional).

Code

import more_itertools as mitlist(mit.iter_except([0, 1, 2].pop, IndexError))# [2, 1, 0]

Details

Here thepop method (or given function) is called for every iteration of the list object until anIndexError is raised.

For your case, given someconnect_function and expected error, you can make an iterator that calls the function repeatedly until an exception is raised, e.g.

mit.iter_except(connect_function, ConnectionError)

At this point, treat it as any other iterator by looping over it or callingnext().

answeredAug 23, 2017 at 17:23
pylang's user avatar

1 Comment

Thanks for pointing out iter_except. This answer would be even better if it implemented a backoff approach, something like below:stackoverflow.com/a/56517713/1231693
2

Maybe decorator based?You can pass as decorator arguments list of exceptions on which we want to retry and/or number of tries.

def retry(exceptions=None, tries=None):    if exceptions:        exceptions = tuple(exceptions)    def wrapper(fun):        def retry_calls(*args, **kwargs):            if tries:                for _ in xrange(tries):                    try:                        fun(*args, **kwargs)                    except exceptions:                        pass                    else:                        break            else:                while True:                    try:                        fun(*args, **kwargs)                    except exceptions:                        pass                    else:                        break        return retry_calls    return wrapperfrom random import randint@retry([NameError, ValueError])def foo():    if randint(0, 1):        raise NameError('FAIL!')    print 'Success'@retry([ValueError], 2)def bar():    if randint(0, 1):        raise ValueError('FAIL!')    print 'Success'@retry([ValueError], 2)def baz():    while True:        raise ValueError('FAIL!')foo()bar()baz()

of course the 'try' part should be moved to another funcion becouse we using it in both loops but it's just example;)

answeredJan 5, 2011 at 20:24
virhilo's user avatar

1 Comment

Bit of a late comment, but the doubling of the code could be avoided by using "for _ in itertools.repeat(None,times=tries):" If tries is None, the loop continues forever, but if tries is a number, it terminates after that many iterations.
2

Here's an utility function that I wrote to wrap the retry until success into a neater package. It uses the same basic structure, but prevents repetition. It could be modified to catch and rethrow the exception on the final try relatively easily.

def try_until(func, max_tries, sleep_time):    for _ in range(0,max_tries):        try:            return func()        except:            sleep(sleep_time)    raise WellNamedException()    #could be 'return sensibleDefaultValue'

Can then be called like this

result = try_until(my_function, 100, 1000)

If you need to pass arguments tomy_function, you can either do this by havingtry_until forward the arguments, or by wrapping it in a no argument lambda:

result = try_until(lambda : my_function(x,y,z), 100, 1000)
answeredNov 18, 2015 at 23:49
Harry Harrison's user avatar

Comments

2

Like most of the others, I'd recommend trying a finite number of times and sleeping between attempts. This way, you don't find yourself in an infinite loop in case something were to actually happen to the remote server.

I'd also recommend continuing only when you get the specific exception you're expecting. This way, you can still handle exceptions you might not expect.

from urllib.error import HTTPErrorimport tracebackfrom time import sleepattempts = 10while attempts > 0:    try:        #code with possible error    except HTTPError:        attempts -= 1        sleep(1)        continue    except:        print(traceback.format_exc())    #the rest of the code    break

Also, you don't need an else block. Because of the continue in the except block, you skip the rest of the loop until the try block works, the while condition gets satisfied, or an exception other than HTTPError comes up.

answeredMar 30, 2018 at 20:29
noname's user avatar

Comments

2

what about theretrying library on pypi?I have been using it for a while and it does exactly what I want and more (retry on error, retry when None, retry with timeout). Below is example from their website:

import randomfrom retrying import retry@retrydef do_something_unreliable():    if random.randint(0, 10) > 1:        raise IOError("Broken sauce, everything is hosed!!!111one")    else:        return "Awesome sauce!"print do_something_unreliable()
answeredOct 19, 2019 at 10:27
David Tam's user avatar

Comments

1
e = ''while e == '':    try:        response = ur.urlopen('https://https://raw.githubusercontent.com/MrMe42/Joe-Bot-Home-Assistant/mac/Joe.py')        e = ' '    except:        print('Connection refused. Retrying...')        time.sleep(1)

This should work. It sets e to '' and the while loop checks to see if it is still ''. If there is an error caught be the try statement, it prints that the connection was refused, waits 1 second and then starts over. It will keep going until there is no error in try, which then sets e to ' ', which kills the while loop.

answeredApr 2, 2018 at 14:50
user9311010's user avatar

Comments

1

Im attempting this now, this is what i came up with;

    placeholder = 1    while placeholder is not None:        try:            #Code            placeholder = None        except Exception as e:            print(str(datetime.time(datetime.now()))[:8] + str(e)) #To log the errors            placeholder = e            time.sleep(0.5)            continue
answeredJul 27, 2020 at 14:10
GrandadsJumper's user avatar

Comments

0

Here is a short piece of code I use to capture the error as a string. Will retry till it succeeds. This catches all exceptions but you can change this as you wish.

start = 0str_error = "Not executed yet."while str_error:    try:        # replace line below with your logic , i.e. time out, max attempts        start = raw_input("enter a number, 0 for fail, last was {0}: ".format(start))        new_val = 5/int(start)        str_error=None    except Exception as str_error:         pass

WARNING: This code will be stuck in a forever loop until no exception occurs. This is just a simple example and MIGHT require you to break out of the loop sooner or sleep between retries.

answeredNov 27, 2013 at 23:38
radtek's user avatar

Comments

Your Answer

Sign up orlog in

Sign up using Google
Sign up using Email and Password

Post as a guest

Required, but never shown

By clicking “Post Your Answer”, you agree to ourterms of service and acknowledge you have read ourprivacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.