Can't use lock around gather in Python asyncio

Question 1

For some reason, this code doesn't work on Ubuntu 20, Python 3.8.10,unless the.gather line is commented out.It works on Ubuntu 24, Python 3.12.3 and Windows 11, Python 3.13.9.It doesn't work on Python 3.8/3.9 on the same Ubuntu 24 machine. It seems to work from Python 3.10 onwards, so the issue appears to be one with the Python version.

This code is a minimum working example of the issue that I'm facing.Assume that the locks are actually necessary.

import asyncioimport timelock = asyncio.Lock()async def action():    print("action")    returnasync def task1():    while True:        print("1 waiting")        await lock.acquire()        print("1 running")        await asyncio.gather(*[action() for _ in range(1)])        lock.release()        print("1 unlocked")        print("A", time.time())        await asyncio.sleep(1)        print("B", time.time())async def task2():    while True:        print("2 waiting")        await lock.acquire()        print("2 running")        lock.release()        print("2 unlocked")        await asyncio.sleep(1)async def main():    task1_ = asyncio.create_task(task1())    task2_ = asyncio.create_task(task2())    while True: await asyncio.sleep(float('inf'))if __name__ == "__main__":    asyncio.run(main())

The output on one of the Python versions that works is:

$ python3 test.py1 waiting1 running2 waitingaction1 unlockedA 1763606395.83234982 running2 unlockedB 1763606396.8438531 waiting1 running2 waitingaction1 unlockedA 1763606396.84516932 running2 unlockedB 1763606397.8484037

Notice that task2 gets to run between A and B (while task1 is asleep).

The output of the non-working versions is:

$ python3 test.py1 waiting1 running2 waitingaction1 unlockedA 1763606434.4998655B 1763606435.50107381 waiting1 runningaction1 unlockedA 1763606435.5020432B 1763606436.50312641 waiting1 runningaction1 unlockedA 1763606436.5039277B 1763606437.5050313

As you can see, task2 doesn't run between A and B.

This issue occurs regardless of usingawait lock.acquire() orasync with lock:.If I comment out the.gather line, it works on all versions.However, I need to run multiple tasks at the same time, so I can't simply comment that out.I haven't been able to find anything in the docs that suggests an issue with gather or locks in older versions of asyncio.So what's going on here, and how do I fix it?

Question 2

The fact that the code works as expected on Python >=3.10 isa side effect of removing theloop parameter.

Prior to Python 3.10,asyncio.Lock was bound to the current event loop at initialization time^[1]. That is, in your case, outside ofasyncio.run(). As a result, on Python <3.10, future objects are created on behalf of the wrong event loop^[2], and attempting to use them results in aRuntimeError. Because of this, your second task failed (since the lock was already held by the first task), and the only active task was the first one.

import asynciolock = asyncio.Lock()async def test():    await lock.acquire()  # actually non-blocking call    await asyncio.wait_for(lock.acquire(), 1e-3)  # blocking call (deadlock)    # `RuntimeError` on Python <3.10    # `TimeoutError` on Python >=3.10asyncio.run(test())

Starting with Python 3.10,asyncio.Lock binds to the current event loop at the first blocking call^[3], and therefore it creates future objects on behalf of the right event loop. If you want to support Python <3.10, do not create asyncio synchronization primitives at the module level or anywhere else outside the event loop.

The reason why you did not find the real cause of the problem is that you useasyncio.create_task() to create new tasks and do not handle their failures in any way. Exceptions from tasks that no one is waiting for are only logged when they are deleted^[4]^[5]. And since you refer to them in themain() function, they will not be deleted until it finishes its execution (for example, by Control-C).

As for removing the line withasyncio.gather(), this simply means that no context switching occurs between acquiring the lock and releasing it, and thus no task ever sees the lock in the locked state. This is because asyncio implements cooperative multitasking, and you can read more about this eitherin the documentation orin the answers to a related question.

If you want to work safely with tasks, consider usingasyncio.TaskGroup. There is alsoa backport to Python <3.11.

A quick way to verify the cause is to add the following lines to the beginning of themain() function:

global locklock = asyncio.Lock()

With this change, your code works as expected on Python <3.10.

Question 3

Nice answer. BTW,asyncio.Queue was doing the same in Python <= 3.9, i.e. creating a loop when instantiated.

Question 4

Thank you. This was a great answer. The only thing I'm a bit confused about is "you useasyncio.create_task() to create new tasks and do not handle their failures in any way". How should I do it properly? Is that whatTaskGroup is used for?

Question 5

Yes, of course. It raises exceptions for all failed tasks usingExceptionGroup, so you will not lose debugging information when using it. In fact, you should just wait for the tasks you create to complete, but there are pitfalls here, and when usingasyncio.gather(), you will only get an exception for the first failed task (all others will be suppressed), so it is better to rely on modern structured concurrency.

Ilya Egorov 5211 gold badge3 silver badges12 bronze badges · Accepted Answer · 2025-11-20 11:50:09Z

The fact that the code works as expected on Python >=3.10 isa side effect of removing theloop parameter.

Prior to Python 3.10,asyncio.Lock was bound to the current event loop at initialization time^[1]. That is, in your case, outside ofasyncio.run(). As a result, on Python <3.10, future objects are created on behalf of the wrong event loop^[2], and attempting to use them results in aRuntimeError. Because of this, your second task failed (since the lock was already held by the first task), and the only active task was the first one.

import asynciolock = asyncio.Lock()async def test():    await lock.acquire()  # actually non-blocking call    await asyncio.wait_for(lock.acquire(), 1e-3)  # blocking call (deadlock)    # `RuntimeError` on Python <3.10    # `TimeoutError` on Python >=3.10asyncio.run(test())

Starting with Python 3.10,asyncio.Lock binds to the current event loop at the first blocking call^[3], and therefore it creates future objects on behalf of the right event loop. If you want to support Python <3.10, do not create asyncio synchronization primitives at the module level or anywhere else outside the event loop.

The reason why you did not find the real cause of the problem is that you useasyncio.create_task() to create new tasks and do not handle their failures in any way. Exceptions from tasks that no one is waiting for are only logged when they are deleted^[4]^[5]. And since you refer to them in themain() function, they will not be deleted until it finishes its execution (for example, by Control-C).

As for removing the line withasyncio.gather(), this simply means that no context switching occurs between acquiring the lock and releasing it, and thus no task ever sees the lock in the locked state. This is because asyncio implements cooperative multitasking, and you can read more about this eitherin the documentation orin the answers to a related question.

If you want to work safely with tasks, consider usingasyncio.TaskGroup. There is alsoa backport to Python <3.11.

A quick way to verify the cause is to add the following lines to the beginning of themain() function:

global locklock = asyncio.Lock()

With this change, your code works as expected on Python <3.10.

Nice answer. BTW,asyncio.Queue was doing the same in Python <= 3.9, i.e. creating a loop when instantiated.
Thank you. This was a great answer. The only thing I'm a bit confused about is "you useasyncio.create_task() to create new tasks and do not handle their failures in any way". How should I do it properly? Is that whatTaskGroup is used for?
Yes, of course. It raises exceptions for all failed tasks usingExceptionGroup, so you will not lose debugging information when using it. In fact, you should just wait for the tasks you create to complete, but there are pitfalls here, and when usingasyncio.gather(), you will only get an exception for the first failed task (all others will be suppressed), so it is better to rely on modern structured concurrency.

Movatterモバイル変換

Collectives™ on Stack Overflow

Can't use lock around gather in Python asyncio

1 Answer1

3 Comments

Your Answer

Sign up orlog in

Post as a guest

Linked

Related

Hot Network Questions

Subscribe to RSS