No matter how much I search (or maybe I'm searching incorrectly), I can't find a decent way to terminate my tasks in Python AsyncIO.
I know what:
- I need to bind to signals (SIGINT,SIGTERM) usingloop.add_signal_handler(signum, lambda: asyncio.create_task(shutdown())) on Unix,signal.signal(signum, lambda *args: loop.call_soon_threadsafe(shutdown, args)) on Windows.
In shutdown:
- I need to get all tasks (asyncio.all_tasks()), cancel them, and wait.
- Close the connection, and do additional cleanup.
-loop.shutdown_asyncgens()
- Stop the loop.
The problem is that if I receive all tasks, this could affect, for example, tasks used internally by the library (like aiokafka), and when waiting for tasks to be canceled, it gets stuck in an infinite wait.
If I close before the tasks are completed, that's fine, but the problem is that if I close the connection before all tasks are canceled, the tasks won't be able to complete gracefully.
I have the following questions:
- Is the only option to manually collect all the tasks and cancel them?
- And for example, if I have aTaskGroup, should I terminate it specifically, or its subtasks?
- And there seems to be a problem withloop.set_exception_handler, which should simply call shutdown, otherwise the tasks might also get stuck, am I understanding this correctly?
I also saw that there's an option with shutdown_event, but I don't think that's suitable because if I have tasks that useawait queue.get(), they might get stuck.
2 Replies2
The question is: why do you need to explicitly cancel the tasks? How long will you even wait for their "graceful" shutdown (whatever that may entail)?
When the process exits in any fashion, the operating system will automatically close all its files, network connections, etc. In many situations the process can die without any warning, such as when killed by OOM,SIGKILL, or when the machine loses power. So your program and its peers must handle that situation. There is rarely a real need to complicate code by explicitly handling graceful shutdown as well.
•Overall, I want to implement a graceful shutdown for proper process handling, at least to send what's already been processed and set statuses so it doesn't have to be reprocessed on subsequent restarts. Ideally, this shouldn't take more than 5 seconds. I'm also just wondering what the best way to do this is. Many people simply use asyncio.all_tasks and cancel them, but that's fine if you don't have other libraries and only work with the tasks you're currently running.
I work within K8s, so I don't think there will be any SIGKILLs (they usually give 30 seconds for a graceful shutdown) or power outages.
Explore related questions
See similar questions with these tags.