Uh oh!
There was an error while loading.Please reload this page.
- Notifications
You must be signed in to change notification settings - Fork32.4k
Open
Description
Bug report
Bug description:
Code:
fromconcurrent.futuresimportThreadPoolExecutordata= [list(range(0,5)),list(range(5,10)),]def_f(x):print(f"Processing{x}")returnTrueprint("=== 1 - No consumption from the iterator ===")executor=ThreadPoolExecutor(max_workers=1)forintsindata:executor.map(_f,ints)executor.shutdown(wait=True)print("=== 2 - Consume all values from the iterator ===")executor=ThreadPoolExecutor(max_workers=1)forintsindata:futures=executor.map(_f,ints)results=list(futures)executor.shutdown(wait=True)print("=== 3 - Consume one value from the iterator ===")executor=ThreadPoolExecutor(max_workers=1)forintsindata:futures=executor.map(_f,ints)first=next(futures)executor.shutdown(wait=True)print("=== 4 - Dropping iterator cancels remaining futures ===")executor=ThreadPoolExecutor(max_workers=1)futures=executor.map(_f,range(0,5))first=next(futures)delfuturesexecutor.shutdown(wait=True)
Result:
=== 1 - No consumption from the iterator ===Processing 0Processing 1Processing 2Processing 3Processing 4Processing 5Processing 6Processing 7Processing 8Processing 9=== 2 - Consume all values from the iterator ===Processing 0Processing 1Processing 2Processing 3Processing 4Processing 5Processing 6Processing 7Processing 8Processing 9=== 3 - Consume one value from the iterator ===Processing 0Processing 1Processing 5Processing 6Processing 7Processing 8Processing 9=== 4 - Dropping iterator cancels remaining futures ===Processing 0Processing 1
The behaviour seems to be:
- If the iterator returned from
map
is never used (case 1), futures are not cancelled - If the iterator returned from
map
is exhausted (case 2), futures are not cancelled - If the iterator returned from
map
ispartially consumed and then dropped (cases 3 & 4), the remaining futures are cancelled
We hit this doing a version of case 3, callingany
on the iterator, which short-circuited, causing the remaining futures to not execute. This tripped us up and seems like quite a confusing behaviour that is not flagged in the docs.
It looks like this is caused by this code:https://github.com/python/cpython/blob/main/Lib/concurrent/futures/_base.py#L669-L671
Possibly related to#108518
CPython versions tested on:
3.12
Operating systems tested on:
Linux