Uh oh!
There was an error while loading.Please reload this page.
- Notifications
You must be signed in to change notification settings - Fork11.1k
Description
The use of the caching memory allocator in numpy with multiple threads seems to result in some buggy behavior. It is not clear if this is due to the cacheing allocator methods being invoked from places where GIL is not held or due to other issues. When we upgraded from numpy 1.13.3 to 1.14.0 we noticed these issues, likely due to PRs such as#8920.
We also identified such commits as potentially problematic:
git diff 1108252...5a08e209863038b411bc506893093c5a4b8377c9 --stat doc/source/reference/c-api.array.rst | 8 ++++---- numpy/core/src/multiarray/alloc.h | 12 ++++++++++++ numpy/core/src/multiarray/arrayobject.c | 8 ++++---- numpy/core/src/multiarray/arraytypes.c.src | 8 ++++---- numpy/core/src/multiarray/compiled_base.c | 9 +++++---- numpy/core/src/multiarray/conversion_utils.c | 5 +++-- numpy/core/src/multiarray/descriptor.c | 21 +++++++++++---------- numpy/core/src/multiarray/dtype_transfer.c | 23 ++++++++++++----------- numpy/core/src/multiarray/getset.c | 11 ++++++----- numpy/core/src/multiarray/item_selection.c | 31 ++++++++++++++++--------------- numpy/core/src/multiarray/methods.c | 17 +++++++++-------- numpy/core/src/multiarray/multiarraymodule.c | 12 ++++++------ numpy/core/src/multiarray/nditer_pywrap.c | 9 +++++---- numpy/core/src/multiarray/scalartypes.c.src | 18 ++++++++---------- numpy/core/src/multiarray/shape.c | 3 ++- 15 files changed, 107 insertions(+), 88 deletions(-)This issue seems to be still be present in the latest master.
For local testing we were able to resolve issues by making the allocator objects into Thread Local Storage object by doing:
$ git diffdiff --git a/numpy/core/src/multiarray/alloc.c b/numpy/core/src/multiarray/alloc.cindex f8305d1..fa6b3c0 100644--- a/numpy/core/src/multiarray/alloc.c+++ b/numpy/core/src/multiarray/alloc.c@@ -32,8 +32,8 @@ typedef struct { npy_uintp available; /* number of cached pointers */ void * ptrs[NCACHE]; } cache_bucket;-static cache_bucket datacache[NBUCKETS];-static cache_bucket dimcache[NBUCKETS_DIM];+static __thread cache_bucket datacache[NBUCKETS];+static __thread cache_bucket dimcache[NBUCKETS_DIM]; /*Although it is doubtful that this solution would be viable for all supported platforms as __thread may not be supported for some of them. This was rather to verify that the allocator cache was improperly accessed across threads.
Reproducing code example:
The following attachment contains what is needed to reproduce this problem. Unfortunately it does not occur every time, so it may need to be run several times before it occurs. The number of cores or speed of the machine used may also make a difference in how easy it is to reproduce. When it does occur, it will be either a segfault or an IndexError.