9.3.1.1.Higher-level parallelism with joblib#

When the underlying implementation uses joblib, the number of workers(threads or processes) that are spawned in parallel can be controlled via then_jobs parameter.

Joblib is able to support both multi-processing and multi-threading. Whetherjoblib chooses to spawn a thread or a process depends on thebackendthat it’s using.

scikit-learn generally relies on theloky backend, which is joblib’sdefault backend. Loky is a multi-processing backend. When doingmulti-processing, in order to avoid duplicating the memory in each process(which isn’t reasonable with big datasets), joblib will create amemmapthat all processes can share, when the data is bigger than 1MB.

In some specific cases (when the code that is run in parallel releases theGIL), scikit-learn will indicate tojoblib that a multi-threadingbackend is preferable.

As a user, you may control the backend that joblib will use (regardless ofwhat scikit-learn recommends) by using a context manager:

fromjoblibimportparallel_backendwithparallel_backend('threading',n_jobs=2):# Your scikit-learn code here

Please refer to thejoblib’s docsfor more details.

In practice, whether parallelism is helpful at improving runtime depends onmany factors. It is usually a good idea to experiment rather than assumingthat increasing the number of workers is always a good thing. In some casesit can be highly detrimental to performance to run multiple copies of someestimators or functions in parallel (seeoversubscription below).

9.3.1.2.Lower-level parallelism with OpenMP#

OpenMP is used to parallelize code written in Cython or C, relying onmulti-threading exclusively. By default, the implementations using OpenMPwill use as many threads as possible, i.e. as many threads as logical cores.

You can control the exact number of threads that are used either:

• via theOMP_NUM_THREADS environment variable, for instance when:running a python script:

  OMP_NUM_THREADS=4pythonmy_script.py

• or viathreadpoolctl as explained bythis piece of documentation.

9.3.1.3.Parallel NumPy and SciPy routines from numerical libraries#

scikit-learn relies heavily on NumPy and SciPy, which internally callmulti-threaded linear algebra routines (BLAS & LAPACK) implemented in librariessuch as MKL, OpenBLAS or BLIS.

You can control the exact number of threads used by BLAS for each libraryusing environment variables, namely:

• MKL_NUM_THREADS sets the number of threads MKL uses,

• OPENBLAS_NUM_THREADS sets the number of threads OpenBLAS uses

• BLIS_NUM_THREADS sets the number of threads BLIS uses

Note that BLAS & LAPACK implementations can also be impacted byOMP_NUM_THREADS. To check whether this is the case in your environment,you can inspect how the number of threads effectively used by those librariesis affected when running the following command in a bash or zsh terminalfor different values ofOMP_NUM_THREADS:

OMP_NUM_THREADS=2python-mthreadpoolctl-inumpyscipy

9.3.1.4.Oversubscription: spawning too many threads#

It is generally recommended to avoid using significantly more processes orthreads than the number of CPUs on a machine. Over-subscription happens whena program is running too many threads at the same time.

Suppose you have a machine with 8 CPUs. Consider a case where you’re runningaGridSearchCV (parallelized with joblib)withn_jobs=8 over aHistGradientBoostingClassifier (parallelized withOpenMP). Each instance ofHistGradientBoostingClassifier will spawn 8 threads(since you have 8 CPUs). That’s a total of8*8=64 threads, whichleads to oversubscription of threads for physical CPU resources and thusto scheduling overhead.

Oversubscription can arise in the exact same fashion with parallelizedroutines from MKL, OpenBLAS or BLIS that are nested in joblib calls.

Starting fromjoblib>=0.14, when theloky backend is used (whichis the default), joblib will tell its childprocesses to limit thenumber of threads they can use, so as to avoid oversubscription. In practicethe heuristic that joblib uses is to tell the processes to usemax_threads=n_cpus//n_jobs, via their corresponding environment variable. Back toour example from above, since the joblib backend ofGridSearchCV isloky, each process willonly be able to use 1 thread instead of 8, thus mitigating theoversubscription issue.

Note that:

• Manually setting one of the environment variables (OMP_NUM_THREADS,MKL_NUM_THREADS,OPENBLAS_NUM_THREADS, orBLIS_NUM_THREADS)will take precedence over what joblib tries to do. The total number ofthreads will ben_jobs*<LIB>_NUM_THREADS. Note that setting thislimit will also impact your computations in the main process, which willonly use<LIB>_NUM_THREADS. Joblib exposes a context manager forfiner control over the number of threads in its workers (see joblib docslinked below).

• When joblib is configured to use thethreading backend, there is nomechanism to avoid oversubscriptions when calling into parallel nativelibraries in the joblib-managed threads.

• All scikit-learn estimators that explicitly rely on OpenMP in their Cython codealways usethreadpoolctl internally to automatically adapt the numbers ofthreads used by OpenMP and potentially nested BLAS calls so as to avoidoversubscription.

You will find additional details about joblib mitigation of oversubscriptioninjoblib documentation.

You will find additional details about parallelism in numerical python librariesinthis document from Thomas J. Fan.

9.3.2.1.Python API#

sklearn.set_config andsklearn.config_context can be used to changeparameters of the configuration which control aspect of parallelism.

9.3.2.2.Environment variables#

These environment variables should be set before importing scikit-learn.

SKLEARN_TESTS_GLOBAL_RANDOM_SEED="all"pytest-v-ktest_your_test_name