NCCL_DEBUG=WARNNCCL_SOCKET_IFNAME==ens1f0

NCCL_SOCKET_IFNAME¶

TheNCCL_SOCKET_IFNAME variable specifies which IP interfaces to use for communication.

NCCL_SOCKET_FAMILY¶

TheNCCL_SOCKET_FAMILY variable allows users to force NCCL to use only IPv4 or IPv6 interface.

NCCL_SOCKET_RETRY_CNT¶

(since 2.24)

TheNCCL_SOCKET_RETRY_CNT variable specifies the number of times NCCL retries to establish a socket connection after anETIMEDOUT,ECONNREFUSED, orEHOSTUNREACH error.

NCCL_SOCKET_RETRY_SLEEP_MSEC¶

(since 2.24)

TheNCCL_SOCKET_RETRY_SLEEP_MSEC variable specifies the number of milliseconds NCCL waits before retrying to establish a socket connection after the firstETIMEDOUT,ECONNREFUSED, orEHOSTUNREACH error.For subsequent errors, the waiting time scales linearly with the error count. The total time will therefore be (N+1) * N/2 *NCCL_SOCKET_RETRY_SLEEP_MSEC, where N is given byNCCL_SOCKET_RETRY_CNT.With the default values ofNCCL_SOCKET_RETRY_CNT andNCCL_SOCKET_RETRY_SLEEP_MSEC, the total retry time will be approx. 60 seconds.

NCCL_SOCKET_NTHREADS¶

(since 2.4.8)

TheNCCL_SOCKET_NTHREADS variable specifies the number of CPU helper threads used per network connection for socket transport. Increasing this value may increase the socket transport performance, at the cost of a higher CPU usage.

NCCL_NSOCKS_PERTHREAD¶

(since 2.4.8)

TheNCCL_NSOCKS_PERTHREAD variable specifies the number of sockets opened by each helper thread of the socket transport. In environments where per-socket speed is limited, setting this variable larger than 1 may improve the network performance.

NCCL_CROSS_NIC¶

TheNCCL_CROSS_NIC variable controls whether NCCL should allow rings/trees to use different NICs,causing inter-node communication to use different NICs on different nodes.

To maximize inter-node communication performance when using multiple NICs, NCCL tries to use thesame NICs when communicatingbetween nodes, to allow for a network design where each NIC on a node connects toa different network switch (network rail), and avoid any risk of traffic flow interference.TheNCCL_CROSS_NIC setting is therefore dependent on the network topology, and in particularon whether the network fabric is rail-optimized or not.

This has no effect on systems with only one NIC.

NCCL_IB_HCA¶

TheNCCL_IB_HCA variable specifies which Host Channel Adapter (RDMA) interfaces to use for communication.

NCCL_IB_TIMEOUT¶

TheNCCL_IB_TIMEOUT variable controls the InfiniBand Verbs Timeout.

The timeout is computed as 4.096 µs * 2 ^timeout, and the correct value is dependent on the size of the network.Increasing that value can help on very large networks, for example, if NCCL is failing on a call toibv_poll_cq witherror 12.

For more information, see section 12.7.34 of the InfiniBand specification Volume 1 (Local Ack Timeout).

NCCL_IB_RETRY_CNT¶

(since 2.1.15)

TheNCCL_IB_RETRY_CNT variable controls the InfiniBand retry count.

For more information, see section 12.7.38 of the InfiniBand specification Volume 1.

NCCL_IB_GID_INDEX¶

(since 2.1.4)

TheNCCL_IB_GID_INDEX variable defines the Global ID index used in RoCE mode.See the InfiniBandshow_gids command in order to set this value.

For more information, see the InfiniBand specification Volume 1or vendor documentation.

NCCL_IB_ADDR_FAMILY¶

(since 2.21)

TheNCCL_IB_ADDR_FAMILY variable defines the IP address family associated tothe infiniband GID dynamically selected by NCCL whenNCCL_IB_GID_INDEX is leftunset.

NCCL_IB_ADDR_RANGE¶

(since 2.21)

TheNCCL_IB_ADDR_RANGE variable defines the range of valid GIDs dynamicallyselected by NCCL whenNCCL_IB_GID_INDEX is left unset.

NCCL_IB_ROCE_VERSION_NUM¶

(since 2.21)

TheNCCL_IB_ROCE_VERSION_NUM variable defines the RoCE version associated tothe infiniband GID dynamically selected by NCCL whenNCCL_IB_GID_INDEX is leftunset.

NCCL_IB_SL¶

(since 2.1.4)

Defines the InfiniBand Service Level.

For more information, see the InfiniBand specification Volume 1or vendor documentation.

NCCL_IB_TC¶

(since 2.1.15)

Defines the InfiniBand traffic class field.

For more information, see the InfiniBand specification Volume 1or vendor documentation.

NCCL_IB_FIFO_TC¶

(since 2.22.3)

Defines the InfiniBand traffic class for control messages.Control messages are short RDMA write operations which controlcredit return, contrary to other RDMA operations transmittinglarge segments of data. This setting allows to have thosemessages use a high priority, low-latency traffic class andavoid being delayed by the rest of the traffic.

NCCL_IB_RETURN_ASYNC_EVENTS¶

(since 2.23)

IB events are reported to the user as warnings.If enabled, NCCL will also stop IB communications upon fatal IB asynchronous events.

NCCL_OOB_NET_ENABLE¶

(since 2.23)The variableNCCL_OOB_NET_ENABLE enables the use of NCCL net for out-of-band communications.Enabling the usage of NCCL net will change the implementation of the allgather performed during the communicator initialization.

NCCL_OOB_NET_IFNAME¶

(since 2.23)If NCCL net is enabled for out-of-band communication (seeNCCL_OOB_NET_ENABLE), theNCCL_OOB_NET_IFNAME variable specifies which network interfaces to use.

NCCL_UID_STAGGER_THRESHOLD¶

(since 2.23)TheNCCL_UID_STAGGER_THRESHOLD variable is used to trigger staggering of communications between NCCL ranks and the ncclUniqueId in order to avoid overflowing the ncclUniqueId.If the number of NCCL ranks communicating exceeds the specified threshold, the communications are staggered using the rank value (see NCCL_UID_STAGGER_RATE below).If the number of NCCL ranks per ncclUniqueId is smaller or equal to the threshold, no staggering is performed.

For example, if we have 128 NCCL ranks, 1 ncclUniqueId, and a threshold at 64, staggering is performed.However, if 2 ncclUniqueIds are used with 128 NCCL ranks and a threshold at 64, no staggering is done.

NCCL_UID_STAGGER_RATE¶

(since 2.23)

TheNCCL_UID_STAGGER_RATE variable is used to define the message rate targeted when staggering the communications between NCCL ranks and the ncclUniqueId.If staggering is used (see NCCL_UID_STAGGER_THRESHOLD above), the message rate is used to compute the time a given NCCL rank has to wait.

NCCL_NET¶

(since 2.10)

Forces NCCL to use a specific network, for example to make sure NCCL uses an external plugin and doesn’t automatically fall back on the internal IB or Socket implementation. Setting this environment variable will override thenetName configuration in all communicators (seencclConfig_t); if not set (undefined), the network module will be determined by the configuration; if not passing configuration, NCCL will automatically choose the best network module.

NCCL_NET_PLUGIN¶

(since 2.11)

Set it to either a suffix string or to a library name to choose among multiple NCCL net plugins. This setting will cause NCCL to look for the net plugin library using the following strategy:

• If NCCL_NET_PLUGIN is set, attempt loading the library with name specified by NCCL_NET_PLUGIN;
• If NCCL_NET_PLUGIN is set and previous failed, attempt loading libnccl-net-<NCCL_NET_PLUGIN>.so;
• If NCCL_NET_PLUGIN is not set, attempt loading libnccl-net.so;
• If no plugin was found (neither user defined nor default), use internal network plugin.

For example, settingNCCL_NET_PLUGIN=foo will cause NCCL to try loadfoo and, iffoo cannot be found,libnccl-net-foo.so (provided that it exists on the system).

NCCL_TUNER_PLUGIN¶

Set it to either a suffix string or to a library name to choose among multiple NCCL tuner plugins. This setting will cause NCCL to look for the tuner plugin library using the following strategy:

• If NCCL_TUNER_PLUGIN is set, attempt loading the library with name specified by NCCL_TUNER_PLUGIN;
• If NCCL_TUNER_PLUGIN is set and previous failed, attempt loading libnccl-net-<NCCL_TUNER_PLUGIN>.so;
• If NCCL_TUNER_PLUGIN is not set, attempt loading libnccl-tuner.so;
• If no plugin was found look for the tuner symbols in the net plugin (refer toNCCL_NET_PLUGIN);
• If no plugin was found (neither through NCCL_TUNER_PLUGIN nor NCCL_NET_PLUGIN), use internal tuner plugin.

For example, settingNCCL_TUNER_PLUGIN=foo will cause NCCL to try loadfoo and, iffoo cannot be found,libnccl-tuner-foo.so (provided that it exists on the system).

NCCL_PROFILER_PLUGIN¶

Set it to either a suffix string or to a library name to choose among multiple NCCL profiler plugins. This setting will cause NCCL to look for the profiler plugin library using the following strategy:

• If NCCL_PROFILER_PLUGIN is set, attempt loading the library with name specified by NCCL_PROFILER_PLUGIN;
• If NCCL_PROFILER_PLUGIN is set and previous failed, attempt loading libnccl-profiler-<NCCL_PROFILER_PLUGIN>.so;
• If NCCL_PROFILER_PLUGIN is not set, attempt loading libnccl-profiler.so;
• If no plugin was found (neither user defined nor default), do not enable profiling.
• If NCCL_PROFILER_PLUGIN is set toSTATIC_PLUGIN, the plugin symbols are searched in the program binary.

For example, settingNCCL_PROFILER_PLUGIN=foo will cause NCCL to try loadfoo and, iffoo cannot be found,libnccl-profiler-foo.so (provided that it exists on the system).

NCCL_ENV_PLUGIN¶

(since 2.28)

TheNCCL_ENV_PLUGIN variable can be used to let NCCL load an external environment plugin. Set it to either a library name or a suffix string to choose among multiple NCCL environment plugins. This setting will cause NCCL to look for the environment plugin library using the following strategy:

• IfNCCL_ENV_PLUGIN is set to a library name, attempt loading that library (e.g.NCCL_ENV_PLUGIN=/path/to/library/libfoo.so will cause NCCL to try load/path/to/library/libfoo.so);
• IfNCCL_ENV_PLUGIN is set to a suffix string, attempt loadinglibnccl-env-<NCCL_ENV_PLUGIN>.so (e.g.NCCL_ENV_PLUGIN=foo will cause NCCL to try loadlibnccl-env-foo.so from the system library path);
• IfNCCL_ENV_PLUGIN is not set, attempt loading the defaultlibnccl-env.so library from the system library path;
• IfNCCL_ENV_PLUGIN is set to “none”, explicitly disable the external plugin and use the internal one;
• If no plugin was found (neither user defined nor default) or the variable is set to “none”, use the internal environmentplugin.

NCCL_IGNORE_CPU_AFFINITY¶

(since 2.4.6)

TheNCCL_IGNORE_CPU_AFFINITY variable can be used to cause NCCL to ignore the job’s supplied CPU affinity and instead use the GPU affinity only.

NCCL_CONF_FILE¶

(since 2.23)

TheNCCL_CONF_FILE variable allows the user to specify a file with the static configuration.This does not accept the~ character as part of the path; please convert to a relative or absolute path first.

NCCL_DEBUG¶

TheNCCL_DEBUG variable controls the debug information that is displayed from NCCL. This variable is commonly used for debugging.

NCCL_DEBUG_FILE¶

(since 2.2.12)

TheNCCL_DEBUG_FILE variable directs the NCCL debug logging output to a file.The filename format can be set tofilename.%h.%p where%h is replaced with thehostname and%p is replaced with the process PID. This does not accept the~ character as part of the path, please convert to a relative or absolute path first.

NCCL_DEBUG_SUBSYS¶

(since 2.3.4)

TheNCCL_DEBUG_SUBSYS variable allows the user to filter theNCCL_DEBUG=INFO output based on subsystems.The value should be a comma separated list of the subsystems to include in the NCCL debug log traces.

Prefixing the subsystem name with ‘^’ will disable the logging for that subsystem.

NCCL_DEBUG_TIMESTAMP_FORMAT¶

(since 2.26)

TheNCCL_DEBUG_TIMESTAMP_FORMAT variable allows the user to changethe format used when printing debug log messages.

The time is printed as a localtime. This can be changed by setting theTZ environment variable. UTCis available by settingTZ=UTC. Valid values for TZ look like:US/Pacific,America/Los_Angeles, etc.

Note that the non-callTRACE level of logs continues to print themicroseconds since the NCCL debug subsystem was initialized. TheTRACE logs can also print the strftime formatted timestamp at thebeginning if so configured (seeNCCL_DEBUG_TIMESTAMP_LEVELS).

(since 2.26)Underscores in the format are rendered as spaces.

NCCL_DEBUG_TIMESTAMP_LEVELS¶

(since 2.26)

TheNCCL_DEBUG_TIMESTAMP_LEVELS variable allows the user to setwhich log lines get a timestamp depending upon the level of the log.

NCCL_COLLNET_ENABLE¶

(since 2.6)

Enable the use of the CollNet plugin.

NCCL_COLLNET_NODE_THRESHOLD¶

(since 2.9.9)

A threshold for the number of nodes below which CollNet will not be enabled.

NCCL_CTA_POLICY¶

(since 2.27)

TheNCCL_CTA_POLICY variable allows the user to set the policy for the NCCL communicator.

NCCL_NETDEVS_POLICY¶

(since 2.28)

TheNCCL_NETDEVS_POLICY variable allows the user to set the policy for the assignment of network devices to the GPUs.For each GPU, NCCL detects automatically available network devices, taking into account their network bandwidth and the node topology.

NCCL_TOPO_FILE¶

(since 2.6)

Path to an XML file to load before detecting the topology. By default, NCCL will load/var/run/nvidia-topologyd/virtualTopology.xml if present.

NCCL_TOPO_DUMP_FILE¶

(since 2.6)

Path to a file to dump the XML topology to after detection.

NCCL_SET_THREAD_NAME¶

(since 2.12)

Give more meaningful names to NCCL CPU threads to ease debugging and analysis.

NCCL_P2P_DISABLE¶

TheNCCL_P2P_DISABLE variable disables the peer to peer (P2P) transport, which uses CUDA direct access between GPUs, using NVLink or PCI.

NCCL_P2P_LEVEL¶

(since 2.3.4)

TheNCCL_P2P_LEVEL variable allows the user to finely control when to use the peer to peer (P2P) transport between GPUs.The level defines the maximum distance between GPUs where NCCL will use the P2P transport. A short string representingthe path type should be used to specify the topographical cutoff for using the P2P transport.

If this isn’t specified, NCCL will attempt to optimally select a value based on the architecture and environment it’s run in.

NCCL_P2P_DIRECT_DISABLE¶

TheNCCL_P2P_DIRECT_DISABLE variable forbids NCCL to directly access user buffers through P2P between GPUs of the same process. This is useful when user buffers are allocated with APIs which do not automatically make them accessible to other GPUs managed by the same process and with P2P access.

NCCL_SHM_DISABLE¶

TheNCCL_SHM_DISABLE variable disables the Shared Memory (SHM) transports. SHM is used between devices when peer-to-peer cannot happen, therefore, host memory is used. NCCL will use the network (i.e. InfiniBand or IP sockets) to communicate between the CPU sockets when SHM is disabled.

NCCL_BUFFSIZE¶

TheNCCL_BUFFSIZE variable controls the size of the buffer used by NCCL when communicating data between pairs of GPUs.

Use this variable if you encounter memory constraint issues when using NCCL or you think that a different buffer size would improve performance.

NCCL_NTHREADS¶

TheNCCL_NTHREADS variable sets the number of CUDA threads per CUDA block. NCCL will launch one CUDA block per communication channel.

Use this variable if you think your GPU clocks are low and you want to increase the number of threads.

You can also use this variable to reduce the number of threads to decrease the GPU workload.

NCCL_MAX_NCHANNELS¶

(NCCL_MAX_NRINGS since 2.0.5, NCCL_MAX_NCHANNELS since 2.5.0)

TheNCCL_MAX_NCHANNELS variable limits the number of channels NCCL can use. Reducing the number of channels also reduces thenumber of CUDA blocks used for communication, hence the impact on GPU computing resources.

The oldNCCL_MAX_NRINGS variable (used until 2.4) still works as an alias in newer versions but is ignored ifNCCL_MAX_NCHANNELS is set.

This environment variable has been superseded byNCCL_MAX_CTAS which can also be set programmatically usingncclCommInitRankConfig.

NCCL_MIN_NCHANNELS¶

(NCCL_MIN_NRINGS since 2.2.0, NCCL_MIN_NCHANNELS since 2.5.0)

TheNCCL_MIN_NCHANNELS variable controls the minimum number of channels you want NCCL to use.Increasing the number of channels also increases the number ofCUDA blocks NCCL uses, which may be useful to improve performance; however, it uses more CUDA compute resources.

This is especially useful when using aggregated collectives on platforms where NCCL would usually only create one channel.

The oldNCCL_MIN_NRINGS variable (used until 2.4) still works as an alias in newer versions, but is ignored ifNCCL_MIN_NCHANNELS is set.

This environment variable has been superseded byNCCL_MIN_CTAS which can also be set programmatically usingncclCommInitRankConfig.

NCCL_CHECKS_DISABLE¶

(since 2.0.5, deprecated in 2.2.12)

TheNCCL_CHECKS_DISABLE variable can be used to disable argument checks on each collective call.Checks are useful during development but can increase the latency. They can be disabled toimprove performance in production.

NCCL_CHECK_POINTERS¶

(since 2.2.12)

TheNCCL_CHECK_POINTERS variable enables checking of the CUDA memory pointers on each collective call.Checks are useful during development but can increase the latency.

NCCL_LAUNCH_MODE¶

(since 2.1.0)

TheNCCL_LAUNCH_MODE variable controls how NCCL launches CUDA kernels.

NCCL_IB_DISABLE¶

TheNCCL_IB_DISABLE variable prevents the IB/RoCE transport from being used by NCCL. Instead, NCCL will fall back tousing IP sockets.

NCCL_IB_AR_THRESHOLD¶

(since 2.6)

Threshold above which we send InfiniBand data in a separate message which canleverage adaptive routing.

NCCL_IB_QPS_PER_CONNECTION¶

(since 2.10)

Number of IB queue pairs to use for each connection between two ranks. This can be useful on multi-level fabrics which need multiple queue pairs to have good routing entropy.SeeNCCL_IB_SPLIT_DATA_ON_QPS for different ways to split data on multiple QPs, as it can affect performance.

NCCL_IB_SPLIT_DATA_ON_QPS¶

(since 2.18)

This parameter controls how we use the queue pairs when we create more than one.Set to 1 (split mode), each message will be split evenly on each queue pair. This may cause a visible latency degradation if many QPs are used.Set to 0 (round-robin mode), queue pairs will be used in round-robin mode for each message we send. Operations which do not send multiple messages will not use all QPs.

NCCL_IB_CUDA_SUPPORT¶

(removed in 2.4.0, see NCCL_NET_GDR_LEVEL)

TheNCCL_IB_CUDA_SUPPORT variable is used to force or disable the usage of GPU Direct RDMA.By default, NCCL enables GPU Direct RDMA if the topology permits it. This variable can disable this behavior or forcethe usage of GPU Direct RDMA in all cases.

NCCL_IB_PCI_RELAXED_ORDERING¶

(since 2.12)

Enable the use of Relaxed Ordering for the IB Verbs transport. Relaxed Ordering can greatly help the performance of InfiniBand networks in virtualized environments.

NCCL_IB_ADAPTIVE_ROUTING¶

(since 2.16)

Enable the use of Adaptive Routing capable data transfers for the IB Verbs transport. Adaptive routing can improve the performance of communications at scale. A system defined Adaptive Routing enabled SL has to be selected accordingly (cf.NCCL_IB_SL).

NCCL_IB_ECE_ENABLE¶

(since 2.23)

Enable the use of Enhanced Connection Establishment (ECE) on IB/RoCE Verbs networks. ECE can be used to enable advanced networking features such as Congestion Control, Adaptive Routing and Selective Repeat. Note: These parameters are not interpreted or controlled by NCCL and are passed through directly to the HCAs via the ECE mechanism.

NCCL_MEM_SYNC_DOMAIN¶

(since 2.16)

Sets the default Memory Sync Domain for NCCL kernels (CUDA 12.0 & sm90 and later). Memory Sync Domains can help eliminate interference between the NCCL kernels and the application compute kernels, when they use different domains.

NCCL_CUMEM_ENABLE¶

(since 2.18)

Use CUDA cuMem* functions to allocate memory in NCCL.

NCCL_CUMEM_HOST_ENABLE¶

(since 2.23)

Use CUDA cuMem* functions to allocate host memory in NCCL. SeeShared memory for more information.

NCCL_NET_GDR_LEVEL (formerly NCCL_IB_GDR_LEVEL)¶

(since 2.3.4. In 2.4.0, NCCL_IB_GDR_LEVEL was renamed to NCCL_NET_GDR_LEVEL)

TheNCCL_NET_GDR_LEVEL variable allows the user to finely control when to use GPU Direct RDMA between a NIC and a GPU.The level defines the maximum distance between the NIC and the GPU. A string representing the path type should be used to specify the topographical cutoff for GpuDirect.

If this isn’t specified, NCCL will attempt to optimally select a value based on the architecture and environment it’s run in.

NCCL_NET_GDR_C2C¶

(since 2.26)

TheNCCL_NET_GDR_C2C variable enables GPU Direct RDMA when sending data via a NIC attached to a CPU (i.e. distance PHB) where the CPU is connected to the GPU via a C2C interconnect. This effectively overrides theNCCL_NET_GDR_LEVEL setting for this particular NIC.

NCCL_NET_GDR_READ¶

TheNCCL_NET_GDR_READ variable enables GPU Direct RDMA when sending data as long as the GPU-NIC distance is within the distance specified byNCCL_NET_GDR_LEVEL. Before 2.4.2, GDR read is disabled by default, i.e. when sending data, the data is first stored in CPU memory, then goes to the InfiniBand card. Since 2.4.2, GDR read is enabled by default for NVLink-based platforms.

Note: Reading directly from GPU memory when sending data is known to be slightly slower than reading from CPU memory on some platforms, such as PCI-E.

NCCL_NET_SHARED_BUFFERS¶

(since 2.8)

Allows the usage of shared buffers for inter-node point-to-point communication.This will use a single large pool for all remote peers, having a constantmemory usage instead of increasing linearly with the number of remote peers.

NCCL_NET_SHARED_COMMS¶

(since 2.12)

Reuse the same connections in the context of PXN. This allows for messageaggregation but can also decrease the entropy of network packets.

NCCL_SINGLE_RING_THRESHOLD¶

(since 2.1, removed in 2.3)

TheNCCL_SINGLE_RING_THRESHOLD variable sets the limit under which NCCL will only use one ring.This will limit bandwidth but improve latency.

NCCL_LL_THRESHOLD¶

(since 2.1, removed in 2.5)

TheNCCL_LL_THRESHOLD variable sets the size limit under which NCCL uses low-latency algorithms.

NCCL_TREE_THRESHOLD¶

(since 2.4, removed in 2.5)

TheNCCL_TREE_THRESHOLD variable sets the size limit under which NCCL uses tree algorithms instead of rings.

NCCL_ALGO¶

(since 2.5)

TheNCCL_ALGO variable defines which algorithms NCCL will use.

NCCL_PROTO¶

(since 2.5)

TheNCCL_PROTO variable defines which protocol(s) NCCL will be allowedto use.

Users are discouraged from setting this variable, with the exception ofdisabling a specific protocol in case a bug in NCCL is suspected. Inparticular, enabling LL128 on platforms that don’t support it can leadto data corruption.

NCCL_NVB_DISABLE¶

(since 2.11)

Disable intra-node communication through NVLink via an intermediate GPU.

NCCL_PXN_DISABLE¶

(since 2.12)

Disable inter-node communication using a non-local NIC, using NVLink andan intermediate GPU.

NCCL_P2P_PXN_LEVEL¶

(since 2.12)

Control in which cases PXN is used for send/receive operations.

NCCL_PXN_C2C¶

(since 2.27)

Allow NCCL to use the PXN mechanism if the peer GPU is connected through C2C + PCIe to the targeted NIC.

NCCL_RUNTIME_CONNECT¶

(since 2.22)

Dynamically connect peers during runtime (e.g., callingncclAllreduce()) instead of init stage.

NCCL_GRAPH_REGISTER¶

(since 2.11)

Enable user buffer registration when NCCL calls are captured by CUDA Graphs.

Effective only when:(i) the CollNet algorithm is being used;(ii) all GPUs within a node have P2P access to each other;(iii) there is at most one GPU per process.

User buffer registration may reduce the number of data copies between user buffers and the internal buffers of NCCL.The user buffers will be automatically de-registered when the CUDA Graphs are destroyed.

NCCL_LOCAL_REGISTER¶

(since 2.19)

Enable user local buffer registration when users explicitly callncclCommRegister.

NCCL_LEGACY_CUDA_REGISTER¶

(since 2.24)

Cuda buffers allocated throughcudaMalloc (and related memory allocators) are legacybuffers. Registering legacy buffer can cause implicit synchronization, which is unsafeand can possibly cause a hang for NCCL. NCCL disables legacy buffer registration bydefault, and users should move to cuMem-based memory allocators for buffer registration.

NCCL_WIN_ENABLE¶

(since 2.27)

Enable window memory registration.

NCCL_SET_STACK_SIZE¶

(since 2.9)

Set CUDA kernel stack size to the maximum stack size amongst all NCCL kernels.

It may avoid a CUDA memory reconfiguration on load. Set to 1 if you experience hang due to CUDA memory reconfiguration.

NCCL_GRAPH_MIXING_SUPPORT¶

(since 2.13)

Enable/disable support for multiple outstanding NCCL calls from parallel CUDA graphs or a CUDA graph and non-captured NCCL calls. NCCL calls are considered outstanding starting from their host-side launch (e.g., a call toncclAllreduce() for non-captured calls orcudaGraphLaunch() for captured calls) and ending when the device kernel execution completes. With graph mixing support disabled, the following use cases are NOT supported:

1. Using a NCCL communicator (or split-shared communicators) from parallel graph launches, where parallel means on different streams without dependencies that would serialize their execution.
2. Launching a non-captured NCCL collective during an outstanding graph launch that uses the same communicator (or split-shared communicators), regardless of stream ordering.

The ability to disable support is motivated by observed hangs in the CUDA launches when support is enabled and multiple ranks have work launched via cudaGraphLaunch from the same thread.

NCCL_DMABUF_ENABLE¶

(since 2.13)

Enable GPU Direct RDMA buffer registration using the Linux dma-buf subsystem.

The Linux dma-buf subsystem allows GPU Direct RDMA capable NICs to read and write CUDA buffers directly without CPU involvement.

NCCL_P2P_NET_CHUNKSIZE¶

(since 2.14)

TheNCCL_P2P_NET_CHUNKSIZE controls the size of messages sent through the network for ncclSend/ncclRecv operations.

NCCL_P2P_LL_THRESHOLD¶

(since 2.14)

TheNCCL_P2P_LL_THRESHOLD is the maximum message size that NCCL will use the LL protocol for P2P operations.

NCCL_ALLOC_P2P_NET_LL_BUFFERS¶

(since 2.14)

NCCL_ALLOC_P2P_NET_LL_BUFFERS instructs communicators to allocate dedicated LL buffers for all P2P network connections. This enables all ranks to use the LL protocol for latency-bound send and receive operations belowNCCL_P2P_LL_THRESHOLD sizes.Intranode P2P transfers always have dedicated LL buffers allocated. If running all-to-all workloads with high numbers of ranks, this will result in a high scaling memory overhead.

NCCL_COMM_BLOCKING¶

(since 2.14)

TheNCCL_COMM_BLOCKING variable controls whether NCCL calls are allowed to block or not. This includes all calls to NCCL, including init/finalize functions, as well as communication functions which may also block due to the lazy initialization of connections for send/receive calls. Setting this environment variable will override theblocking configuration in all communicators (seencclConfig_t); if not set (undefined), communicator behavior will be determined by the configuration; if not passing configuration, communicators are blocking.

NCCL_CGA_CLUSTER_SIZE¶

(since 2.16)

Set CUDA Cooperative Group Array (CGA) cluster size. On sm90 and later we have an extra level of hierarchy where wecan group together several blocks within the Grid, called Thread Block Clusters. Setting this to non-zero will causeNCCL to launch the communication kernels with the Cluster Dimension attribute set accordingly. Setting this environmentvariable will override thecgaClusterSize configuration in all communicators (seencclConfig_t); if not set(undefined), CGA cluster size will be determined by the configuration; if not passing configuration, NCCL willautomatically choose the best value.

NCCL_MAX_CTAS¶

(since 2.17)

Set the maximal number of CTAs the NCCL should use. Setting this environment variable will override themaxCTAs configuration in all communicators (seencclConfig_t); if not set (undefined), maximal CTAs will be determined by the configuration; if not passing configuration, NCCL will automatically choose the best value.

NCCL_MIN_CTAS¶

(since 2.17)

Set the minimal number of CTAs the NCCL should use. Setting this environment variable will override theminCTAs configuration in all communicators (seencclConfig_t); if not set (undefined), minimal CTAs will be determined by the configuration; if not passing configuration, NCCL will automatically choose the best value.

NCCL_NVLS_ENABLE¶

(since 2.17)

Enable the use of NVLink SHARP (NVLS). NVLink SHARP is available in third-generation NVSwitch systems (NVLink4) with Hopper and later GPU architectures, allowing collectives such asncclAllReduce to be offloaded to the NVSwitch domain.The default value is 2, so NVLS will be disabled automatically on systems which do not support the feature.

NCCL_IB_MERGE_NICS¶

(since 2.20)

Enable NCCL to combine dual-port IB NICs into a single logical network device. This allows NCCL to more easily aggregate dual-port NIC bandwidth.

NCCL_MNNVL_ENABLE¶

(since 2.21)

Enable NCCL to use Multi-Node NVLink (MNNVL) when available. If the system or driver are not Multi-Node NVLink capable then MNNVL will automatically be disabled. This feature also requires NCCL CUMEM support (NCCL_CUMEM_ENABLE) to be enabled.MNNVL requires a fully configured and operational IMEX domain for all the nodes that form the NVLink domain. See the CUDA documentation for more details on IMEX domains.

NCCL_MNNVL_UUID¶

(since 2.25)Can be used to set the Multi-Node NVLink (MNNVL) UUID to a user defined value. The supplied value will be assigned to both the upper and lower 64-bit words of the 128-bit UUID. Normally the MNNVL UUID is assigned by the Fabric Manager, and it should not need to be overridden.

NCCL_MNNVL_CLIQUE_ID¶

(since 2.25)Can be used to set the Multi-Node NVLink (MNNVL) Clique Id to a user defined value. Normally the Clique Id is assigned by the Fabric Manager, but this environment variable can be used to “soft” partition MNNVL jobs. i.e. NCCL will only treat ranks with the same <UUID,CLIQUE_ID> as being part of the same NVLink domain.

NCCL_RAS_ENABLE¶

(since 2.24)

Enable NCCL’s reliability, availability, and serviceability (RAS) subsystem, which can be used to query the health ofNCCL jobs during execution (seeRAS).

NCCL_RAS_ADDR¶

(since 2.24)

Specify the IP address and port number of a socket that the RAS subsystem will listen on for client connections. RAScan share this socket between multiple processes but that would not be desirable if multiple independent NCCL jobs sharea single node (and if those jobs belong to different users, the OS will not allow the socket to be shared). In suchcases, each job should be started with a different value (e.g.,localhost:12345,localhost:12346, etc.). Sincelocalhost is normally used, only those with access to the nodes where the job is running can connect to the socket.If desired, the address of an externally accessible network interface can be specified instead, which will make RASaccessible from other nodes (such as a cluster’s head node), but that has security implications that should beconsidered.

NCCL_RAS_TIMEOUT_FACTOR¶

(since 2.24)

Specify the multiplier factor to apply to all the timeouts of the RAS subsystem. RAS relies on multiple timeouts,ranging from 5 to 60 seconds, to determine the state of the application and to maintain its internal communication, withcomplex interdependecies between different timeouts. This variable can be used to scale up all these timeouts in asafe, consistent manner, should any of the defaults turn out to be too small; e.g., if the NCCL application is subjectto high-overhead debugging/tracing/etc., which makes its execution less predictable. If one wants to use thencclras client in such circumstances, its timeout may need to be increased as well (or disabled).

NCCL_LAUNCH_ORDER_IMPLICIT¶

(since 2.26)

Implicitly order NCCL operations from different communicators on the same device using the host program order. This ensures the operations will not deadlock. When the CUDA runtime and driver are 12.3+, overlapped execution is permitted. On older CUDA versions the operations will be serialized.

NCCL_LAUNCH_RACE_FATAL¶

(since 2.26)

Attempt to catch host threads racing to launch to the same device and if so return a fatal error. Such a race would violate the determinacy of the program order relied upon by NCCL_LAUNCH_ORDER_IMPLICIT.