1.5.1.BF16x9

The BF16x9 algorithm is used for emulating FP32 arithmetic. An FP32 value can be exactly represented as three BF16 values as follows:

We can fully reconstruct the FP32 value from the BF16 values without any loss of accuracy. Using this, we define an FMA operation (d = ab + c) as follows:

In practice, the BF16 tensor cores are utilized rather than FMA units and this idea naturally extends into complex arithmetic as well.

While BF16x9 can be supported on all hardware, it only provides a performance advantage when peak BF16 throughput is more than nine times greater than peak FP32 throughput. It also requires special hardware features to apply the additional scaling factors in a performant manner. As a result, BF16x9 is only supported on select architectures. See theFloating Point Emulation Support Overview table for more details.

1.5.2.Fixed-Point

Fixed-point emulation is used for emulating FP64 arithmetic and follows theOzaki Scheme. Fixed-point representations emulate floating point through the addition of a shared power of two scaling factor and by encoding the remaining dynamic range of floating point within mantissa bits. The scaling factor is shared for elements in the same row of the A matrix or column of the B matrix and is used to logically scale all elements to be between -1 and 1 inclusively.

Due to the large dynamic range of FP64, there is no single configuration of fixed-point which is both performant and accurate for all floating point inputs. Therefore, we enable two flavors of fixed-point emulation:Dynamic Mantissa Control andFixed Mantissa Control. These configurations can be set withcublasSetFixedPointEmulationMantissaControl().

size_tgetFixedPointWorkspaceSizeInBytes(intm,intn,intk,intbatchCount,boolisComplex,cudaEmulationMantissaControlmantissaControl,intmaxMantissaBitCount){constexprdoubleMULTIPLIER=1.25;intmult=isComplex?2:1;intnumSlices=ceildiv(maxMantissaBitCount+1,8);intpadded_m=ceildiv(m,1024)*1024;intpadded_n=ceildiv(n,1024)*1024;intpadded_k=ceildiv(k,128)*128;intnum_blocks_k=ceildiv(k,64);size_tgemm_workspace=sizeof(int8_t)*((size_t)padded_m*padded_k+(size_t)padded_n*padded_k)*mult*numSlices;gemm_workspace+=sizeof(int32_t)*((size_t)padded_m+padded_n)*mult;if(isComplex){gemm_workspace+=sizeof(double)*(size_t)m*n*mult*mult;}size_tadp_workspace=0;if(mantissaControl==CUDA_EMULATION_MANTISSA_CONTROL_DYNAMIC){adp_workspace=sizeof(int32_t)*((size_t)m*num_blocks_k+(size_t)n*num_blocks_k+(size_t)m*n)*mult;}constexprsize_tCONSTANT_SIZE=128*1024*1024;return(size_t)(std::max(gemm_workspace,adp_workspace)*batchCount*MULTIPLIER)+CONSTANT_SIZE;}

1.5.3.Default Library Configurations

Library default values for emulation are subject to change.

1.5.4.Support For Floating Point Special Values

The implementations of floating point emulation algorithms maintain the accuracy of the emulated precision for both normal and denormalized values but may not adhere to the IEEE-754 standard with respect to\(\text{Inf}\),\(\text{NaN}\), or signed zeros. If the underlying emulated algorithm cannot implicitly support a given special value, and the library is configured to support it (seecublasSetEmulationSpecialValuesSupport()), then extra steps are taken to support it. The following table shows which special values are implicitly supported for each emulation algorithm.

2.1.1.Error Status

All cuBLAS library function calls return the error statuscublasStatus_t.

2.1.2.cuBLAS Context

The application must initialize a handle to the cuBLAS library context by calling thecublasCreate() function. Then, the handle is explicitly passed to every subsequent library function call. Once the application finishes using the library, it must call the functioncublasDestroy() to release the resources associated with the cuBLAS library context.

This approach allows the user to explicitly control the library setup when using multiple host threads and multiple GPUs. For example, the application can usecudaSetDevice() to associate different devices with different host threads and in each of those host threads it can initialize a unique handle to the cuBLAS library context, which will use the particular device associated with that host thread. Then, the cuBLAS library function calls made with different handles will automatically dispatch the computation to different devices.

The device associated with a particular cuBLAS context is assumed to remain unchanged between the correspondingcublasCreate() andcublasDestroy() calls. In order for the cuBLAS library to use a different device in the same host thread, the application must set the new device to be used by callingcudaSetDevice() and then create another cuBLAS context, which will be associated with the new device, by callingcublasCreate(). When multiple devices are available, applications must ensure that the device associated with a given cuBLAS context is current (e.g. by callingcudaSetDevice()) before invoking cuBLAS functions with this context.

A cuBLAS library context is tightly coupled with the CUDA context that is current at the time of thecublasCreate() call. An application that uses multiple CUDA contexts is required to create a cuBLAS context per CUDA context and make sure the former never outlives the latter. Starting from version 12.8, cuBLAS detects if the underlying CUDA context is tied to a graphics context and follows the shared memory size limits that are set in such case.

2.1.3.Thread Safety

The library is thread safe and its functions can be called from multiple host threads, even with the same handle. When multiple threads share the same handle, extreme care needs to be taken when the handle configuration is changed because that change will affect potentially subsequent cuBLAS calls in all threads. It is even more true for the destruction of the handle. So it is not recommended that multiple thread share the same cuBLAS handle.

Additional considerations apply when the same handle is used from multiple threads with a user provided workspace. SeecublasSetWorkspace() for details.

2.1.4.Results Reproducibility

By design, all cuBLAS API routines from a given toolkit version, generate the same bit-wise results at every run when executed on GPUs with the same architecture and the same number of SMs. However, bit-wise reproducibility is not guaranteed across toolkit versions because the implementation might differ due to some implementation changes.

This guarantee no longer holds when multiple CUDA streams are active orfixed-point emulation is used. If multiple concurrent streams are active, the library may optimize total performance by picking different internal implementations.

Any of those settings will allow for deterministic behavior even with multiple concurrent streams sharing a single cuBLAS handle.

This behavior is expected to change in a future release.

For some routines such ascublas<t>symv() andcublas<t>hemv(), an alternate significantly faster routine can be chosen using the routinecublasSetAtomicsMode(). In that case, the results are not guaranteed to be bit-wise reproducible because atomics are used for the computation.

2.1.5.Scalar Parameters

There are two categories of the functions that use scalar parameters :

• Functions that takealpha and/orbeta parameters by reference on the host or the device as scaling factors, such asgemm.

• Functions that return a scalar result on the host or the device such asamax(),amin,asum(),rotg(),rotmg(),dot() andnrm2().

For the functions of the first category, when the pointer mode is set toCUBLAS_POINTER_MODE_HOST, the scalar parametersalpha and/orbeta can be on the stack or allocated on the heap, shouldn’t be placed in managed memory. Underneath, the CUDA kernels related to those functions will be launched with the value ofalpha and/orbeta. Therefore if they were allocated on the heap, they can be freed just after the return of the call even though the kernel launch is asynchronous. When the pointer mode is set toCUBLAS_POINTER_MODE_DEVICE,alpha and/orbeta must be accessible on the device and their values should not be modified until the kernel is done. Note that sincecudaFree() does an implicitcudaDeviceSynchronize(),cudaFree() can still be called onalpha and/orbeta just after the call but it would defeat the purpose of using this pointer mode in that case.

For the functions of the second category, when the pointer mode is set toCUBLAS_POINTER_MODE_HOST, these functions block the CPU, until the GPU has completed its computation and the results have been copied back to the Host. When the pointer mode is set toCUBLAS_POINTER_MODE_DEVICE, these functions return immediately. In this case, similar to matrix and vector results, the scalar result is ready only when execution of the routine on the GPU has completed. This requires proper synchronization in order to read the result from the host.

In either case, the pointer modeCUBLAS_POINTER_MODE_DEVICE allows the library functions to execute completely asynchronously from the Host even whenalpha and/orbeta are generated by a previous kernel. For example, this situation can arise when iterative methods for solution of linear systems and eigenvalue problems are implemented using the cuBLAS library.

2.1.6.Parallelism with Streams

If the application uses the results computed by multiple independent tasks, CUDA™ streams can be used to overlap the computation performed in these tasks.

The application can conceptually associate each stream with each task. In order to achieve the overlap of computation between the tasks, the user should create CUDA™ streams using the functioncudaStreamCreate() and set the stream to be used by each individual cuBLAS library routine by callingcublasSetStream() just before calling the actual cuBLAS routine. Note thatcublasSetStream() resets the user-provided workspace to the default workspace pool; seecublasSetWorkspace(). Then, the computation performed in separate streams would be overlapped automatically when possible on the GPU. This approach is especially useful when the computation performed by a single task is relatively small and is not enough to fill the GPU with work.

We recommend using the new cuBLAS API with scalar parameters and results passed by reference in the device memory to achieve maximum overlap of the computation when using streams.

A particular application of streams, batching of multiple small kernels, is described in the following section.

2.1.7.Batching Kernels

In this section, we explain how to use streams to batch the execution of small kernels. For instance, suppose that we have an application where we need to make many small independent matrix-matrix multiplications with dense matrices.

It is clear that even with millions of small independent matrices we will not be able to achieve the sameGFLOPS rate as with a one large matrix. For example, a single\(n \times n\) large matrix-matrix multiplication performs\(n^{3}\) operations for\(n^{2}\) input size, while 1024\(\frac{n}{32} \times \frac{n}{32}\) small matrix-matrix multiplications perform\(1024\left( \frac{n}{32} \right)^{3} = \frac{n^{3}}{32}\) operations for the same input size. However, it is also clear that we can achieve a significantly better performance with many small independent matrices compared with a single small matrix.

The architecture family of GPUs allows us to execute multiple kernels simultaneously. Hence, in order to batch the execution of independent kernels, we can run each of them in a separate stream. In particular, in the above example we could create 1024 CUDA™ streams using the functioncudaStreamCreate(), then preface each call tocublas<t>gemm() with a call tocublasSetStream() with a different stream for each of the matrix-matrix multiplications (note thatcublasSetStream() resets user-provided workspace to the default workspace pool, seecublasSetWorkspace()). This will ensure that when possible the different computations will be executed concurrently. Although the user can create many streams, in practice it is not possible to have more than 32 concurrent kernels executing at the same time.

2.1.8.Cache Configuration

On some devices, L1 cache and shared memory use the same hardware resources. The cache configuration can be set directly with the CUDA Runtime function cudaDeviceSetCacheConfig. The cache configuration can also be set specifically for some functions using the routine cudaFuncSetCacheConfig. Please refer to the CUDA Runtime API documentation for details about the cache configuration settings.

Because switching from one configuration to another can affect kernels concurrency, the cuBLAS Library does not set any cache configuration preference and relies on the current setting. However, some cuBLAS routines, especially Level-3 routines, rely heavily on shared memory. Thus the cache preference setting might affect adversely their performance.

2.1.9.Static Library Support

The cuBLAS Library is also delivered in a static form aslibcublas_static.a on Linux. The static cuBLAS library and all other static math libraries depend on a common thread abstraction layer library calledlibculibos.a.

For example, on Linux, to compile a small application using cuBLAS, against the dynamic library, the following command can be used:

nvccmyCublasApp.c-lcublas-omyCublasApp

Whereas to compile against the static cuBLAS library, the following command must be used:

nvccmyCublasApp.c-lcublas_static-lculibos-omyCublasApp

It is also possible to use the native Host C++ compiler. Depending on the Host operating system, some additional libraries likepthread ordl might be needed on the linking line. The following command on Linux is suggested :

g++myCublasApp.c-lcublas_static-lculibos-lcudart_static-lpthread-ldl-I<cuda-toolkit-path>/include-L<cuda-toolkit-path>/lib64-omyCublasApp

Note that in the latter case, the librarycuda is not needed. The CUDA Runtime will try to open explicitly thecuda library if needed. In the case of a system which does not have the CUDA driver installed, this allows the application to gracefully manage this issue and potentially run if a CPU-only path is available.

Starting with release 11.2, using the typed functions instead of the extension functions (cublas**Ex()) helps in reducing the binary size when linking to static cuBLAS Library.

2.1.10.GEMM Algorithms Numerical Behavior

Some GEMM algorithms split the computation along the dimension K to increase the GPU occupancy, especially when the dimension K is large compared to dimensions M and N. When this type of algorithm is chosen by the cuBLAS heuristics or explicitly by the user, the results of each split is summed deterministically into the resulting matrix to get the final result.

For the routinescublas<t>gemmEx() andcublasGemmEx(), when the compute type is greater than the output type, the sum of the split chunks can potentially lead to some intermediate overflows thus producing a final resulting matrix with some overflows. Those overflows might not have occurred if all the dot products had been accumulated in the compute type before being converted at the end in the output type. This computation side-effect can be easily exposed when the computeType isCUDA_R_32F and Atype, Btype and Ctype areCUDA_R_16F. This behavior can be controlled using the compute precision modeCUBLAS_MATH_DISALLOW_REDUCED_PRECISION_REDUCTION withcublasSetMathMode()

2.1.11.Tensor Core Usage

Tensor cores were first introduced with Volta GPUs (compute capability 7.0 and above) and significantly accelerate matrix multiplications. Starting with cuBLAS version 11.0.0, the library may automatically make use of Tensor Core capabilities wherever possible, unless they are explicitly disabled by selecting pedantic compute modes in cuBLAS (seecublasSetMathMode(),cublasMath_t).

It should be noted that the library will pick a Tensor Core enabled implementation wherever it determines that it would provide the best performance.

The best performance when using Tensor Cores can be achieved when the matrix dimensions and pointers meet certain memory alignment requirements. Specifically, all of the following conditions must be satisfied to get the most performance out of Tensor Cores:

• ((op_A==CUBLAS_OP_N?m:k)*AtypeSize)%16==0

• ((op_B==CUBLAS_OP_N?k:n)*BtypeSize)%16==0

• (m*CtypeSize)%16==0

• (lda*AtypeSize)%16==0

• (ldb*BtypeSize)%16==0

• (ldc*CtypeSize)%16==0

• intptr_t(A)%16==0

• intptr_t(B)%16==0

• intptr_t(C)%16==0

To conduct matrix multiplication with FP8 types (see8-bit Floating Point Data Types (FP8) Usage), you must ensure that your matrix dimensions and pointers meet the optimal requirements listed above. Aside from FP8, there are no longer any restrictions on matrix dimensions and memory alignments to use Tensor Cores (starting with cuBLAS version 11.0.0).

2.1.12.CUDA Graphs Support

cuBLAS routines can be captured in CUDA Graph stream capture without restrictions in most situations.

The exception are routines that output results into host buffers (e.g.cublas<t>dot() while pointer modeCUBLAS_POINTER_MODE_HOST is configured), as it enforces synchronization.

For input coefficients (such asalpha,beta) behavior depends on the pointer mode setting:

• In the case ofCUBLAS(LT)_POINTER_MODE_HOST, coefficient values are captured in the graph.

• In the case of pointer modes with device pointers, coefficient value is accessed using the device pointer at the time of graph execution.

2.1.13.64-bit Integer Interface

cuBLAS version 12 introduced 64-bit integer capable functions. Each 64-bit integer function is equivalent to a 32-bit integer function with the following changes:

• The function name has_64 suffix.

• The dimension (problem size) data type changed fromint toint64_t. Examples of dimension:m,n, andk.

• The leading dimension data type changed fromint toint64_t. Examples of leading dimension:lda,ldb, andldc.

• The vector increment data type changed fromint toint64_t. Examples of vector increment:incx andincy.

For example, consider the following 32-bit integer functions:

cublasStatus_tcublasSetMatrix(introws,intcols,intelemSize,constvoid*A,intlda,void*B,intldb);cublasStatus_tcublasIsamax(cublasHandle_thandle,intn,constfloat*x,intincx,int*result);cublasStatus_tcublasSsyr(cublasHandle_thandle,cublasFillMode_tuplo,intn,constfloat*alpha,constfloat*x,intincx,float*A,intlda);

The equivalent 64-bit integer functions are:

cublasStatus_tcublasSetMatrix_64(int64_trows,int64_tcols,int64_telemSize,constvoid*A,int64_tlda,void*B,int64_tldb);cublasStatus_tcublasIsamax_64(cublasHandle_thandle,int64_tn,constfloat*x,int64_tincx,int64_t*result);cublasStatus_tcublasSsyr_64(cublasHandle_thandle,cublasFillMode_tuplo,int64_tn,constfloat*alpha,constfloat*x,int64_tincx,float*A,int64_tlda);

Not every function has a 64-bit integer equivalent. For instance,cublasSetMathMode() doesn’t have any arguments that could meaningfully beint64_t. For documentation brevity, the 64-bit integer APIs are not explicitly listed, but only mentioned that they exist for the relevant functions.

2.2.1.cublasHandle_t

ThecublasHandle_t type is a pointer type to an opaque structure holding the cuBLAS library context. The cuBLAS library context must be initialized usingcublasCreate() and the returned handle must be passed to all subsequent library function calls. The context should be destroyed at the end usingcublasDestroy().

2.2.2.cublasStatus_t

The type is used for function status returns. All cuBLAS library functions return their status, which can have the following values.

2.2.3.cublasOperation_t

ThecublasOperation_t type indicates which operation needs to be performed with the dense matrix. Its values correspond to Fortran characters‘N’ or‘n’ (non-transpose),‘T’ or‘t’ (transpose) and‘C’ or‘c’ (conjugate transpose) that are often used as parameters to legacy BLAS implementations.

2.2.4.cublasFillMode_t

The type indicates which part (lower or upper) of the dense matrix was filled and consequently should be used by the function. Its values correspond to Fortran charactersL orl (lower) andU oru (upper) that are often used as parameters to legacy BLAS implementations.

2.2.5.cublasDiagType_t

The type indicates whether the main diagonal of the dense matrix is unity and consequently should not be touched or modified by the function. Its values correspond to Fortran characters‘N’ or‘n’ (non-unit) and‘U’ or‘u’ (unit) that are often used as parameters to legacy BLAS implementations.

2.2.6.cublasSideMode_t

The type indicates whether the dense matrix is on the left or right side in the matrix equation solved by a particular function. Its values correspond to Fortran characters‘L’ or‘l’ (left) and‘R’ or‘r’ (right) that are often used as parameters to legacy BLAS implementations.

2.2.7.cublasPointerMode_t

ThecublasPointerMode_t type indicates whether the scalar values are passed by reference on the host or device. It is important to point out that if several scalar values are present in the function call, all of them must conform to the same single pointer mode. The pointer mode can be set and retrieved usingcublasSetPointerMode() andcublasGetPointerMode() routines, respectively.

2.2.8.cublasAtomicsMode_t

The type indicates whether cuBLAS routines which has an alternate implementation using atomics can be used. The atomics mode can be set and queried usingcublasSetAtomicsMode() andcublasGetAtomicsMode() and routines, respectively.

2.2.9.cublasGemmAlgo_t

cublasGemmAlgo_t type is an enumerant to specify the algorithm for matrix-matrix multiplication on GPU architectures up tosm_75. Onsm_80 and newer GPU architectures, this enumarant has no effect. cuBLAS has the following algorithm options:

2.2.10.cublasMath_t

cublasMath_t enumerate type is used incublasSetMathMode() to choose compute precision modes as defined in the following table. Since this setting does not directly control the use of Tensor Cores, the modeCUBLAS_TENSOR_OP_MATH is being deprecated, and will be removed in a future release.

2.2.11.cublasComputeType_t

cublasComputeType_t enumerate type is used incublasGemmEx() andcublasLtMatmul() (including all batched and strided batched variants) to choose compute precision modes as defined below.

2.2.12.cublasEmulationStrategy_t

cublasEmulationStrategy_t enumerate type is used incublasSetEmulationStrategy() to choose how to leverage floating point emulation algorithms.

2.3.1.cudaDataType_t

ThecudaDataType_t type is an enumerant to specify the data precision. It is used when the data reference does not carry the type itself (e.g void *)

For example, it is used in the routinecublasSgemmEx().

2.3.2.cudaEmulationStrategy_t

ThecudaEmulationStrategy_t is a parameter to specify how to leverage floating point emulation algorithms. This is equivalent tocublasEmulationStrategy_t.

2.3.3.cudaEmulationMantissaControl_t

ThecudaEmulationMantissaControl_t is an enumerated type to specify how to configure how the number of mantissa bits are calculated in floating point emulation algorithms.See SeecublasSetFixedPointEmulationMantissaControl() andcublasGetFixedPointEmulationMaxMantissaBitCount().

2.3.4.cudaEmulationSpecialValuesSupport_t

ThecudaEmulationSpecialValuesSupport_t is an enumerated type to specify how to configure which floating point special values are required to be supported byfloating point emulation algorithms. SeecublasSetEmulationSpecialValuesSupport() andcublasGetEmulationSpecialValuesSupport().

2.3.5.libraryPropertyType_t

ThelibraryPropertyType_t is used as a parameter to specify which property is requested when using the routinecublasGetProperty()

2.4.1.cublasCreate()

cublasStatus_tcublasCreate(cublasHandle_t*handle)

This function initializes the cuBLAS library and creates a handle to an opaque structure holding the cuBLAS library context. It allocates hardware resources on the host and device and must be called prior to making any other cuBLAS library calls.

The cuBLAS library context is tied to the current CUDA device. To use the library on multiple devices, one cuBLAS handle needs to be created for each device. See alsocuBLAS Context.

For a given device, multiple cuBLAS handles with different configurations can be created. For multi-threaded applications that use the same device from different threads, the recommended programming model is to create one cuBLAS handle per thread and use that cuBLAS handle for the entire life of the thread.

BecausecublasCreate() allocates some internal resources and the release of those resources by callingcublasDestroy() will implicitly callcudaDeviceSynchronize(), it is recommended to minimize the number of times these functions are called.

2.4.2.cublasDestroy()

cublasStatus_tcublasDestroy(cublasHandle_thandle)

This function releases hardware resources used by the cuBLAS library. This function is usually the last call with a particular handle to the cuBLAS library. BecausecublasCreate() allocates some internal resources and the release of those resources by callingcublasDestroy() will implicitly callcudaDeviceSynchronize(), it is recommended to minimize the number of times these functions are called.

2.4.3.cublasGetVersion()

cublasStatus_tcublasGetVersion(cublasHandle_thandle,int*version)

This function returns the version number of the cuBLAS library.

2.4.4.cublasGetProperty()

cublasStatus_tcublasGetProperty(libraryPropertyTypetype,int*value)

This function returns the value of the requested property in memory pointed to by value. Refer tolibraryPropertyType for supported types.

2.4.5.cublasGetStatusName()

constchar*cublasGetStatusName(cublasStatus_tstatus)

This function returns the string representation of a given status.

2.4.6.cublasGetStatusString()

constchar*cublasGetStatusString(cublasStatus_tstatus)

This function returns the description string for a given status.

2.4.7.cublasSetStream()

cublasStatus_tcublasSetStream(cublasHandle_thandle,cudaStream_tstreamId)

This function sets the cuBLAS library stream, which will be used to execute all subsequent calls to the cuBLAS library functions. If the cuBLAS library stream is not set, all kernels use thedefault NULL stream. In particular, this routine can be used to change the stream between kernel launches and then to reset the cuBLAS library stream back to NULL. Additionally this function unconditionally resets the cuBLAS library workspace back to the default workspace pool (seecublasSetWorkspace()).

2.4.8.cublasSetWorkspace()

cublasStatus_tcublasSetWorkspace(cublasHandle_thandle,void*workspace,size_tworkspaceSizeInBytes)

This function sets the cuBLAS library workspace to a user-owned device buffer, which will be used to execute all subsequent calls to the cuBLAS library functions (on the currently set stream). If the cuBLAS library workspace is not set, all kernels will use the default workspace pool allocated during the cuBLAS context creation. In particular, this routine can be used to change the workspace between kernel launches. The workspace pointer has to be aligned to at least 256 bytes, otherwiseCUBLAS_STATUS_INVALID_VALUE error is returned. ThecublasSetStream() function unconditionally resets the cuBLAS library workspace back to the default workspace pool. Calling this function, including withworkspaceSizeInBytes equal to 0, will prevent the cuBLAS library from utilizing the default workspace. Too small value ofworkspaceSizeInBytes may cause some routines to fail withCUBLAS_STATUS_ALLOC_FAILED error returned or cause large regressions in performance. Workspace size equal to or larger than 16KiB is enough to preventCUBLAS_STATUS_ALLOC_FAILED error, while a larger workspace can provide performance benefits for some routines.

The table below shows the recommended size of user-provided workspace.This is based on the cuBLAS default workspace pool size which is GPU architecture dependent.

The possible error values returned by this function and their meanings are listed below.

2.4.9.cublasGetStream()

cublasStatus_tcublasGetStream(cublasHandle_thandle,cudaStream_t*streamId)

This function gets the cuBLAS library stream, which is being used to execute all calls to the cuBLAS library functions. If the cuBLAS library stream is not set, all kernels use thedefault NULL stream.

2.4.10.cublasGetPointerMode()

cublasStatus_tcublasGetPointerMode(cublasHandle_thandle,cublasPointerMode_t*mode)

This function obtains the pointer mode used by the cuBLAS library. Please see the section on thecublasPointerMode_t type for more details.

2.4.11.cublasSetPointerMode()

cublasStatus_tcublasSetPointerMode(cublasHandle_thandle,cublasPointerMode_tmode)

This function sets the pointer mode used by the cuBLAS library. Thedefault is for the values to be passed by reference on the host. Please see the section on thecublasPointerMode_t type for more details.

2.4.12.cublasSetVector()

cublasStatus_tcublasSetVector(intn,intelemSize,constvoid*x,intincx,void*y,intincy)

This function supports the64-bit Integer Interface.

This function copiesn elements from a vectorx in host memory space to a vectory in GPU memory space. Elements in both vectors are assumed to have a size ofelemSize bytes. The storage spacing between consecutive elements is given byincx for the source vectorx and byincy for the destination vectory.

Since column-major format for two-dimensional matrices is assumed, if a vector is part of a matrix, a vector increment equal to1 accesses a (partial) column of that matrix. Similarly, using an increment equal to the leading dimension of the matrix results in accesses to a (partial) row of that matrix.

2.4.13.cublasGetVector()

cublasStatus_tcublasGetVector(intn,intelemSize,constvoid*x,intincx,void*y,intincy)

This function supports the64-bit Integer Interface.

This function copiesn elements from a vectorx in GPU memory space to a vectory in host memory space. Elements in both vectors are assumed to have a size ofelemSize bytes. The storage spacing between consecutive elements is given byincx for the source vector andincy for the destination vectory.

Since column-major format for two-dimensional matrices is assumed, if a vector is part of a matrix, a vector increment equal to1 accesses a (partial) column of that matrix. Similarly, using an increment equal to the leading dimension of the matrix results in accesses to a (partial) row of that matrix.

2.4.14.cublasSetMatrix()

cublasStatus_tcublasSetMatrix(introws,intcols,intelemSize,constvoid*A,intlda,void*B,intldb)

This function supports the64-bit Integer Interface.

This function copies a tile ofrowsxcols elements from a matrixA in host memory space to a matrixB in GPU memory space. It is assumed that each element requires storage ofelemSize bytes and that both matrices are stored in column-major format, with the leading dimension of the source matrixA and destination matrixB given inlda andldb, respectively. The leading dimension indicates the number of rows of the allocated matrix, even if only a submatrix of it is being used.

2.4.15.cublasGetMatrix()

cublasStatus_tcublasGetMatrix(introws,intcols,intelemSize,constvoid*A,intlda,void*B,intldb)

This function supports the64-bit Integer Interface.

This function copies a tile ofrowsxcols elements from a matrixA in GPU memory space to a matrixB in host memory space. It is assumed that each element requires storage ofelemSize bytes and that both matrices are stored in column-major format, with the leading dimension of the source matrixA and destination matrixB given inlda andldb, respectively. The leading dimension indicates the number of rows of the allocated matrix, even if only a submatrix of it is being used.

2.4.16.cublasSetVectorAsync()

cublasStatus_tcublasSetVectorAsync(intn,intelemSize,constvoid*hostPtr,intincx,void*devicePtr,intincy,cudaStream_tstream)

This function supports the64-bit Integer Interface.

This function has the same functionality ascublasSetVector(), with the exception that the data transfer is done asynchronously (with respect to the host) using the given CUDA™ stream parameter.

2.4.17.cublasGetVectorAsync()

cublasStatus_tcublasGetVectorAsync(intn,intelemSize,constvoid*devicePtr,intincx,void*hostPtr,intincy,cudaStream_tstream)

This function supports the64-bit Integer Interface.

This function has the same functionality ascublasGetVector(), with the exception that the data transfer is done asynchronously (with respect to the host) using the given CUDA™ stream parameter.

2.4.18.cublasSetMatrixAsync()

cublasStatus_tcublasSetMatrixAsync(introws,intcols,intelemSize,constvoid*A,intlda,void*B,intldb,cudaStream_tstream)

This function supports the64-bit Integer Interface.

This function has the same functionality ascublasSetMatrix(), with the exception that the data transfer is done asynchronously (with respect to the host) using the given CUDA™ stream parameter.

2.4.19.cublasGetMatrixAsync()

cublasStatus_tcublasGetMatrixAsync(introws,intcols,intelemSize,constvoid*A,intlda,void*B,intldb,cudaStream_tstream)

This function supports the64-bit Integer Interface.

This function has the same functionality ascublasGetMatrix(), with the exception that the data transfer is done asynchronously (with respect to the host) using the given CUDA™ stream parameter.

2.4.20.cublasSetAtomicsMode()

cublasStatus_tcublasSetAtomicsMode(cublasHandlethandle,cublasAtomicsMode_tmode)

Some routines likecublas<t>symv() andcublas<t>hemv() have an alternate implementation that use atomics to cumulate results. This implementation is generally significantly faster but can generate results that are not strictly identical from one run to the others. Mathematically, those different results are not significant but when debugging those differences can be prejudicial.

This function allows or disallows the usage of atomics in the cuBLAS library for all routines which have an alternate implementation. When not explicitly specified in the documentation of any cuBLAS routine, it means that this routine does not have an alternate implementation that use atomics. When atomics mode is disabled, each cuBLAS routine should produce the same results from one run to the other when called with identical parameters on the same Hardware.

The default atomics mode of default initializedcublasHandle_t object isCUBLAS_ATOMICS_NOT_ALLOWED. Please see the section on the type for more details.

2.4.21.cublasGetAtomicsMode()

cublasStatus_tcublasGetAtomicsMode(cublasHandle_thandle,cublasAtomicsMode_t*mode)

This function queries the atomic mode of a specific cuBLAS context.

The default atomics mode of default initializedcublasHandle_t object isCUBLAS_ATOMICS_NOT_ALLOWED. Please see the section on the type for more details.

2.4.22.cublasSetMathMode()

cublasStatus_tcublasSetMathMode(cublasHandle_thandle,cublasMath_tmode)

ThecublasSetMathMode() function enables you to choose the compute precision modes as defined bycublasMath_t. Users are allowed to set the compute precision mode as a logical combination of them (except the deprecatedCUBLAS_TENSOR_OP_MATH). For example,cublasSetMathMode(handle,CUBLAS_DEFAULT_MATH|CUBLAS_MATH_DISALLOW_REDUCED_PRECISION_REDUCTION). Please note that the default math mode isCUBLAS_DEFAULT_MATH.

For matrix and compute precisions allowed forcublasGemmEx() andcublasLtMatmul() APIs and their strided variants please refer to:cublasGemmEx() ,cublasGemmBatchedEx(),cublasGemmStridedBatchedEx(), andcublasLtMatmul().

2.4.23.cublasGetMathMode()

cublasStatus_tcublasGetMathMode(cublasHandle_thandle,cublasMath_t*mode)

This function returns the math mode used by the library routines.

2.4.24.cublasSetSmCountTarget()

cublasStatus_tcublasSetSmCountTarget(cublasHandle_thandle,intsmCountTarget)

ThecublasSetSmCountTarget() function allows overriding the number of multiprocessors available to the library during kernels execution.

This option can be used to improve the library performance when cuBLAS routines are known to run concurrently with other work on different CUDA streams. For example, on an NVIDIA A100 GPU, which has 108 multiprocessors, when there is a concurrent kenrel running with grid size of 8, one can usecublasSetSmCountTarget() withsmCountTarget set to100 to override the library heuristics to optimize for running on the remaining 100 multiprocessors.

When set to0, the library returns to its default behavior. The input value should not exceed the device’s multiprocessor count, which can be obtained usingcudaDeviceGetAttribute. Negative values are not accepted.

The user must ensure thread safety when modifying the library handle with this routine similar to when usingcublasSetStream(), etc.

2.4.25.cublasGetSmCountTarget()

cublasStatus_tcublasGetSmCountTarget(cublasHandle_thandle,int*smCountTarget)

This function obtains the value previously programmed to the library handle.

2.4.26.cublasSetEmulationStrategy()

cublasStatus_tcublasSetEmulationStrategy(cublasHandle_thandle,cublasEmulationStrategy_temulationStrategy)

ThecublasSetEmulationStrategy() function enables you to select how the library should make use offloating point emulation. For more details, please seecublasEmulationStrategy_t.

2.4.27.cublasGetEmulationStrategy()

cublasStatus_tcublasGetEmulationStrategy(cublasHandle_thandle,cublasEmulationStrategy_t*emulationStrategy)

This function obtains the value previously programmed to the library handle.

2.4.28.cublasGetEmulationSpecialValuesSupport()

cublasStatus_tcublasGetEmulationSpecialValuesSupport(cublasHandle_thandle,cudaEmulationSpecialValuesSupport*mask)

This function obtains the value previously programmed to the library handle.

2.4.29.cublasSetEmulationSpecialValuesSupport()

cublasStatus_tcublasSetEmulationSpecialValuesSupport(cublasHandle_thandle,cudaEmulationSpecialValuesSupportmask)

This function sets the value previously programmed to the library handle.

2.4.30.cublasGetFixedPointEmulationMantissaControl()

cublasStatus_tcublasGetFixedPointEmulationMantissaControl(cublasHandle_thandle,cudaEmulationMantissaControl*mantissaControl)

This function obtains the value previously programmed to the library handle.

2.4.31.cublasSetFixedPointEmulationMantissaControl()

cublasStatus_tcublasSetFixedPointEmulationMantissaControl(cublasHandle_thandle,cudaEmulationMantissaControlmantissaControl)

This function sets the value previously programmed to the library handle.

2.4.32.cublasGetFixedPointEmulationMaxMantissaBitCount()

cublasStatus_tcublasGetFixedPointEmulationMaxMantissaBitCount(cublasHandle_thandle,int*maxMantissaBitCount)

This function obtains the value previously programmed to the library handle.

2.4.33.cublasSetFixedPointEmulationMaxMantissaBitCount()

cublasStatus_tcublasSetFixedPointEmulationMaxMantissaBitCount(cublasHandle_thandle,intmaxMantissaBitCount)

This function sets the value previously programmed to the library handle.

2.4.34.cublasGetFixedPointEmulationMantissaBitOffset()

cublasStatus_tcublasGetFixedPointEmulationMantissaBitOffset(cublasHandle_thandle,int*mantissaBitOffset)

This function obtains the value previously programmed to the library handle.

2.4.35.cublasSetFixedPointEmulationMantissaBitOffset()

cublasStatus_tcublasSetFixedPointEmulationMantissaBitOffset(cublasHandle_thandle,intmantissaBitOffset)

This function sets the value previously programmed to the library handle.

2.4.36.cublasGetFixedPointEmulationMantissaBitCountPointer()

cublasStatus_tcublasGetFixedPointEmulationMantissaBitCountPointer(cublasHandle_thandle,int**mantissaBitCount)

This function obtains the value previously programmed to the library handle.

2.4.37.cublasSetFixedPointEmulationMantissaBitCountPointer()

cublasStatus_tcublasSetFixedPointEmulationMantissaBitCountPointer(cublasHandle_thandle,int*mantissaBitCount)

This function sets the value previously programmed to the library handle.

2.4.38.cublasLoggerConfigure()

cublasStatus_tcublasLoggerConfigure(intlogIsOn,intlogToStdOut,intlogToStdErr,constchar*logFileName)

This function configures logging during runtime. Besides this type of configuration, it is possible to configure logging with special environment variables which will be checked by libcublas:

• CUBLAS_LOGINFO_DBG - setting this environment variable to1 means turning logging on (by default logging is off).

• CUBLAS_LOGDEST_DBG - this environment variable encodes where to write the log to:stdout,stderr mean to write log messages to standard output or error streams, respectively. Other values are interpreted as file names.

Parameters

2.4.39.cublasGetLoggerCallback()

cublasStatus_tcublasGetLoggerCallback(cublasLogCallback*userCallback)

This function retrieves function pointer to previously installed custom user defined callback function viacublasSetLoggerCallback() or zero otherwise.

The possible error values returned by this function and their meanings are listed below.

2.4.40.cublasSetLoggerCallback()

cublasStatus_tcublasSetLoggerCallback(cublasLogCallbackuserCallback)

This function installs a custom user defined callback function via cublas C public API.

2.5.1.cublasI<t>amax()

cublasStatus_tcublasIsamax(cublasHandle_thandle,intn,constfloat*x,intincx,int*result)cublasStatus_tcublasIdamax(cublasHandle_thandle,intn,constdouble*x,intincx,int*result)cublasStatus_tcublasIcamax(cublasHandle_thandle,intn,constcuComplex*x,intincx,int*result)cublasStatus_tcublasIzamax(cublasHandle_thandle,intn,constcuDoubleComplex*x,intincx,int*result)

This function supports the64-bit Integer Interface.

This function finds the (smallest) index of the element of the maximum magnitude. Hence, the result is the first\(i\) such that\(\left| \mathbf{Im}\left( {x\lbrack j\rbrack} \right) \middle| + \middle| \mathbf{Re}\left( {x\lbrack j\rbrack} \right) \right|\) is maximum for\(i = 1,\ldots,n\) and\(j = 1 + \left( {i - 1} \right)*\text{ incx}\) . Notice that the last equation reflects 1-based indexing used for compatibility with Fortran.

The possible error values returned by this function and their meanings are listed below.

For references please refer to NETLIB documentation:

isamax(),idamax(),icamax(),izamax()

2.5.2.cublasI<t>amin()

cublasStatus_tcublasIsamin(cublasHandle_thandle,intn,constfloat*x,intincx,int*result)cublasStatus_tcublasIdamin(cublasHandle_thandle,intn,constdouble*x,intincx,int*result)cublasStatus_tcublasIcamin(cublasHandle_thandle,intn,constcuComplex*x,intincx,int*result)cublasStatus_tcublasIzamin(cublasHandle_thandle,intn,constcuDoubleComplex*x,intincx,int*result)

This function supports the64-bit Integer Interface.

This function finds the (smallest) index of the element of the minimum magnitude. Hence, the result is the first\(i\) such that\(\left| \mathbf{Im}\left( {x\lbrack j\rbrack} \right) \middle| + \middle| \mathbf{Re}\left( {x\lbrack j\rbrack} \right) \right|\) is minimum for\(i = 1,\ldots,n\) and\(j = 1 + \left( {i - 1} \right)*\text{incx}\) Notice that the last equation reflects 1-based indexing used for compatibility with Fortran.

The possible error values returned by this function and their meanings are listed below.

For references please refer to NETLIB documentation:

isamin()

2.5.3.cublas<t>asum()

cublasStatus_tcublasSasum(cublasHandle_thandle,intn,constfloat*x,intincx,float*result)cublasStatus_tcublasDasum(cublasHandle_thandle,intn,constdouble*x,intincx,double*result)cublasStatus_tcublasScasum(cublasHandle_thandle,intn,constcuComplex*x,intincx,float*result)cublasStatus_tcublasDzasum(cublasHandle_thandle,intn,constcuDoubleComplex*x,intincx,double*result)

This function supports the64-bit Integer Interface.

This function computes the sum of the absolute values of the elements of vectorx. Hence, the result is\(\left. \sum_{i = 1}^{n} \middle| \mathbf{Im}\left( {x\lbrack j\rbrack} \right) \middle| + \middle| \mathbf{Re}\left( {x\lbrack j\rbrack} \right) \right|\) where\(j = 1 + \left( {i - 1} \right)*\text{incx}\) . Notice that the last equation reflects 1-based indexing used for compatibility with Fortran.

The possible error values returned by this function and their meanings are listed below.

For references please refer to NETLIB documentation:

sasum(),dasum(),scasum(),dzasum()

2.5.4.cublas<t>axpy()

cublasStatus_tcublasSaxpy(cublasHandle_thandle,intn,constfloat*alpha,constfloat*x,intincx,float*y,intincy)cublasStatus_tcublasDaxpy(cublasHandle_thandle,intn,constdouble*alpha,constdouble*x,intincx,double*y,intincy)cublasStatus_tcublasCaxpy(cublasHandle_thandle,intn,constcuComplex*alpha,constcuComplex*x,intincx,cuComplex*y,intincy)cublasStatus_tcublasZaxpy(cublasHandle_thandle,intn,constcuDoubleComplex*alpha,constcuDoubleComplex*x,intincx,cuDoubleComplex*y,intincy)