Enabling TunableOp and Tuning Separately#

The TunableOp feature is enabled separately from enabling the tuning phaseitself. Enabling TunableOp means that PyTorch will replace any standardoperators with their Tunable implementations. Any call to a TunableOp firstchecks whether it has already been tuned for the given operator inputs. If so,it will immediately call the tuned operation; no further tuning will take placeeven when the tuning setting is enabled. Instead if no tuning result is found,and tuning is enabled, the TunableOp will benchmark every registeredimplementation of that operator for the given set of inputs and select thefastest.

File Input and Output#

The first time any TunableOp is invoked, the internal database of tunedoperations will be prepared by attempting to read the results from the givenfile. The default filename is ‘tunableop_results.csv’. To support tuning whenmultiple GPUs are used across multiple processes, the GPU device ordinal isautomatically inserted into the filename to avoid multiple processes overwritingthe same file.

If tuning is enabled and new tunings are discovered during the course of yourworkload, it will also write out to this same filename with all tunings, boththe ones it read in at startup as well as the new ones found at runtime. Thiscan be used, for example, to build up a tunings file across many workloads byreusing the same file. The output file is automatically created when theapplication terminates. This behavior can be controlled by the C++ and PythonAPIs but not the environment variables.

Assuming you specified a filename, you’ll end up with a CSV file with contentslike so:

Validator,PT_VERSION,2.2.0Validator,ROCM_VERSION,6.0.0.0-12969-1544e39Validator,HIPBLASLT_VERSION,0.6.0-a9c5cc7Validator,ROCBLAS_VERSION,4.0.0-72e57364-dirtyGemmTunableOp_float_NT,nt_25088_4096_64,Gemm_Hipblaslt_1219,1.262GemmTunableOp_float_NT,nt_4096_4096_64,Gemm_Rocblas_1216,0.033

Note the “Validator” lines. If you change a library version, or ROCm version, orPyTorch version, TunableOp will detect this and reject the tunings file becausethe prior tunings are likely affected by other software changes.

The remaining lines are the tuned solutions for each TunableOp encounteredduring your execution. Each line consists of 4 comma-separated fields: operatorname, operator parameters, solution name, and average execution time. Theexecution time is an optional field. The CSV file can be edited, but withcaution. For example, the solution name (field 3) can be changed to “Default”and it will fall back to the original PyTorch untuned implementation. Or, in thecase of ROCm’s hipBLAS or hipBLASLt libraries, if you know the specific solutionindex you can override the solution that TunableOp selected by replacing thevalue. The operator name and parameters (fields 1 and 2) are internally namedand should not be modified. In the case of GemmTunableOp, field 1 indicates thedatatype and whether the inputs are transposed (T) or not (N) and field 2indicates the M, N, K input shapes.

There is an option to enable verbose output but it is only recommended fordebugging purposes. This will produce a lot of diagnostic messages but may beuseful to see if TunableOp is being used at all. Otherwise, TunableOp iscompletely silent, besides file output, unless there is a warning or errorduring its use. The verbose option is only available by setting the environmentvariable PYTORCH_TUNABLEOP_VEROBSE=1.

A Note on Tuning Behavior, Warmup, and Cache Effects#

Tuning an operator consists of iterating through the list or registeredimplementations and profiling each one. The profile is established by running asingle implementation in a loop multiple times and taking the average executiontime. There is also an optional warmup phase prior to tuning that can help withreaching stable power states by the hardware. During tuning of a workload thevarious hardware caches will more likely produce hits than when not tuning.There are options for flushing the instruction cache and rotate the input tensorswhich might help produce a more faithful profile of the tuned operator as if theoperator were run within a larger workload instead of in a tight, repetitive loop.

By default, each possible solution for a given operator will be run for either100 iterations or as many iterations that can be run within 30ms, whichever issmaller, and its average execution will be calculated. The fastest solutionamong all that were successfully profiled will be chosen. A profile might failif the given solution doesn’t achieve the same accuracy as the defaultimplementation or if the solution returns an error code.

Current Tunable Operators#

Offline Tuning#

exportPYTORCH_TUNABLEOP_ENABLED=1exportPYTORCH_TUNABLEOP_TUNING=0exportPYTORCH_TUNABLEOP_RECORD_UNTUNED=1...

importtorch.cuda.tunableastunableimportosos.putenv("PYTORCH_TUNABLEOP_ENABLED","1")os.putenv("PYTORCH_TUNABLEOP_TUNING","1")os.putenv("PYTORCH_TUNABLEOP_RECORD_UNTUNED","0")tunable.tune_gemm_in_file("tunableop_untuned0.csv")

if__name__=="__main__":num_gpus=8# number of GPUs that will be used during the tuning processtunable.mgpu_tune_gemm_in_file("tunableop_untuned?.csv",num_gpus)

Tuning Context#

The behavior of TunableOp is currently manipulated through environmentvariables, the C++ interface of at::cuda::tunable::getTuningContext(), or thetorch.cuda.tunable python interfaces. The environment variables take precedenceover any setting you manipulate using the C++ or Python APIs.