Hardware#

The following GPU variants were used for testing:

• H100 SXM 80GB (DGX H100)

• H200 SXM 141GB (DGX H200)

• B200 180GB (DGX B200)

• GB200 192GB (GB200 NVL72)

• RTX 6000 Pro Blackwell Server Edition

Other hardware variants may have different TDP, memory bandwidth, core count, or other features leading to performance differences on these workloads.

FP4 Models#

nvidia/Llama-3.3-70B-Instruct-FP4nvidia/Llama-3.1-405B-Instruct-FP4nvidia/Qwen3-235B-A22B-FP4nvidia/Qwen3-30B-A3B-FP4nvidia/DeepSeek-R1-0528-FP4

FP8 Models#

nvidia/Llama-3.1-8B-Instruct-FP8nvidia/Llama-3.3-70B-Instruct-FP8nvidia/Llama-3.1-405B-Instruct-FP8nvidia/Llama-4-Maverick-17B-128E-Instruct-FP8nvidia/Qwen3-235B-A22B-FP8

Command Overview#

Starting with v0.19, testing was performed using the PyTorch backend - this workflow does not require an engine to be built.

Variables#

Preparing a Dataset#

In order to prepare a dataset, you can use the providedscript.To generate a synthetic dataset, run the following command:

pythonbenchmarks/cpp/prepare_dataset.py--tokenizer=$model_name--stdouttoken-norm-dist--num-requests=$num_requests--input-mean=$isl--output-mean=$osl--input-stdev=0--output-stdev=0>$dataset_file

The command will generate a text file located at the path specified$dataset_file where all requests are of the sameinput/output sequence length combinations. The script works by using the tokenizer to retrieve the vocabulary size andrandomly sample token IDs from it to create entirely random sequences. In the command above, all requests will be uniformbecause the standard deviations for both input and output sequences are set to 0.

For each input and output sequence length combination, the table below details the$num_requests that were used. Forshorter input and output lengths, a larger number of messages were used to guarantee that the system hit a steady statebecause requests enter and exit the system at a much faster rate. For longer input/output sequence lengths, requestsremain in the system longer and therefore require less requests to achieve steady state.

Running the Benchmark#

To run the benchmark with the generated data set, simply use thetrtllm-benchthroughput subcommand. The benchmarker willrun an offline maximum throughput scenario such that all requests are queued in rapid succession. You simply need to providea model name (HuggingFace reference or path to a local model), agenerated dataset, and a file containing any desired extra options to the LLM APIs (details intensorrt_llm/llmapi/llm_args.py:LlmArgs).

For dense / non-MoE models:

trtllm-bench--tp$tp_size--pp$pp_size--model$model_namethroughput--dataset$dataset_file--backendpytorch--extra_llm_api_options$llm_options

llm_options.yml

cuda_graph_config:enable_padding:truebatch_sizes:-1-2-4-8-16-32-64-128-256-384-512-1024-2048-4096-8192

For MoE models:

trtllm-bench--tp$tp_size--pp$pp_size--ep$ep_size--model$model_namethroughput--dataset$dataset_file--backendpytorch--extra_llm_api_options$llm_options

llm_options.yml

enable_attention_dp:truecuda_graph_config:enable_padding:truebatch_sizes:-1-2-4-8-16-32-64-128-256-384-512-1024-2048-4096-8192

In many cases, we also use a higher KV cache percentage by setting--kv_cache_free_gpu_mem_fraction0.95 in the benchmark command. This allows us to obtain better performance than the default setting of0.90. We fall back to0.90 or lower if out-of-memory errors are encountered.

The results will be printed to the terminal upon benchmark completion. For example,

============================================================PERFORMANCEOVERVIEW===========================================================RequestThroughput(req/sec):43.2089TotalOutputThroughput(tokens/sec):5530.7382PerUserOutputThroughput(tokens/sec/user):2.0563PerGPUOutputThroughput(tokens/sec/gpu):5530.7382TotalTokenThroughput(tokens/sec):94022.5497TotalLatency(ms):115716.9214Averagerequestlatency(ms):75903.4456PerUserOutputSpeed[1/TPOT](tokens/sec/user):5.4656Averagetime-to-first-token[TTFT](ms):52667.0339Averagetime-per-output-token[TPOT](ms):182.9639--Per-RequestTime-per-Output-Token[TPOT]Breakdown(ms)[TPOT]MINIMUM:32.8005[TPOT]MAXIMUM:208.4667[TPOT]AVERAGE:182.9639[TPOT]P50:204.0463[TPOT]P90:206.3863[TPOT]P95:206.5064[TPOT]P99:206.5821--Per-RequestTime-to-First-Token[TTFT]Breakdown(ms)[TTFT]MINIMUM:3914.7621[TTFT]MAXIMUM:107501.2487[TTFT]AVERAGE:52667.0339[TTFT]P50:52269.7072[TTFT]P90:96583.7187[TTFT]P95:101978.4566[TTFT]P99:106563.4497--RequestLatencyBreakdown(ms)-----------------------[Latency]P50:78509.2102[Latency]P90:110804.0017[Latency]P95:111302.9101[Latency]P99:111618.2158[Latency]MINIMUM:24189.0838[Latency]MAXIMUM:111668.0964[Latency]AVERAGE:75903.4456

[!WARNING] In some cases, the benchmarker may not print anything at all. This behavior usuallymeans that the benchmark has hit an out of memory issue. Try reducing the KV cache percentageusing the--kv_cache_free_gpu_mem_fraction option to lower the percentage of used memory.