For more details on`max_batch_size` and`max_num_tokens`, refer to[Tuning Max Batch Size and Max Num Tokens](../performance/performance-tuning-guide/tuning-max-batch-size-and-max-num-tokens.md).

MLA currently supports the chunked context feature on both Hopper and Blackwell GPUs. You can use`--enable_chunked_context` to enable it. This feature is primarily designed to reduce TPOT (Time Per Output Token). The default chunk size is set to`max_num_tokens`. If you want to achieve a lower TPOT, you can appropriately reduce the chunk size. However, please note that this will also decrease overall throughput. Therefore, a trade-off needs to be considered.

MLA chunked context support has been added on Blackwell GPUs, while it's not supported on Hopper yet. On Hopper, note that`max_num_tokens` has to be at least larger than the max input sequence length of the samples in dataset.

For more details on`max_num_tokens`, refer to[Tuning Max Batch Size and Max Num Tokens](../performance/performance-tuning-guide/tuning-max-batch-size-and-max-num-tokens.md).

KV cache reuse is supported for MLA on SM90 and SM100. It is enabled by default. Due to extra operations like memcpy and GEMMs, GPU memory consumption may be higher and the E2E performance may have regression in some cases. Users could pass`KvCacheConfig(enable_block_reuse=False)` to LLM API to disable it.

Chunked Prefill is supported for MLA only on SM100 currently. You should add`--enable_chunked_prefill` to enable it. The GPU memory consumption is highly correlated with`max_num_tokens` and`max_batch_size`. If encountering out-of-memory errors, you may make these values smaller. (`max_num_tokens` must be divisible by kv cache's`tokens_per_block`)

Chunked Prefill is supported for MLA only onSM90 andSM100 currently. You should add`--enable_chunked_prefill` to enable it. The GPU memory consumption is highly correlated with`max_num_tokens` and`max_batch_size`. If encountering out-of-memory errors, you may make these values smaller. (`max_num_tokens` must be divisible by kv cache's`tokens_per_block`)

More specifically, we can imitate what we did in the[Quick Start](#quick-start):