What is the memory impact (or other issues) if we keep the KV cache related data in variable instead of writing them to disk file?

Should we add an assert/exception/suitable-safe-early-return here if the input model is not GenAI based i.e. it doesn't have expected IO binding / names? (e.g. if this list is empty or if there are no GQA nodes seen etc.?)

I think we currently support 8-bit KV Cache with GenAI Builder exported ONNX LLMs only, right?

For INT4 AWQ/RTN + 8-bit KV Cache quantization, can we avoid 2 session runs by preparing KV tensor names before creating augmented model, augmenting model for these KV tensors too, and post-processing for save-kv-cache-calib-data after AWQ/RTN loop?

Just checking if we can avoid 2 session runs, and thereby speedup the combined quantization of matmul and kv-cache.

Yes, that's possible, but this change won't apply to int8/fp8 path and for awq_lite, awq_clip, rtn, they need to be implemented separately which means not much code can be reused. If you fell it's worthy, I can definitely implement in this way.

I think we can simplify this a bit by creating a map, with assert/valueError for unsupported type. Something like below:

output.type.tensor_type.elemt_type = output_type_map[kv_cache_type]

where output_type_map = {"int8": , "fp8": }

Possibly we can create a util for validating dtype, model, input model - whether it is currently supported or not.

I didn't get how KV-Cache quantization meta-data is used with int8/fp8 quantization. Can you please elaborate the flow?

Both int8/fp8 quantization callquantize_static fromort_patching. The change happens inort_pathching.py. Ifkv_quant_mode is not None, it will save additional calibration data on disk.

I think if kv-quant-type is per-channel then things wont go well, since we are not supporting it but not checking / flagging it as well. Is it?

Add the check

Does this method require matching function signature with the corresponding method from onnxruntime? I guess not so but would like to be clear.