tests/integration/test_lists/qa/llm_function_full.txt (1)

603-605:Optional: expand coverage and watch runtime

• Consider adding one multiturn case for a Qwen2-VL or NVILA model later for broader backend coverage, if runtime budget allows.
• If these prove slow/flake in CI, annotate with TIMEOUT like other heavy tests in this file or place them in a nightly-focused list.

Happy to draft the additional test list entries or help profile runtime to set TIMEOUTs.

tests/integration/defs/test_e2e.py (1)

2600-2601:Avoid dumping full generated output to CI logs

Printing the entire model output can flood CI logs and isn’t needed given the detailed assertion. Remove or gate behind a debug flag.

-    print("output:", output)

Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro

Learnt from: moraxuPR: NVIDIA/TensorRT-LLM#6303File: tests/integration/test_lists/qa/examples_test_list.txt:494-494Timestamp: 2025-07-28T17:06:08.621ZLearning: In TensorRT-LLM testing, it's common to have both CLI flow tests (test_cli_flow.py) and PyTorch API tests (test_llm_api_pytorch.py) for the same model. These serve different purposes: CLI flow tests validate the traditional command-line workflow, while PyTorch API tests validate the newer LLM API backend. Both are legitimate and should coexist.

Learnt from: galagamPR: NVIDIA/TensorRT-LLM#6487File: tests/unittest/_torch/auto_deploy/unit/singlegpu/test_ad_trtllm_bench.py:1-12Timestamp: 2025-08-06T13:58:07.506ZLearning: In TensorRT-LLM, test files (files under tests/ directories) do not require NVIDIA copyright headers, unlike production source code files. Test files typically start directly with imports, docstrings, or code.

Learnt from: moraxuPR: NVIDIA/TensorRT-LLM#6303File: tests/integration/test_lists/qa/examples_test_list.txt:494-494Timestamp: 2025-07-28T17:06:08.621ZLearning: In TensorRT-LLM testing, it's common to have both CLI flow tests (test_cli_flow.py) and PyTorch API tests (test_llm_api_pytorch.py) for the same model. These serve different purposes: CLI flow tests validate the traditional command-line workflow, while PyTorch API tests validate the newer LLM API backend. Both are legitimate and should coexist.

Learnt from: galagamPR: NVIDIA/TensorRT-LLM#6487File: tests/unittest/_torch/auto_deploy/unit/singlegpu/test_ad_trtllm_bench.py:1-12Timestamp: 2025-08-06T13:58:07.506ZLearning: In TensorRT-LLM, test files (files under tests/ directories) do not require NVIDIA copyright headers, unlike production source code files. Test files typically start directly with imports, docstrings, or code.

examples/llm-api/quickstart_multimodal.py (3)

150-158:Narrowexcept and punctuate the docstring (Ruff D415, E722).

Apply this diff:

-def clear_cuda_cache():-    """Clear CUDA cache to prevent memory issues"""-    try:-        import torch-        if torch.cuda.is_available():-            torch.cuda.empty_cache()-    except:-        pass+def clear_cuda_cache():+    """Clear CUDA cache to prevent memory issues."""+    try:+        import torch+    except ImportError:+        # Torch not installed – nothing to clear.+        return+    try:+        if torch.cuda.is_available():+            torch.cuda.empty_cache()+    except Exception as err:+        # Log once, but don't kill the caller.+        print(f"[warn] Failed to clear CUDA cache: {err}")

---

196-209:Respect CLI parameters instead of hard-coded loader defaults.

Apply this diff:

-                inputs = default_multimodal_input_loader(+                inputs = default_multimodal_input_loader(                     tokenizer=llm.tokenizer,                     model_dir=llm._hf_model_dir,                     model_type=model_type,                     modality=args.modality,                     prompts=[cur_prompt],                     media=args.media,-                    image_data_format="pt",-                    num_frames=8,-                    device="cpu")+                    image_data_format=args.image_format,+                    num_frames=args.num_frames,+                    device=args.device)

---

193-239:Conversation history is not persisted between turns – context never grows.

Each iteration uses a freshargs.prompt[i]; the assistant reply appended tocur_prompt is discarded. Persist a running history so the model receives prior turns.

Apply this diff:

-        generated_outputs = []  # Store generated outputs for return+        generated_outputs = []  # Store generated outputs for return+        conversation_history = ""@@-                # Use multimodal input loader to process input with conversation context-                cur_prompt = args.prompt[i]+                # Use multimodal input loader to process input with conversation context+                cur_user = args.prompt[i]+                cur_prompt = (f"{conversation_history} user: {cur_user}"+                              if conversation_history else cur_user)@@-                generated_outputs.append({+                generated_outputs.append({                     "turn": i + 1,-                    "user_input": cur_prompt,+                    "user_input": cur_user,                     "assistant_response": response,                     "media": args.media                 })@@-                # Add to cur_prompt-                cur_prompt = cur_prompt + response+                # Persist conversation context for next turn+                conversation_history = f"{cur_prompt} assistant: {response}"

tests/integration/defs/test_e2e.py (1)

2601-2620:Harden matching, fix Ruff E501, and remove noisy prints in multi-turn check

• Add an assertion that parsed outputs count equals number of prompts to catch parser/format drift early.
• Make keyword matching robust and case-insensitive on both sides.
• Remove debug prints that spam CI logs.
• Split long lines (Line 2610, assert message) to satisfy E501.

-    output = llm_venv.run_cmd(cmd, caller=check_output)-    print("output:", output)-    # Set match ratio based on model-    match_ratio = 4.0 / 5-    if model_name == "Phi-4-multimodal-instruct":-        match_ratio = 0.6--    # Check output accuracy-    for prompt_output, prompt_keywords in zip(-            parse_output(output), expected_keywords[model_name]["image"]):-        matches = [-            keyword in prompt_output.lower() for keyword in prompt_keywords-        ]-        obs_match_ratio = 1. * sum(matches) / len(matches)-        print("prompt_output:", prompt_output)-        print("prompt_keywords:", prompt_keywords)-        print("matches:", matches)-        print("obs_match_ratio:", obs_match_ratio)-        assert obs_match_ratio >= match_ratio, f"Incorrect output!\nGenerated \"{prompt_output}\"\nExpected keywords \"{prompt_keywords}\"\n Matched keywords: {matches}\n Observed match ratio {obs_match_ratio} below threshold {match_ratio}"+    output = llm_venv.run_cmd(cmd, caller=check_output)+    outputs = parse_output(output)+    expected = expected_keywords[model_name]["image"]+    assert len(outputs) == len(expected), (+        f"Expected {len(expected)} outputs, got {len(outputs)}. "+        f"Raw output (truncated): {output[:1000]}..."+    )+    # Set match ratio based on model+    match_ratio = 4.0 / 5+    if model_name == "Phi-4-multimodal-instruct":+        match_ratio = 0.6++    # Check output accuracy+    for prompt_output, prompt_keywords in zip(outputs, expected):+        matches = [keyword.lower() in prompt_output.lower()+                   for keyword in prompt_keywords]+        obs_match_ratio = 1.0 * sum(matches) / len(matches)+        assert obs_match_ratio >= match_ratio, (+            f"Incorrect output!\nGenerated \"{prompt_output}\"\n"+            f"Expected keywords \"{prompt_keywords}\"\n"+            f"Matched keywords: {matches}\n"+            f"Observed match ratio {obs_match_ratio} below threshold {match_ratio}"+        )

examples/llm-api/quickstart_multimodal.py (2)

111-119:Validate--conversation_turns to be >= 1.

Prevent silent no-op when negative by enforcing a positive integer at parse time.

Apply this diff:

-    parser.add_argument(-        "--conversation_turns",-        type=int,-        default=2,-        help="Number of conversation turns for automated testing.")+    parser.add_argument(+        "--conversation_turns",+        type=positive_int,+        default=2,+        help="Number of conversation turns for automated testing (>= 1).")

Add this helper (outside the shown range, e.g., above add_multimodal_args):

defpositive_int(value:str)->int:iv=int(value)ifiv<1:raiseargparse.ArgumentTypeError("conversation_turns must be >= 1")returniv

---

212-217:Redundantmodel_class is None check insideif args.load_lora.

model_class is guaranteed to be set in the earlierif args.load_lora: block; this branch cannot seemodel_class as None.

Apply this diff:

-                if args.load_lora:-                    if model_class is None:-                        raise ValueError(-                            "model_class must be provided when load_lora is True"-                        )-                    lora_request = model_class.lora_request(+                if args.load_lora:+                    lora_request = model_class.lora_request(                         len(inputs), args.modality, llm._hf_model_dir)

Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro

Learnt from: moraxuPR: NVIDIA/TensorRT-LLM#6303File: tests/integration/test_lists/qa/examples_test_list.txt:494-494Timestamp: 2025-07-28T17:06:08.621ZLearning: In TensorRT-LLM testing, it's common to have both CLI flow tests (test_cli_flow.py) and PyTorch API tests (test_llm_api_pytorch.py) for the same model. These serve different purposes: CLI flow tests validate the traditional command-line workflow, while PyTorch API tests validate the newer LLM API backend. Both are legitimate and should coexist.

Learnt from: galagamPR: NVIDIA/TensorRT-LLM#6487File: tests/unittest/_torch/auto_deploy/unit/singlegpu/test_ad_trtllm_bench.py:1-12Timestamp: 2025-08-06T13:58:07.506ZLearning: In TensorRT-LLM, test files (files under tests/ directories) do not require NVIDIA copyright headers, unlike production source code files. Test files typically start directly with imports, docstrings, or code.

Learnt from: moraxuPR: NVIDIA/TensorRT-LLM#6303File: tests/integration/test_lists/qa/examples_test_list.txt:494-494Timestamp: 2025-07-28T17:06:08.621ZLearning: In TensorRT-LLM testing, it's common to have both CLI flow tests (test_cli_flow.py) and PyTorch API tests (test_llm_api_pytorch.py) for the same model. These serve different purposes: CLI flow tests validate the traditional command-line workflow, while PyTorch API tests validate the newer LLM API backend. Both are legitimate and should coexist.

Learnt from: galagamPR: NVIDIA/TensorRT-LLM#6487File: tests/unittest/_torch/auto_deploy/unit/singlegpu/test_ad_trtllm_bench.py:1-12Timestamp: 2025-08-06T13:58:07.506ZLearning: In TensorRT-LLM, test files (files under tests/ directories) do not require NVIDIA copyright headers, unlike production source code files. Test files typically start directly with imports, docstrings, or code.

Learnt from: yibinl-nvidiaPR: NVIDIA/TensorRT-LLM#6506File: examples/models/core/mixtral/requirements.txt:3-3Timestamp: 2025-08-01T15:14:45.673ZLearning: In TensorRT-LLM, examples directory can have different dependency versions than the root requirements.txt file. Version conflicts between root and examples dependencies are acceptable because examples are designed to be standalone and self-contained.

Learnt from: CRPR: NVIDIA/TensorRT-LLM#0File: CODING_GUIDELINES.md:0-0Timestamp: 2025-08-06T21:22:55.018ZLearning: Applies to **/*.py : When using try-except blocks in Python, limit the except to the smallest set of errors possible.

examples/llm-api/quickstart_multimodal.py (4)

150-158:Fix docstring punctuation and avoid bare except in clear_cuda_cache()

Align with Ruff D415/E722; narrow exceptions and log unexpected errors.

Apply:

 def clear_cuda_cache():-    """Clear CUDA cache to prevent memory issues"""+    """Clear CUDA cache to prevent memory issues."""     try:         import torch         if torch.cuda.is_available():             torch.cuda.empty_cache()-    except:-        pass+    except ImportError:+        # Torch not installed – nothing to clear.+        return+    except Exception as err:+        # Log once, but don't kill the caller.+        print(f"[warn] Failed to clear CUDA cache: {err}")

---

196-209:Respect CLI args for image format, frames, and device

Hard-coded values ignore user input; pass through args to the loader.

Apply:

                     prompts=[cur_prompt],                     media=args.media,-                    image_data_format="pt",-                    num_frames=8,-                    device="cpu")+                    image_data_format=args.image_format,+                    num_frames=args.num_frames,+                    device=args.device)

---

239-246:Always clear CUDA cache via finally

Errors skip clear_cuda_cache(); use finally to guarantee cleanup.

Apply:

-            except Exception as e:-                print(f"Error in turn {i+1}: {e}")-                import traceback-                traceback.print_exc()-                continue--            clear_cuda_cache()+            except Exception as e:+                print(f"Error in turn {i+1}: {e}")+                import traceback+                traceback.print_exc()+            finally:+                clear_cuda_cache()

---

236-238:Multi-turn: conversation context is not persisted to next turn

The updated prompt isn’t saved; next iteration reuses the original prompt. Persist it for true multi-turn behavior.

Apply:

-                # Add to cur_prompt-                cur_prompt = cur_prompt + response+                # Persist conversation context for next turn+                args.prompt[i] = f"{cur_prompt} assistant: {response}"

tests/integration/defs/test_e2e.py (2)

2573-2584:Ensure the test exercises all prompts by adding --conversation_turns

Avoid coupling to script defaults; make intent explicit.

Apply:

         "--modality",         "image",         "--multiturn",+        "--conversation_turns",+        str(len(accuracy_inputs["image"]["prompt"])),         "--prompt",

---

2601-2620:Harden checks, reduce log noise, and fix E501 with structured assertions

• Parse once and assert counts.
• Case-insensitive matching.
• Remove verbose prints.
• Break long assert to satisfy E501.

Apply:

-    output = llm_venv.run_cmd(cmd, caller=check_output)-    print("output:", output)+    output = llm_venv.run_cmd(cmd, caller=check_output)+    outputs = parse_output(output)+    expected = expected_keywords[model_name]["image"]+    assert len(outputs) == len(expected), (+        f"Expected {len(expected)} outputs, got {len(outputs)}. "+        f"Raw output (truncated): {output[:1000]}..."+    )@@-    for prompt_output, prompt_keywords in zip(-            parse_output(output), expected_keywords[model_name]["image"]):-        matches = [-            keyword in prompt_output.lower() for keyword in prompt_keywords-        ]-        obs_match_ratio = 1. * sum(matches) / len(matches)-        print("prompt_output:", prompt_output)-        print("prompt_keywords:", prompt_keywords)-        print("matches:", matches)-        print("obs_match_ratio:", obs_match_ratio)-        assert obs_match_ratio >= match_ratio, f"Incorrect output!\nGenerated \"{prompt_output}\"\nExpected keywords \"{prompt_keywords}\"\n Matched keywords: {matches}\n Observed match ratio {obs_match_ratio} below threshold {match_ratio}"+    for prompt_output, prompt_keywords in zip(outputs, expected):+        matches = [keyword.lower() in prompt_output.lower()+                   for keyword in prompt_keywords]+        obs_match_ratio = 1.0 * sum(matches) / len(matches)+        assert obs_match_ratio >= match_ratio, (+            f"Incorrect output!\nGenerated \"{prompt_output}\"\n"+            f"Expected keywords \"{prompt_keywords}\"\n"+            f"Matched keywords: {matches}\n"+            f"Observed match ratio {obs_match_ratio} below threshold {match_ratio}"+        )

examples/llm-api/quickstart_multimodal.py (2)

111-120:Make conversation_turns optional; default to number of provided prompts

Defaulting to 2 is surprising. Let users omit it and derive from len(prompts).

Apply:

-    parser.add_argument(-        "--conversation_turns",-        type=int,-        default=2,-        help="Number of conversation turns for automated testing.")+    parser.add_argument(+        "--conversation_turns",+        type=int,+        default=None,+        help="Number of conversation turns (default: use number of provided prompts).")

---

190-191:Derive max_turns when conversation_turns is omitted

Guard for None to run all provided prompts by default.

Apply:

-        max_turns = min(args.conversation_turns, len(args.prompt))+        max_turns = (len(args.prompt) if args.conversation_turns is None+                     else min(args.conversation_turns, len(args.prompt)))

Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro

Learnt from: moraxuPR: NVIDIA/TensorRT-LLM#6303File: tests/integration/test_lists/qa/examples_test_list.txt:494-494Timestamp: 2025-07-28T17:06:08.621ZLearning: In TensorRT-LLM testing, it's common to have both CLI flow tests (test_cli_flow.py) and PyTorch API tests (test_llm_api_pytorch.py) for the same model. These serve different purposes: CLI flow tests validate the traditional command-line workflow, while PyTorch API tests validate the newer LLM API backend. Both are legitimate and should coexist.

examples/llm-api/quickstart_multimodal.py (4)

150-158:Fix bare except and finish the docstring sentence.

Ruff flags D415 and E722. Limit exceptions and add a period.

-def clear_cuda_cache():-    """Clear CUDA cache to prevent memory issues"""-    try:-        import torch-        if torch.cuda.is_available():-            torch.cuda.empty_cache()-    except:-        pass+def clear_cuda_cache():+    """Clear CUDA cache to prevent memory issues."""+    try:+        import torch+    except ImportError:+        # Torch not installed – nothing to clear.+        return+    try:+        if torch.cuda.is_available():+            torch.cuda.empty_cache()+    except Exception as err:+        # Log once, but don't kill the caller.+        print(f"[warn] Failed to clear CUDA cache: {err}")

---

196-209:Respect user CLI args instead of hard-coded defaults.

Useargs.image_format,args.num_frames, andargs.device to match single-turn behavior and user expectations.

-                inputs = default_multimodal_input_loader(+                inputs = default_multimodal_input_loader(                     tokenizer=llm.tokenizer,                     model_dir=llm._hf_model_dir,                     model_type=model_type,                     modality=args.modality,                     prompts=[cur_prompt],                     media=args.media,-                    image_data_format="pt",-                    num_frames=8,-                    device="cpu")+                    image_data_format=image_format,+                    num_frames=args.num_frames,+                    device=args.device)

---

239-246:Always clear CUDA cache using finally.

On exceptions, thecontinue skipsclear_cuda_cache(). Use afinally block.

-            except Exception as e:-                print(f"Error in turn {i+1}: {e}")-                import traceback-                traceback.print_exc()-                continue--            clear_cuda_cache()+            except Exception as e:+                print(f"Error in turn {i+1}: {e}")+                import traceback+                traceback.print_exc()+            finally:+                clear_cuda_cache()

---

236-238:Multi-turn context is not persisted — conversation state never grows.

You buildcur_prompt but don’t save it, so the next turn reloads the original prompt. Persist it.

-                # Add to cur_prompt-                cur_prompt = cur_prompt + response+                # Persist conversation context for next turn+                args.prompt[i] = f"{cur_prompt} assistant: {response}"

Alternative: maintain ahistory list per sample and concatenate before calling the loader for each turn.

tests/integration/defs/test_e2e.py (2)

2573-2584:Make the number of turns explicit to avoid silent truncation.

--conversation_turns defaults to 2; if more prompts are passed later, only two turns would run. Set it to the number of provided prompts for clarity and robustness.

         "--multiturn",+        "--conversation_turns",+        str(len(accuracy_inputs["image"]["prompt"])),         "--prompt",

---

2609-2620:Harden matching (case-insensitive), remove noisy prints, and fix E501 long line.

This reduces flakiness, cleans logs, and complies with line length rules.

-    for prompt_output, prompt_keywords in zip(-            parse_output(output), expected_keywords[model_name]["image"]):-        matches = [-            keyword in prompt_output.lower() for keyword in prompt_keywords-        ]-        obs_match_ratio = 1. * sum(matches) / len(matches)-        print("prompt_output:", prompt_output)-        print("prompt_keywords:", prompt_keywords)-        print("matches:", matches)-        print("obs_match_ratio:", obs_match_ratio)-        assert obs_match_ratio >= match_ratio, f"Incorrect output!\nGenerated \"{prompt_output}\"\nExpected keywords \"{prompt_keywords}\"\n Matched keywords: {matches}\n Observed match ratio {obs_match_ratio} below threshold {match_ratio}"+    for prompt_output, prompt_keywords in zip(outputs, expected):+        matches = [keyword.lower() in prompt_output.lower()+                   for keyword in prompt_keywords]+        obs_match_ratio = 1.0 * sum(matches) / len(matches)+        assert obs_match_ratio >= match_ratio, (+            f"Incorrect output!\nGenerated \"{prompt_output}\"\n"+            f"Expected keywords \"{prompt_keywords}\"\n"+            f"Matched keywords: {matches}\n"+            f"Observed match ratio {obs_match_ratio} below threshold {match_ratio}"+        )

tests/integration/defs/test_e2e.py (1)

2602-2602:Reduce CI log noise — drop raw output print and pre-parse instead.

Printing the entire raw output is noisy. Parse once and assert; include truncated raw text only in failure messages if needed.

-    output = llm_venv.run_cmd(cmd, caller=check_output)-    print("output:", output)+    output = llm_venv.run_cmd(cmd, caller=check_output)+    outputs = parse_output(output)+    expected = expected_keywords[model_name]["image"]+    assert len(outputs) == len(expected), (+        f"Expected {len(expected)} outputs, got {len(outputs)}. "+        f"Raw output (truncated): {output[:1000]}..."+    )

Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro

Learnt from: moraxuPR: NVIDIA/TensorRT-LLM#6303File: tests/integration/test_lists/qa/examples_test_list.txt:494-494Timestamp: 2025-07-28T17:06:08.621ZLearning: In TensorRT-LLM testing, it's common to have both CLI flow tests (test_cli_flow.py) and PyTorch API tests (test_llm_api_pytorch.py) for the same model. These serve different purposes: CLI flow tests validate the traditional command-line workflow, while PyTorch API tests validate the newer LLM API backend. Both are legitimate and should coexist.