tests/unittest/_torch/multi_gpu_modeling/test_llama3.py (1)

7-26:Gate and bound the integration test for portability and runtime.

Skip when insufficient GPUs and cap generation length to reduce CI time; similarity check should still hold.

+import pytest+import torch@@-def test_llama_3_3():+@pytest.mark.skipif(torch.cuda.device_count() < 4, reason="requires >=4 GPUs")+def test_llama_3_3():@@-    outputs = llm.generate(prompts)+    outputs = llm.generate(prompts, max_new_tokens=16)

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Learnt from: fredricz-20070104PR: NVIDIA/TensorRT-LLM#7645File: tests/integration/test_lists/qa/llm_function_core.txt:648-648Timestamp: 2025-09-09T09:40:45.658ZLearning: In TensorRT-LLM test lists, it's common and intentional for the same test to appear in multiple test list files when they serve different purposes (e.g., llm_function_core.txt for comprehensive core functionality testing and llm_function_core_sanity.txt for quick sanity checks). This duplication allows tests to be run in different testing contexts.

Learnt from: moraxuPR: NVIDIA/TensorRT-LLM#6303File: tests/integration/test_lists/qa/examples_test_list.txt:494-494Timestamp: 2025-07-28T17:06:08.621ZLearning: In TensorRT-LLM testing, it's common to have both CLI flow tests (test_cli_flow.py) and PyTorch API tests (test_llm_api_pytorch.py) for the same model. These serve different purposes: CLI flow tests validate the traditional command-line workflow, while PyTorch API tests validate the newer LLM API backend. Both are legitimate and should coexist.

Learnt from: pengbowang-nvPR: NVIDIA/TensorRT-LLM#7192File: tests/integration/test_lists/test-db/l0_dgx_b200.yml:56-72Timestamp: 2025-08-26T09:49:04.956ZLearning: In TensorRT-LLM test configuration files, the test scheduling system handles wildcard matching with special rules that prevent duplicate test execution even when the same tests appear in multiple yaml files with overlapping GPU wildcards (e.g., "*b200*" and "*gb200*").

Learnt from: galagamPR: NVIDIA/TensorRT-LLM#6487File: tests/unittest/_torch/auto_deploy/unit/singlegpu/test_ad_trtllm_bench.py:1-12Timestamp: 2025-08-06T13:58:07.506ZLearning: In TensorRT-LLM, test files (files under tests/ directories) do not require NVIDIA copyright headers, unlike production source code files. Test files typically start directly with imports, docstrings, or code.

tensorrt_llm/_torch/models/modeling_llama.py (1)

771-777:Same state-mutation issue on the Llama (non‑4) path

Replicate the fix here to keeppost_mlp_fusion_op immutable during forward.

Apply this diff:

-                # The next layernorm exists but it could be the last decoder layer.-                # Adjust the scale and fusion pattern.-                if self.next_attn is not None and (-                        self.is_nvfp4 or self.is_fp8_quant) and hasattr(-                            self.next_attn.qkv_proj, 'input_scale'):-                    scale = self.next_attn.qkv_proj.input_scale-                else:-                    self.post_mlp_fusion_op = AllReduceFusionOp.RESIDUAL_RMS_NORM-                    scale = None+                # The next layernorm exists but it could be the last decoder layer.+                # Adjust the scale and fusion pattern without mutating layer state.+                fusion_op = self.post_mlp_fusion_op+                scale = None+                if self.next_attn is not None and (self.is_nvfp4 or self.is_fp8_quant):+                    next_proj = getattr(self.next_attn, "qkv_proj", None)+                    scale = getattr(next_proj, "input_scale", None)+                    if scale is None:+                        fusion_op = AllReduceFusionOp.RESIDUAL_RMS_NORM@@-                all_reduce_output = self.all_reduce(+                all_reduce_output = self.all_reduce(                     hidden_states,                     all_reduce_params=AllReduceParams(-                        fusion_op=self.post_mlp_fusion_op,+                        fusion_op=fusion_op,                         residual=residual,                         norm_weight=self.next_layer_layernorm.weight,                         scale=scale,                         eps=self.next_layer_layernorm.variance_epsilon,                     ))

Also applies to: 779-787

tensorrt_llm/_torch/models/modeling_llama.py (1)

1-1:License header missing (pre‑existing)

Per repo guidelines, prepend the NVIDIA Apache‑2.0 header with the current year. Not blocking this PR; consider a follow‑up sweep.

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Learnt from: moraxuPR: NVIDIA/TensorRT-LLM#6303File: tests/integration/test_lists/qa/examples_test_list.txt:494-494Timestamp: 2025-07-28T17:06:08.621ZLearning: In TensorRT-LLM testing, it's common to have both CLI flow tests (test_cli_flow.py) and PyTorch API tests (test_llm_api_pytorch.py) for the same model. These serve different purposes: CLI flow tests validate the traditional command-line workflow, while PyTorch API tests validate the newer LLM API backend. Both are legitimate and should coexist.

Learnt from: fredricz-20070104PR: NVIDIA/TensorRT-LLM#7645File: tests/integration/test_lists/qa/llm_function_core.txt:648-648Timestamp: 2025-09-09T09:40:45.658ZLearning: In TensorRT-LLM test lists, it's common and intentional for the same test to appear in multiple test list files when they serve different purposes (e.g., llm_function_core.txt for comprehensive core functionality testing and llm_function_core_sanity.txt for quick sanity checks). This duplication allows tests to be run in different testing contexts.

Learnt from: pengbowang-nvPR: NVIDIA/TensorRT-LLM#7192File: tests/integration/test_lists/test-db/l0_dgx_b200.yml:56-72Timestamp: 2025-08-26T09:49:04.956ZLearning: In TensorRT-LLM test configuration files, the test scheduling system handles wildcard matching with special rules that prevent duplicate test execution even when the same tests appear in multiple yaml files with overlapping GPU wildcards (e.g., "*b200*" and "*gb200*").

tests/unittest/_torch/multi_gpu_modeling/test_llama3.py (1)

7-7:On combining with existing LLM API tests

Echoing the earlier thread: consolidating withtest_llm_api_pytorch’s Llama‑3.3 suite is fine to do in a follow‑up, as agreed.

If you want to confirm current coverage overlap quickly:

#!/bin/bash# Find any existing Llama-3.3 tests in PyTorch API suitesrg -nP'Llama[-_ ]?3\.3|Llama3_3' -g'tests/**/test_llm_api_pytorch*.py' -C2

tests/unittest/_torch/multi_gpu_modeling/test_llama3.py (2)

15-17:Prompt choice risk: reduce time‑sensitive/non‑deterministic content

“The president of the United States is” can drift. Consider more stable prompts (math/facts) or keep but rely on robust assertions below.

---

19-26:Determinism and robustness: constrain decoding and assert on signals, not long text

Constrain decoding for reproducibility and speed; prefer short, robust assertions over fuzzy long‑text similarity.

Apply this diff:

-    outputs = llm.generate(prompts)+    outputs = llm.generate(+        prompts,+        max_new_tokens=24,+        temperature=0.0,+        top_k=1,+        random_seed=1234,+    )-    expected_outputs = [-        " a city of romance, art, fashion, and cuisine. Paris, also known as the City of Light, is a must-visit destination for anyone interested in",-        " the head of state and head of government of the United States. The president is also the commander-in-chief of the armed forces. The president is elected by the",-    ]-    for i, output in enumerate(outputs):-        assert similar(output.outputs[0].text, expected_outputs[i])+    for i, res in enumerate(outputs):+        text = res.outputs[0].text.lower()+        if i == 0:+            assert "paris" in text+        else:+            assert "president" in text and "united states" in text

Additionally, if you adopt the above, drop the now-unusedsimilar import.

-from utils.util import similar

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Learnt from: moraxuPR: NVIDIA/TensorRT-LLM#6303File: tests/integration/test_lists/qa/examples_test_list.txt:494-494Timestamp: 2025-07-28T17:06:08.621ZLearning: In TensorRT-LLM testing, it's common to have both CLI flow tests (test_cli_flow.py) and PyTorch API tests (test_llm_api_pytorch.py) for the same model. These serve different purposes: CLI flow tests validate the traditional command-line workflow, while PyTorch API tests validate the newer LLM API backend. Both are legitimate and should coexist.

Learnt from: fredricz-20070104PR: NVIDIA/TensorRT-LLM#7645File: tests/integration/test_lists/qa/llm_function_core.txt:648-648Timestamp: 2025-09-09T09:40:45.658ZLearning: In TensorRT-LLM test lists, it's common and intentional for the same test to appear in multiple test list files when they serve different purposes (e.g., llm_function_core.txt for comprehensive core functionality testing and llm_function_core_sanity.txt for quick sanity checks). This duplication allows tests to be run in different testing contexts.

Learnt from: galagamPR: NVIDIA/TensorRT-LLM#6487File: tests/unittest/_torch/auto_deploy/unit/singlegpu/test_ad_trtllm_bench.py:1-12Timestamp: 2025-08-06T13:58:07.506ZLearning: In TensorRT-LLM, test files (files under tests/ directories) do not require NVIDIA copyright headers, unlike production source code files. Test files typically start directly with imports, docstrings, or code.

tests/unittest/_torch/multi_gpu_modeling/test_llama3.py (3)

1-5:Lazy-load TRT‑LLM; add pytest/torch; avoid heavy top‑level import.

Prevents collection-time failures and aligns with import style guidance. (Previously raised.)

-from utils.llm_data import llm_models_root-from utils.util import similar--from tensorrt_llm import LLM+import pytest+import torch+from utils.llm_data import llm_models_root

---

7-14:Guard for TRT‑LLM availability, GPUs, and model dir; then import and build.

Skip cleanly if environment cannot satisfy TP=2, PP=2 (needs 4 GPUs). (Previously raised.)

 def test_llama_3_3():     model_dir = llm_models_root(     ) / "llama-3.3-models" / "Llama-3.3-70B-Instruct-FP8"     tp = 2     pp = 2-    llm = LLM(model_dir, tensor_parallel_size=tp, pipeline_parallel_size=pp)+    try:+        import tensorrt_llm as trtllm+    except Exception as e:+        pytest.skip(f"tensorrt-llm unavailable: {e}")+    required = tp * pp+    if not torch.cuda.is_available() or torch.cuda.device_count() < required:+        found = torch.cuda.device_count() if torch.cuda.is_available() else 0+        pytest.skip(f"requires >= {required} GPUs; found {found}")+    if not model_dir.exists():+        pytest.skip(f"model dir not found: {model_dir}")++    llm = trtllm.LLM(+        model_dir,+        tensor_parallel_size=tp,+        pipeline_parallel_size=pp,+    )

---

1-26:Move TRT-LLM import into the test and add GPU/model checks.

Top-levelfrom tensorrt_llm import LLM found at tests/unittest/_torch/multi_gpu_modeling/test_llama3.py:4; no torch.cuda/pytest skip or model_dir.exists() check. Change required:

• Move the tensorrt_llm import into test_llama_3_3() (lazy import) so collection doesn't trigger heavy runtime.
• At the start of the test, perform runtime guards and skip when unmet (e.g., compute required_gpus = tp * pp; if not torch.cuda.is_available() or torch.cuda.device_count() < required_gpus: pytest.skip(...); if not model_dir.exists(): pytest.skip(...)).

tensorrt_llm/_torch/models/modeling_llama.py (1)

561-565:Add brief inline context for hasattr guard.

Note that next_attn.qkv_proj can be a dummy Linear (skip_forward) on last PP rank so FP8QDQ monkey-patch doesn’t attach input_scale.

tests/unittest/_torch/multi_gpu_modeling/test_llama3.py (2)

21-26:Deflake assertions: avoid content that shifts over time; assert non‑empty outputs.

Use stable checks to reduce CI flakes in release pipeline.

-    expected_outputs = [-        " a city of romance, art, fashion, and cuisine. Paris, also known as the City of Light, is a must-visit destination for anyone interested in",-        " the head of state and head of government of the United States. The president is also the commander-in-chief of the armed forces. The president is elected by the",-    ]-    for i, output in enumerate(outputs):-        assert similar(output.outputs[0].text, expected_outputs[i])+    assert len(outputs) == len(prompts)+    for out in outputs:+        text = out.outputs[0].text+        assert isinstance(text, str) and len(text) > 0

---

7-26:Optional: keep a minimal, deterministic check.

If you prefer a content check, change the second prompt to a stable fact (e.g., math) and assert a short prefix.

-    prompts = [-        "The capital of France is",-        "The president of the United States is",-    ]+    prompts = [+        "The capital of France is",+        "The square root of 144 is",+    ]@@-    outputs = llm.generate(prompts)+    outputs = llm.generate(prompts)@@-    expected_outputs = [-        " a city of romance, art, fashion, and cuisine. Paris, also known as the City of Light, is a must-visit destination for anyone interested in",-        " the head of state and head of government of the United States. The president is also the commander-in-chief of the armed forces. The president is elected by the",-    ]-    for i, output in enumerate(outputs):-        assert similar(output.outputs[0].text, expected_outputs[i])+    assert outputs[0].outputs[0].text.lstrip().startswith("Paris")+    assert outputs[1].outputs[0].text.strip().startswith("12")

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Learnt from: moraxuPR: NVIDIA/TensorRT-LLM#6303File: tests/integration/test_lists/qa/examples_test_list.txt:494-494Timestamp: 2025-07-28T17:06:08.621ZLearning: In TensorRT-LLM testing, it's common to have both CLI flow tests (test_cli_flow.py) and PyTorch API tests (test_llm_api_pytorch.py) for the same model. These serve different purposes: CLI flow tests validate the traditional command-line workflow, while PyTorch API tests validate the newer LLM API backend. Both are legitimate and should coexist.

Learnt from: fredricz-20070104PR: NVIDIA/TensorRT-LLM#7645File: tests/integration/test_lists/qa/llm_function_core.txt:648-648Timestamp: 2025-09-09T09:40:45.658ZLearning: In TensorRT-LLM test lists, it's common and intentional for the same test to appear in multiple test list files when they serve different purposes (e.g., llm_function_core.txt for comprehensive core functionality testing and llm_function_core_sanity.txt for quick sanity checks). This duplication allows tests to be run in different testing contexts.

Learnt from: galagamPR: NVIDIA/TensorRT-LLM#6487File: tests/unittest/_torch/auto_deploy/unit/singlegpu/test_ad_trtllm_bench.py:1-12Timestamp: 2025-08-06T13:58:07.506ZLearning: In TensorRT-LLM, test files (files under tests/ directories) do not require NVIDIA copyright headers, unlike production source code files. Test files typically start directly with imports, docstrings, or code.

Learnt from: EmmaQiaoChPR: NVIDIA/TensorRT-LLM#7370File: tests/unittest/trt/model_api/test_model_quantization.py:24-27Timestamp: 2025-08-29T14:07:45.863ZLearning: In TensorRT-LLM's CI infrastructure, pytest skip markers (pytest.mark.skip) are properly honored even when test files have __main__ blocks that call test functions directly. The testing system correctly skips tests without requiring modifications to the __main__ block execution pattern.

Learnt from: pengbowang-nvPR: NVIDIA/TensorRT-LLM#7192File: tests/integration/test_lists/test-db/l0_dgx_b200.yml:56-72Timestamp: 2025-08-26T09:49:04.956ZLearning: In TensorRT-LLM test configuration files, the test scheduling system handles wildcard matching with special rules that prevent duplicate test execution even when the same tests appear in multiple yaml files with overlapping GPU wildcards (e.g., "*b200*" and "*gb200*").

Learnt from: achartierPR: NVIDIA/TensorRT-LLM#6763File: tests/integration/defs/triton_server/conftest.py:16-22Timestamp: 2025-08-11T20:09:24.389ZLearning: In the TensorRT-LLM test infrastructure, the team prefers simple, direct solutions (like hard-coding directory traversal counts) over more complex but robust approaches when dealing with stable directory structures. They accept the maintenance cost of updating tests if the layout changes.

tests/unittest/_torch/multi_gpu_modeling/test_llama3.py (3)

7-7:Follow‑up: potential consolidation with test_llm_api_pytorch

Per hyukn’s note, consider merging/parametrizing with TestLlama3_3_70BInstruct later. Fine to defer to a subsequent PR.

---

1-5:Lazy‑load TRT‑LLM; add pytest/torch; avoid top‑level heavy import

Importing LLM at module import time can explode before we can skip; also prefer module‑namespace import. Move TRT‑LLM import inside the test and add pytest/torch imports here.

-from utils.llm_data import llm_models_root-from utils.util import similar--from tensorrt_llm import LLM+import pytest+import torch+from utils.llm_data import llm_models_root+from utils.util import similar

---

7-13:Guard for TRT‑LLM availability, GPUs, and model dir; use module namespace

Skip cleanly when deps/resources are missing; import tensorrt_llm lazily; and use trtllm.LLM.

 def test_llama_3_3():     model_dir = llm_models_root(     ) / "llama-3.3-models" / "Llama-3.3-70B-Instruct-FP8"     tp = 2     pp = 2-    llm = LLM(model_dir, tensor_parallel_size=tp, pipeline_parallel_size=pp)+    # Lazy import so we can skip instead of erroring during collection+    try:+        import tensorrt_llm as trtllm+    except Exception as e:+        pytest.skip(f"tensorrt-llm unavailable: {e}")++    required = tp * pp+    if not torch.cuda.is_available() or torch.cuda.device_count() < required:+        found = torch.cuda.device_count() if torch.cuda.is_available() else 0+        pytest.skip(f"requires >= {required} GPUs; found {found}")+    if not model_dir.exists():+        pytest.skip(f"model dir not found: {model_dir}")++    llm = trtllm.LLM(model_dir, tensor_parallel_size=tp, pipeline_parallel_size=pp)

tests/unittest/_torch/multi_gpu_modeling/test_llama3.py (4)

14-17:Use time‑insensitive prompts to avoid fragile assertions

“The president of the United States is” is time‑variant. Prefer stable facts.

-    prompts = [-        "The capital of France is",-        "The president of the United States is",-    ]+    prompts = [+        "The capital of France is",+        "The capital of Japan is",+    ]

---

19-19:Make decoding deterministic and cheaper

Set a small max token budget and disable sampling (or temperature=0) to stabilize CI and reduce memory.

-    outputs = llm.generate(prompts)+    # If API differs, use the repo's SamplingParams equivalent with temperature=0 and max_new_tokens≈16+    outputs = llm.generate(prompts, max_new_tokens=16, temperature=0.0)

If generate() doesn’t accept these kwargs, switch to your SamplingParams/GenerationConfig and pass it in. Please confirm the exact parameter names used in this repo.

---

21-26:Avoid brittle full‑text matches; assert key facts with diagnostics

Long expected blobs are unstable across model/builds. Check for essential substrings and add a length check plus helpful failure messages.

-    expected_outputs = [-        " a city of romance, art, fashion, and cuisine. Paris, also known as the City of Light, is a must-visit destination for anyone interested in",-        " the head of state and head of government of the United States. The president is also the commander-in-chief of the armed forces. The president is elected by the",-    ]-    for i, output in enumerate(outputs):-        assert similar(output.outputs[0].text, expected_outputs[i])+    expected_contains = ["Paris", "Tokyo"]+    assert len(outputs) == len(prompts)+    for i, out in enumerate(outputs):+        text = out.outputs[0].text+        assert expected_contains[i].lower() in text.lower(), (+            f"Output[{i}] missing '{expected_contains[i]}'. Got: {text[:160]!r}"+        )

---

26-26:Release resources explicitly (optional)

Free the session aggressively to reduce GPU memory pressure on CI runners.

-        assert similar(output.outputs[0].text, expected_outputs[i])+        ...+    # Optional explicit cleanup+    del llm

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Learnt from: moraxuPR: NVIDIA/TensorRT-LLM#6303File: tests/integration/test_lists/qa/examples_test_list.txt:494-494Timestamp: 2025-07-28T17:06:08.621ZLearning: In TensorRT-LLM testing, it's common to have both CLI flow tests (test_cli_flow.py) and PyTorch API tests (test_llm_api_pytorch.py) for the same model. These serve different purposes: CLI flow tests validate the traditional command-line workflow, while PyTorch API tests validate the newer LLM API backend. Both are legitimate and should coexist.

Learnt from: fredricz-20070104PR: NVIDIA/TensorRT-LLM#7645File: tests/integration/test_lists/qa/llm_function_core.txt:648-648Timestamp: 2025-09-09T09:40:45.658ZLearning: In TensorRT-LLM test lists, it's common and intentional for the same test to appear in multiple test list files when they serve different purposes (e.g., llm_function_core.txt for comprehensive core functionality testing and llm_function_core_sanity.txt for quick sanity checks). This duplication allows tests to be run in different testing contexts.

Learnt from: galagamPR: NVIDIA/TensorRT-LLM#6487File: tests/unittest/_torch/auto_deploy/unit/singlegpu/test_ad_trtllm_bench.py:1-12Timestamp: 2025-08-06T13:58:07.506ZLearning: In TensorRT-LLM, test files (files under tests/ directories) do not require NVIDIA copyright headers, unlike production source code files. Test files typically start directly with imports, docstrings, or code.

Learnt from: EmmaQiaoChPR: NVIDIA/TensorRT-LLM#7370File: tests/unittest/trt/model_api/test_model_quantization.py:24-27Timestamp: 2025-08-29T14:07:45.863ZLearning: In TensorRT-LLM's CI infrastructure, pytest skip markers (pytest.mark.skip) are properly honored even when test files have __main__ blocks that call test functions directly. The testing system correctly skips tests without requiring modifications to the __main__ block execution pattern.

Learnt from: pengbowang-nvPR: NVIDIA/TensorRT-LLM#7192File: tests/integration/test_lists/test-db/l0_dgx_b200.yml:56-72Timestamp: 2025-08-26T09:49:04.956ZLearning: In TensorRT-LLM test configuration files, the test scheduling system handles wildcard matching with special rules that prevent duplicate test execution even when the same tests appear in multiple yaml files with overlapping GPU wildcards (e.g., "*b200*" and "*gb200*").

Learnt from: achartierPR: NVIDIA/TensorRT-LLM#6763File: tests/integration/defs/triton_server/conftest.py:16-22Timestamp: 2025-08-11T20:09:24.389ZLearning: In the TensorRT-LLM test infrastructure, the team prefers simple, direct solutions (like hard-coding directory traversal counts) over more complex but robust approaches when dealing with stable directory structures. They accept the maintenance cost of updating tests if the layout changes.