Movatterモバイル変換

[0]ホーム
URL:

Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

 Sign up
Appearance settings

[None][chore] Make use_low_precision_moe_combine as a llm arg#7598

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Merged

Conversation

@zongfeijing
Copy link
Collaborator

@zongfeijingzongfeijing commentedSep 8, 2025•
edited
Loading

Summary by CodeRabbit

  • New Features
    • Added an optional --low_precision_combine flag to the advanced quickstart and configuration to enable low-precision combine for Mixture-of-Experts. This can improve performance on setups supporting NVFP4 quantization. Off by default.
    • The setting is applied end-to-end, from CLI/API config through model loading, ensuring consistent behavior across backends.
    • No changes in default behavior when the flag is not set.

Description

Test Coverage

PR Checklist

Please review the following before submitting your PR:

  • PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

  • PR FollowsTRT-LLM CODING GUIDELINES to the best of your knowledge.

  • Test cases are provided for new code paths (seetest instructions)

  • Any new dependencies have been scanned for license and vulnerabilities

  • CODEOWNERS updated if ownership changes

  • Documentation updated as needed

  • The reviewers assigned automatically/manually are appropriate for the PR.

  • Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

/bot [-h] ['run', 'kill', 'skip', 'reuse-pipeline'] ...

Provide a user friendly way for developers to interact with a Jenkins server.

Run/bot [-h|--help] to print this help message.

See details below for each supported subcommand.

run [--reuse-test (optional)pipeline-id --disable-fail-fast --skip-test --stage-list "A10-PyTorch-1, xxx" --gpu-type "A30, H100_PCIe" --test-backend "pytorch, cpp" --add-multi-gpu-test --only-multi-gpu-test --disable-multi-gpu-test --post-merge --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx" --detailed-log --debug(experimental)]

Launch build/test pipelines. All previously running jobs will be killed.

--reuse-test (optional)pipeline-id(OPTIONAL) : Allow the new pipeline to reuse build artifacts and skip successful test stages from a specified pipeline or the last pipeline if no pipeline-id is indicated. If the Git commit ID has changed, this option will be always ignored. The DEFAULT behavior of the bot is to reuse build artifacts and successful test results from the last pipeline.

--disable-reuse-test(OPTIONAL) : Explicitly prevent the pipeline from reusing build artifacts and skipping successful test stages from a previous pipeline. Ensure that all builds and tests are run regardless of previous successes.

--disable-fail-fast(OPTIONAL) : Disable fail fast on build/tests/infra failures.

--skip-test(OPTIONAL) : Skip all test stages, but still run build stages, package stages and sanity check stages. Note: DoesNOT update GitHub check status.

--stage-list "A10-PyTorch-1, xxx"(OPTIONAL) : Only run the specified test stages. Examples: "A10-PyTorch-1, xxx". Note: DoesNOT update GitHub check status.

--gpu-type "A30, H100_PCIe"(OPTIONAL) : Only run the test stages on the specified GPU types. Examples: "A30, H100_PCIe". Note: DoesNOT update GitHub check status.

--test-backend "pytorch, cpp"(OPTIONAL) : Skip test stages which don't match the specified backends. Only support [pytorch, cpp, tensorrt, triton]. Examples: "pytorch, cpp" (does not run test stages with tensorrt or triton backend). Note: DoesNOT update GitHub pipeline status.

--only-multi-gpu-test(OPTIONAL) : Only run the multi-GPU tests. Note: DoesNOT update GitHub check status.

--disable-multi-gpu-test(OPTIONAL) : Disable the multi-GPU tests. Note: DoesNOT update GitHub check status.

--add-multi-gpu-test(OPTIONAL) : Force run the multi-GPU tests in addition to running L0 pre-merge pipeline.

--post-merge(OPTIONAL) : Run the L0 post-merge pipeline instead of the ordinary L0 pre-merge pipeline.

--extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx"(OPTIONAL) : Run the ordinary L0 pre-merge pipeline and specified test stages. Examples: --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx".

--detailed-log(OPTIONAL) : Enable flushing out all logs to the Jenkins console. This will significantly increase the log volume and may slow down the job.

--debug(OPTIONAL) :Experimental feature. Enable access to the CI container for debugging purpose. Note: Specify exactly one stage in thestage-list parameter to access the appropriate container environment. Note: DoesNOT update GitHub check status.

For guidance on mapping tests to stage names, seedocs/source/reference/ci-overview.md
and thescripts/test_to_stage_mapping.py helper.

kill

kill

Kill all running builds associated with pull request.

skip

skip --comment COMMENT

Skip testing for latest commit on pull request.--comment "Reason for skipping build/test" is required. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.

reuse-pipeline

reuse-pipeline

Reuse a previous pipeline to validate current commit. This action will also kill all currently running builds associated with the pull request. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.

@zongfeijing
Copy link
CollaboratorAuthor

/bot run

@coderabbitai
Copy link
Contributor

coderabbitaibot commentedSep 8, 2025•
edited
Loading

Caution

Review failed

An error occurred during the review process. Please try again later.

✨ Finishing Touches
🧪 Generate unit tests
  • Create PR with unit tests
  • Post copyable unit tests in a comment

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment@coderabbitai help to get the list of available commands and usage tips.

Copy link
Contributor

@coderabbitaicoderabbitaibot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Actionable comments posted: 0

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (6)
tensorrt_llm/_torch/model_config.py (1)

1-1:Add required NVIDIA Apache-2.0 header.

Per repo guidelines, prepend the 2025 header.

Apply:

+# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#     http://www.apache.org/licenses/LICENSE-2.0+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.+# SPDX-License-Identifier: Apache-2.0
tensorrt_llm/_torch/pyexecutor/config.py (1)

1-1:Add required NVIDIA Apache-2.0 header.

Apply:

+# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#     http://www.apache.org/licenses/LICENSE-2.0+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.+# SPDX-License-Identifier: Apache-2.0
tensorrt_llm/_torch/modules/fused_moe/fused_moe_wide_ep.py (1)

1-1:Add required NVIDIA Apache-2.0 header.

Apply:

+# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#     http://www.apache.org/licenses/LICENSE-2.0+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.+# SPDX-License-Identifier: Apache-2.0
tensorrt_llm/_torch/pyexecutor/model_engine.py (1)

1-1:Add required NVIDIA Apache-2.0 header.

Apply:

+# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#     http://www.apache.org/licenses/LICENSE-2.0+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.+# SPDX-License-Identifier: Apache-2.0
examples/llm-api/quickstart_advanced.py (1)

1-1:Add required NVIDIA Apache-2.0 header.

Apply:

+# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#     http://www.apache.org/licenses/LICENSE-2.0+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.+# SPDX-License-Identifier: Apache-2.0
tensorrt_llm/llmapi/llm_args.py (1)

1-1:Add required NVIDIA Apache-2.0 header.

Apply:

+# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#     http://www.apache.org/licenses/LICENSE-2.0+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.+# SPDX-License-Identifier: Apache-2.0
🧹 Nitpick comments (3)
tensorrt_llm/_torch/modules/fused_moe/fused_moe_wide_ep.py (2)

192-193:Warn when the flag is ignored (non-NVFP4).

Currently silently no-ops if NVFP4 isn’t present; surface a one-time info to aid debugging.

Apply:

-            self.use_low_precision_combine = model_config.moe_low_precision_combine and qm.has_nvfp4(-            )+            self.use_low_precision_combine = (+                model_config.moe_low_precision_combine and qm.has_nvfp4()+            )+            if model_config.moe_low_precision_combine and not qm.has_nvfp4():+                logger.info_once(+                    "moe_low_precision_combine is set but ignored because NVFP4 quantization is not enabled.",+                    key="moe_lp_combine_ignored",+                )

58-75:Avoid dataclass instance as default argument.

model_config=ModelConfig() creates a shared instance across layers.

Apply:

-        model_config: ModelConfig = ModelConfig(),+        model_config: ModelConfig = None,@@-        super().__init__(+        model_config = model_config or ModelConfig()+        super().__init__(
tensorrt_llm/llmapi/llm_args.py (1)

194-199:MoeConfig knob: LGTM; consider marking status.

Marking as beta clarifies maturity in generated docs.

Apply:

-    low_precision_combine: bool = Field(-        default=False,-        description=-        "Use low precision combine in MoE operations (only for NVFP4 quantization). When enabled, uses lower precision for combining expert outputs to improve performance."-    )+    low_precision_combine: bool = Field(+        default=False,+        description="Use low precision combine in MoE operations (only for NVFP4 quantization). When enabled, uses lower precision for combining expert outputs to improve performance.",+        status="beta",+    )
📜 Review details

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between9938f4f and73045d8.

📒 Files selected for processing (6)
  • examples/llm-api/quickstart_advanced.py (2 hunks)
  • tensorrt_llm/_torch/model_config.py (1 hunks)
  • tensorrt_llm/_torch/modules/fused_moe/fused_moe_wide_ep.py (1 hunks)
  • tensorrt_llm/_torch/pyexecutor/config.py (1 hunks)
  • tensorrt_llm/_torch/pyexecutor/model_engine.py (1 hunks)
  • tensorrt_llm/llmapi/llm_args.py (2 hunks)
🧰 Additional context used
📓 Path-based instructions (3)
**/*.{h,hpp,hh,hxx,cpp,cxx,cc,cu,cuh,py}

📄 CodeRabbit inference engine (CODING_GUIDELINES.md)

Use only spaces, no tabs; indent with 4 spaces.

Files:

  • tensorrt_llm/_torch/model_config.py
  • tensorrt_llm/_torch/pyexecutor/config.py
  • tensorrt_llm/_torch/modules/fused_moe/fused_moe_wide_ep.py
  • tensorrt_llm/_torch/pyexecutor/model_engine.py
  • tensorrt_llm/llmapi/llm_args.py
  • examples/llm-api/quickstart_advanced.py
**/*.py

📄 CodeRabbit inference engine (CODING_GUIDELINES.md)

**/*.py: Python code must target Python 3.8+.
Indent Python code with 4 spaces; do not use tabs.
Maintain module namespace when importing; prefer 'from package.subpackage import foo' then 'foo.SomeClass()' instead of importing the class directly.
Python filenames should be snake_case (e.g., some_file.py).
Python classes use PascalCase names.
Functions and methods use snake_case names.
Local variables use snake_case; prefix 'k' for variables that start with a number (e.g., k_99th_percentile).
Global variables use upper SNAKE_CASE prefixed with 'G' (e.g., G_MY_GLOBAL).
Constants use upper SNAKE_CASE (e.g., MY_CONSTANT).
Avoid shadowing variables from an outer scope.
Initialize all externally visible members of a class in the constructor.
Prefer docstrings for interfaces that may be used outside a file; comments for in-function or file-local interfaces.
Use Google-style docstrings for classes and functions (Sphinx-parsable).
Document attributes and variables inline so they render under the class/function docstring.
Avoid reflection when a simpler, explicit approach suffices (e.g., avoid dict(**locals()) patterns).
In try/except, catch the most specific exceptions possible.
For duck-typing try/except, keep the try body minimal and use else for the main logic.

Files:

  • tensorrt_llm/_torch/model_config.py
  • tensorrt_llm/_torch/pyexecutor/config.py
  • tensorrt_llm/_torch/modules/fused_moe/fused_moe_wide_ep.py
  • tensorrt_llm/_torch/pyexecutor/model_engine.py
  • tensorrt_llm/llmapi/llm_args.py
  • examples/llm-api/quickstart_advanced.py
**/*.{cpp,cxx,cc,h,hpp,hh,hxx,cu,cuh,py}

📄 CodeRabbit inference engine (CODING_GUIDELINES.md)

Prepend the NVIDIA Apache-2.0 copyright header with current year to the top of all source files (e.g., .cpp, .h, .cu, .py).

Files:

  • tensorrt_llm/_torch/model_config.py
  • tensorrt_llm/_torch/pyexecutor/config.py
  • tensorrt_llm/_torch/modules/fused_moe/fused_moe_wide_ep.py
  • tensorrt_llm/_torch/pyexecutor/model_engine.py
  • tensorrt_llm/llmapi/llm_args.py
  • examples/llm-api/quickstart_advanced.py
🧬 Code graph analysis (3)
tensorrt_llm/_torch/modules/fused_moe/fused_moe_wide_ep.py (2)
tensorrt_llm/_torch/modules/linear.py (1)
  • has_nvfp4 (1610-1613)
tensorrt_llm/_torch/modules/fused_moe/interface.py (1)
  • has_nvfp4 (133-136)
tensorrt_llm/llmapi/llm_args.py (1)
tensorrt_llm/builder.py (1)
  • default (50-58)
examples/llm-api/quickstart_advanced.py (1)
tensorrt_llm/llmapi/llm_args.py (1)
  • MoeConfig (168-202)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
  • GitHub Check: Pre-commit Check
🔇 Additional comments (6)
tensorrt_llm/_torch/model_config.py (1)

120-121:Add MoE flag: LGTM.

Name and default make sense; kw-only placement is consistent. No functional concerns.

tensorrt_llm/_torch/pyexecutor/config.py (1)

63-63:Wire-through config: LGTM.

Field naming matches ModelConfig and LLM args.

tensorrt_llm/_torch/pyexecutor/model_engine.py (1)

310-311:LGTM
load_config acceptskwargs andModelConfig definesmoe_low_precision_combine, so the flag is forwarded correctly.

examples/llm-api/quickstart_advanced.py (2)

76-81:CLI flag: LGTM.

Help text clearly scopes NVFP4-only behavior.

237-238:Wiring to MoeConfig: LGTM.

Correct field name and default behavior.

tensorrt_llm/llmapi/llm_args.py (1)

2595-2596:Propagate to PyTorchConfig: LGTM.

Field name aligns with PyTorchConfig.

@tensorrt-cicd
Copy link
Collaborator

PR_Github #17971 [ run ] triggered by Bot

@tensorrt-cicd
Copy link
Collaborator

PR_Github #17971 [ run ] completed with stateSUCCESS
/LLM/release-1.1.0rc2/L0_MergeRequest_PR pipeline #91 completed with status: 'SUCCESS'
Pipeline passed with automatic retried tests. Check thererun report for details.

@zongfeijing
Copy link
CollaboratorAuthor

/bot run --disable-fail-fast

@tensorrt-cicd
Copy link
Collaborator

PR_Github #18028 [ run ] triggered by Bot

@kaiyuxkaiyuxenabled auto-merge (squash)September 8, 2025 16:06
@kaiyuxkaiyux changed the title[None][chore] Make low_precision_combine as a llm arg[None][chore] Make use_low_precision_moe_combine as a llm argSep 8, 2025
Copy link
Collaborator

@QiJuneQiJune left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

LGTM

@tensorrt-cicd
Copy link
Collaborator

PR_Github #18028 [ run ] completed with stateSUCCESS
/LLM/release-1.1.0rc2/L0_MergeRequest_PR pipeline #97 completed with status: 'FAILURE'

@kaiyux
Copy link
Member

/bot run --disable-fail-fast

@tensorrt-cicd
Copy link
Collaborator

PR_Github #18078 [ run ] triggered by Bot

@tensorrt-cicd
Copy link
Collaborator

PR_Github #18078 [ run ] completed with stateFAILURE
/LLM/release-1.1.0rc2/L0_MergeRequest_PR pipeline #100 completed with status: 'FAILURE'

Signed-off-by: Zongfei Jing <20381269+zongfeijing@users.noreply.github.com>
Signed-off-by: Zongfei Jing <20381269+zongfeijing@users.noreply.github.com>
@kaiyux
Copy link
Member

/bot run --stage-list "DGX_B200-8_GPUs-PyTorch-1"

@kaiyux
Copy link
Member

All stages have been passed except "DGX_B200-8_GPUs-PyTorch-1" due to an infra issue.

@tensorrt-cicd
Copy link
Collaborator

PR_Github #18084 [ run ] triggered by Bot

@kaiyux
Copy link
Member

/bot run --stage-list "DGX_B200-8_GPUs-PyTorch-1" --only-multi-gpu-test

@kaiyux
Copy link
Member

/bot run --stage-list "DGX_B200-8_GPUs-PyTorch-1"

@tensorrt-cicd
Copy link
Collaborator

PR_Github #18085 [ run ] triggered by Bot

@tensorrt-cicd
Copy link
Collaborator

PR_Github #18086 [ run ] triggered by Bot

@tensorrt-cicd
Copy link
Collaborator

PR_Github #18086 [ run ] completed with stateABORTED

@tensorrt-cicd
Copy link
Collaborator

PR_Github #18084 [ run ] completed with stateABORTED

@tensorrt-cicd
Copy link
Collaborator

PR_Github #18085 [ run ] completed with stateSUCCESS
/LLM/release-1.1.0rc2/L0_MergeRequest_PR pipeline #102(Partly Tested) completed with status: 'SUCCESS'

@kaiyux
Copy link
Member

/bot skip --comment "pipeline passed"

@tensorrt-cicd
Copy link
Collaborator

PR_Github #18094 [ skip ] triggered by Bot

@tensorrt-cicd
Copy link
Collaborator

PR_Github #18094 [ skip ] completed with stateSUCCESS
Skipping testing for commitcf86c0b

@kaiyuxkaiyux merged commit75745c7 intoNVIDIA:release/1.1.0rc2Sep 8, 2025
5 checks passed
zongfeijing added a commit to zongfeijing/TensorRT-LLM that referenced this pull requestSep 22, 2025
Signed-off-by: Zongfei Jing <20381269+zongfeijing@users.noreply.github.com>
zongfeijing added a commit to zongfeijing/TensorRT-LLM that referenced this pull requestSep 23, 2025
Signed-off-by: Zongfei Jing <20381269+zongfeijing@users.noreply.github.com>
zongfeijing added a commit that referenced this pull requestSep 29, 2025
…a llm arg (#7898)Signed-off-by: Zongfei Jing <20381269+zongfeijing@users.noreply.github.com>
dominicshanshan pushed a commit to dominicshanshan/TensorRT-LLM that referenced this pull requestNov 1, 2025
…ne as a llm arg (NVIDIA#7898)Signed-off-by: Zongfei Jing <20381269+zongfeijing@users.noreply.github.com>
dominicshanshan pushed a commit to dominicshanshan/TensorRT-LLM that referenced this pull requestNov 3, 2025
…ne as a llm arg (NVIDIA#7898)Signed-off-by: Zongfei Jing <20381269+zongfeijing@users.noreply.github.com>
dominicshanshan pushed a commit to dominicshanshan/TensorRT-LLM that referenced this pull requestNov 3, 2025
…ne as a llm arg (NVIDIA#7898)Signed-off-by: Zongfei Jing <20381269+zongfeijing@users.noreply.github.com>
dominicshanshan pushed a commit to dominicshanshan/TensorRT-LLM that referenced this pull requestNov 3, 2025
…ne as a llm arg (NVIDIA#7898)Signed-off-by: Zongfei Jing <20381269+zongfeijing@users.noreply.github.com>
Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment

Reviewers

@SuperjomnSuperjomnSuperjomn left review comments

@coderabbitaicoderabbitai[bot]coderabbitai[bot] left review comments

@litaotjulitaotjulitaotju approved these changes

@QiJuneQiJuneQiJune approved these changes

@kaiyuxkaiyuxkaiyux approved these changes

@yilin-voidyilin-voidAwaiting requested review from yilin-void

@HuiGao-NVHuiGao-NVAwaiting requested review from HuiGao-NVHuiGao-NV is a code owner automatically assigned from NVIDIA/trt-llm-torch-modules

@laikhtewarilaikhtewariAwaiting requested review from laikhtewarilaikhtewari is a code owner automatically assigned from NVIDIA/trt-llm-doc-owners

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@zongfeijing@tensorrt-cicd@kaiyux@Superjomn@litaotju@QiJune
[8]ページ先頭
©2009-2025 Movatter.jp