Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

[None][fix] Fix the aux_stream in Llama4MinLatencyFusedMoE#9035

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Merged

Conversation

@jinyangyuan-nvidia
Copy link
Collaborator

@jinyangyuan-nvidiajinyangyuan-nvidia commentedNov 10, 2025
edited
Loading

Summary by CodeRabbit

  • Refactor
    • Improved auxiliary stream configuration: replaced single default auxiliary stream with a labeled, dictionary-based approach to supply multiple auxiliary GPU streams. This enables finer-grained control of parallel operations and more flexible, efficient execution for latency-sensitive models.

Description

Test Coverage

PR Checklist

Please review the following before submitting your PR:

  • PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

  • PR FollowsTRT-LLM CODING GUIDELINES to the best of your knowledge.

  • Test cases are provided for new code paths (seetest instructions)

  • Any new dependencies have been scanned for license and vulnerabilities

  • CODEOWNERS updated if ownership changes

  • Documentation updated as needed

  • Updatetava architecture diagram if there is a significant design change in PR.

  • The reviewers assigned automatically/manually are appropriate for the PR.

  • Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

/bot [-h] ['run', 'kill', 'skip', 'reuse-pipeline'] ...

Provide a user friendly way for developers to interact with a Jenkins server.

Run/bot [-h|--help] to print this help message.

See details below for each supported subcommand.

run [--reuse-test (optional)pipeline-id --disable-fail-fast --skip-test --stage-list "A10-PyTorch-1, xxx" --gpu-type "A30, H100_PCIe" --test-backend "pytorch, cpp" --add-multi-gpu-test --only-multi-gpu-test --disable-multi-gpu-test --post-merge --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx" --detailed-log --debug(experimental)]

Launch build/test pipelines. All previously running jobs will be killed.

--reuse-test (optional)pipeline-id(OPTIONAL) : Allow the new pipeline to reuse build artifacts and skip successful test stages from a specified pipeline or the last pipeline if no pipeline-id is indicated. If the Git commit ID has changed, this option will be always ignored. The DEFAULT behavior of the bot is to reuse build artifacts and successful test results from the last pipeline.

--disable-reuse-test(OPTIONAL) : Explicitly prevent the pipeline from reusing build artifacts and skipping successful test stages from a previous pipeline. Ensure that all builds and tests are run regardless of previous successes.

--disable-fail-fast(OPTIONAL) : Disable fail fast on build/tests/infra failures.

--skip-test(OPTIONAL) : Skip all test stages, but still run build stages, package stages and sanity check stages. Note: DoesNOT update GitHub check status.

--stage-list "A10-PyTorch-1, xxx"(OPTIONAL) : Only run the specified test stages. Examples: "A10-PyTorch-1, xxx". Note: DoesNOT update GitHub check status.

--gpu-type "A30, H100_PCIe"(OPTIONAL) : Only run the test stages on the specified GPU types. Examples: "A30, H100_PCIe". Note: DoesNOT update GitHub check status.

--test-backend "pytorch, cpp"(OPTIONAL) : Skip test stages which don't match the specified backends. Only support [pytorch, cpp, tensorrt, triton]. Examples: "pytorch, cpp" (does not run test stages with tensorrt or triton backend). Note: DoesNOT update GitHub pipeline status.

--only-multi-gpu-test(OPTIONAL) : Only run the multi-GPU tests. Note: DoesNOT update GitHub check status.

--disable-multi-gpu-test(OPTIONAL) : Disable the multi-GPU tests. Note: DoesNOT update GitHub check status.

--add-multi-gpu-test(OPTIONAL) : Force run the multi-GPU tests in addition to running L0 pre-merge pipeline.

--post-merge(OPTIONAL) : Run the L0 post-merge pipeline instead of the ordinary L0 pre-merge pipeline.

--extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx"(OPTIONAL) : Run the ordinary L0 pre-merge pipeline and specified test stages. Examples: --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx".

--detailed-log(OPTIONAL) : Enable flushing out all logs to the Jenkins console. This will significantly increase the log volume and may slow down the job.

--debug(OPTIONAL) :Experimental feature. Enable access to the CI container for debugging purpose. Note: Specify exactly one stage in thestage-list parameter to access the appropriate container environment. Note: DoesNOT update GitHub check status.

For guidance on mapping tests to stage names, seedocs/source/reference/ci-overview.md
and thescripts/test_to_stage_mapping.py helper.

kill

kill

Kill all running builds associated with pull request.

skip

skip --comment COMMENT

Skip testing for latest commit on pull request.--comment "Reason for skipping build/test" is required. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.

reuse-pipeline

reuse-pipeline

Reuse a previous pipeline to validate current commit. This action will also kill all currently running builds associated with the pull request. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.

@jinyangyuan-nvidiajinyangyuan-nvidia changed the title[https://nvbugs/5613089][fix] Fix the aux_stream in Llama4MinLatencyFusedMoE[None][fix] Fix the aux_stream in Llama4MinLatencyFusedMoENov 10, 2025
@coderabbitai
Copy link
Contributor

coderabbitaibot commentedNov 10, 2025
edited
Loading

📝 Walkthrough

Walkthrough

The changes replace a single auxiliary stream parameter with a dictionary-based configuration in two Llama model classes. Parameter type shifts fromtorch.cuda.Stream toOptional[Dict[AuxStreamType, torch.cuda.Stream]] in bothLlama4MinLatencyLinear andLlama4MinLatencyFusedMoE constructors. The import statement is updated to includeAuxStreamType.

Changes

Cohort / File(s)Summary
Auxiliary stream configuration refactoring
tensorrt_llm/_torch/models/modeling_llama_min_latency.py
UpdatedLlama4MinLatencyLinear.__init__ andLlama4MinLatencyFusedMoE.__init__ to acceptaux_stream_dict: Optional[Dict[AuxStreamType, torch.cuda.Stream]] instead ofaux_stream: torch.cuda.Stream. Both constructors now passaux_stream_dict=aux_stream_dict to their parent class. AddedAuxStreamType to imports from..utils.

Sequence Diagram(s)

sequenceDiagram    participant Caller    participant LlamaClass as Llama4MinLatency{Linear/FusedMoE}    participant BaseClass    Note over Caller,LlamaClass `#DDDDFF`: New call site supplies aux_stream_dict    Caller->>LlamaClass: __init__(..., aux_stream_dict=Dict[AuxStreamType, Stream])    LlamaClass->>BaseClass: super().__init__(..., aux_stream_dict=aux_stream_dict)    BaseClass-->>LlamaClass: initialized with labeled auxiliary streams    LlamaClass-->>Caller: instance ready
Loading

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~5–10 minutes

  • Verify constructor callers are updated to pass a dictionary (or None) instead of a single stream.
  • Check type imports and any type-checking/config tooling implications (e.g., mypy).
  • Ensure runtime behavior for absent/partial aux_stream_dict entries is handled by the base class.

Pre-merge checks and finishing touches

❌ Failed checks (2 warnings)
Check nameStatusExplanationResolution
Docstring Coverage⚠️ WarningDocstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.You can run@coderabbitai generate docstrings to improve docstring coverage.
Description check⚠️ WarningThe PR description is missing critical implementation details. The Description and Test Coverage sections are empty, leaving it unclear what problem is being fixed and why.Add a detailed description explaining the issue, the solution approach, and the test coverage for the aux_stream changes in Llama4MinLatencyFusedMoE and Llama4MinLatencyLinear.
✅ Passed checks (1 passed)
Check nameStatusExplanation
Title check✅ PassedThe title clearly identifies the specific change: fixing aux_stream handling in Llama4MinLatencyFusedMoE, which aligns with the changeset's main objective of replacing scalar aux_stream with aux_stream_dict.
✨ Finishing touches
  • 📝 Generate docstrings
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment

Thanks for usingCodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment@coderabbitai help to get the list of available commands and usage tips.

@kaiyux
Copy link
Member

@jinyangyuan-nvidia Thanks a lot for the quick fix!

Copy link
Collaborator

@jiahancjiahanc left a comment
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

I think theLlama4MinLatencyMoE also need update?
Also other classes likeLlama4MinLatencyDecoderLayer which calles theLlama4MinLatencyMoE

@jinyangyuan-nvidia
Copy link
CollaboratorAuthor

jinyangyuan-nvidia commentedNov 10, 2025
edited
Loading

The original code does not passaux_stream toself.experts. This PR has been modified to passaux_stream_dict to it.

Copy link
Collaborator

@jiahancjiahanc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Thanks for the fix!

@jinyangyuan-nvidia
Copy link
CollaboratorAuthor

/bot run

@tensorrt-cicd
Copy link
Collaborator

PR_Github #23980 [ run ] triggered by Bot. Commit:9c37e06

@kaiyuxkaiyuxenabled auto-merge (squash)November 10, 2025 07:27
@tensorrt-cicd
Copy link
Collaborator

PR_Github #23980 [ run ] completed with stateSUCCESS. Commit:9c37e06
/LLM/main/L0_MergeRequest_PR pipeline #18057 completed with status: 'FAILURE'

@jinyangyuan-nvidia
Copy link
CollaboratorAuthor

/bot run

@tensorrt-cicd
Copy link
Collaborator

PR_Github #24014 [ run ] triggered by Bot. Commit:9c37e06

@tensorrt-cicd
Copy link
Collaborator

PR_Github #24014 [ run ] completed with stateSUCCESS. Commit:9c37e06
/LLM/main/L0_MergeRequest_PR pipeline #18091 completed with status: 'FAILURE'

@kaiyux
Copy link
Member

/bot run

@tensorrt-cicd
Copy link
Collaborator

PR_Github #24063 [ run ] triggered by Bot. Commit:9c37e06

@tensorrt-cicd
Copy link
Collaborator

PR_Github #24063 [ run ] completed with stateSUCCESS. Commit:9c37e06
/LLM/main/L0_MergeRequest_PR pipeline #18135 completed with status: 'FAILURE'

@jinyangyuan-nvidia
Copy link
CollaboratorAuthor

/bot run

@tensorrt-cicd
Copy link
Collaborator

PR_Github #24084 [ run ] triggered by Bot. Commit:9c37e06

@tensorrt-cicd
Copy link
Collaborator

PR_Github #24084 [ run ] completed with stateSUCCESS. Commit:9c37e06
/LLM/main/L0_MergeRequest_PR pipeline #18150 completed with status: 'FAILURE'

@jinyangyuan-nvidia
Copy link
CollaboratorAuthor

/bot run

@tensorrt-cicd
Copy link
Collaborator

PR_Github #24114 [ run ] triggered by Bot. Commit:78ac556

@tensorrt-cicd
Copy link
Collaborator

PR_Github #24114 [ run ] completed with stateFAILURE. Commit:78ac556
/LLM/main/L0_MergeRequest_PR pipeline #18171 completed with status: 'FAILURE'

@jinyangyuan-nvidia
Copy link
CollaboratorAuthor

/bot run

@tensorrt-cicd
Copy link
Collaborator

PR_Github #24186 [ run ] triggered by Bot. Commit:b16d9c9

@tensorrt-cicd
Copy link
Collaborator

PR_Github #24186 [ run ] completed with stateSUCCESS. Commit:b16d9c9
/LLM/main/L0_MergeRequest_PR pipeline #18236 completed with status: 'FAILURE'

@jinyangyuan-nvidia
Copy link
CollaboratorAuthor

/bot run

@tensorrt-cicd
Copy link
Collaborator

PR_Github #24225 [ run ] triggered by Bot. Commit:fdb922e

@tensorrt-cicd
Copy link
Collaborator

PR_Github #24225 [ run ] completed with stateSUCCESS. Commit:fdb922e
/LLM/main/L0_MergeRequest_PR pipeline #18271 completed with status: 'FAILURE'

@kaiyux
Copy link
Member

/bot run

@tensorrt-cicd
Copy link
Collaborator

PR_Github #24244 [ run ] triggered by Bot. Commit:fdb922e

@tensorrt-cicd
Copy link
Collaborator

PR_Github #24244 [ run ] completed with stateSUCCESS. Commit:fdb922e
/LLM/main/L0_MergeRequest_PR pipeline #18289 completed with status: 'FAILURE'

Signed-off-by: Jinyang Yuan <154768711+jinyangyuan-nvidia@users.noreply.github.com>
@kaiyuxkaiyuxforce-pushed thedev/fix_llama_moe_aux_stream branch fromfdb922e toefe2c03CompareNovember 13, 2025 07:08
@kaiyux
Copy link
Member

/bot run --disable-fail-fast

@tensorrt-cicd
Copy link
Collaborator

PR_Github #24420 [ run ] triggered by Bot. Commit:efe2c03

@tensorrt-cicd
Copy link
Collaborator

PR_Github #24420 [ run ] completed with stateSUCCESS. Commit:efe2c03
/LLM/main/L0_MergeRequest_PR pipeline #18425 completed with status: 'SUCCESS'
Pipeline passed with automatic retried tests. Check thererun report for details.

@kaiyuxkaiyux merged commit12f339f intoNVIDIA:mainNov 13, 2025
5 checks passed
zheyuf pushed a commit to zheyuf/TensorRT-LLM that referenced this pull requestNov 19, 2025
Signed-off-by: Jinyang Yuan <154768711+jinyangyuan-nvidia@users.noreply.github.com>
greg-kwasniewski1 pushed a commit to nv-auto-deploy/TensorRT-LLM that referenced this pull requestNov 20, 2025
Signed-off-by: Jinyang Yuan <154768711+jinyangyuan-nvidia@users.noreply.github.com>
Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment

Reviewers

@byshiuebyshiuebyshiue approved these changes

@kaiyuxkaiyuxkaiyux approved these changes

@2ez4bz2ez4bzAwaiting requested review from 2ez4bz2ez4bz is a code owner automatically assigned from NVIDIA/trt-llm-torch-models-devs

+1 more reviewer

@jiahancjiahancjiahanc approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@jinyangyuan-nvidia@kaiyux@tensorrt-cicd@byshiue@jiahanc

[8]ページ先頭

©2009-2025 Movatter.jp