NotificationsYou must be signed in to change notification settings
Fork26.3k
Star96k

[ONNX] Support running bfloat16 models with ONNX Runtime#149646

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Jump to bottom

Closed

justinchuby wants to merge5 commits intopytorch:mainfromjustinchuby:justinchu/ort-bfloat16

Closed

[ONNX] Support running bfloat16 models with ONNX Runtime#149646

justinchuby wants to merge5 commits intopytorch:mainfromjustinchuby:justinchu/ort-bfloat16

Conversation

Copy link

Collaborator

justinchuby commentedMar 20, 2025•
edited
Loading

Use ORTValue objects to support bfloat16 and other dtypes as inputs. This only supports cuda as ort only implements bfloat16 on cuda.

justinchuby requested review fromshubhambhokare1,titaiwangms andwschin ascode owners

March 20, 2025 18:40

pytorch-botbot added the release notes: onnxtorch.onnx related changes that should show up in the release notes label

Mar 20, 2025

Copy link

pytorch-botbot commentedMar 20, 2025•
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results athud.pytorch.org/pr/149646

📄 PreviewPython docs built from this PR
📄 PreviewC++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit thebot commands wiki or ouroffice hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit27dc679 with merge basee945247 ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

pytorchbot added the open source label

Mar 20, 2025

bdhirsh added the triagedThis issue has been looked at a team member, and triaged and prioritized into an appropriate module label

Mar 24, 2025

justinchuby added module: onnx

Related to torch.onnx

topic: improvementstopic category labels

Mar 28, 2025

justinchuby assignedtitaiwangms

Mar 28, 2025

titaiwangms reviewed

Mar 28, 2025

View reviewed changes

torch/onnx/_internal/exporter/_onnx_program.pyShow resolvedHide resolved

Copy link

CollaboratorAuthor

justinchuby commentedMar 28, 2025

@pytorchbot rebase

Copy link

Collaborator

pytorchmergebot commentedMar 28, 2025

@pytorchbot started a rebase job ontorefs/remotes/origin/viable/strict. Check the current statushere

Copy link

Collaborator

pytorchmergebot commentedMar 28, 2025

Successfully rebasedjustinchu/ort-bfloat16 ontorefs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, viagit checkout justinchu/ort-bfloat16 && git pull --rebase)

pytorchmergebot force-pushed thejustinchu/ort-bfloat16 branch from04d44a3 to01f698aCompare

March 28, 2025 21:26

Copy link

CollaboratorAuthor

justinchuby commentedMar 31, 2025

@pytorchbot rebase

Copy link

Collaborator

pytorchmergebot commentedMar 31, 2025

@pytorchbot started a rebase job ontorefs/remotes/origin/viable/strict. Check the current statushere

Copy link

Collaborator

pytorchmergebot commentedMar 31, 2025

pytorchmergebot force-pushed thejustinchu/ort-bfloat16 branch from01f698a to475444bCompare

March 31, 2025 18:45

justinchuby mentioned this pull request

Mar 31, 2025

Avoid circular imports in tracing_state_functions#150325

Closed

pytorchmergebot pushed a commit that referenced this pull request

Apr 9, 2025

Avoid circular imports in tracing_state_functions (#150325)

72755a4

tracing_state_functions references some torch functions from submodules like `torch.onnx.is_in_onnx_export` that could trigger module initialization & circular imports. I turned the mapping into a function so that the dictionary is not initialized at torch import.(discovered in#149646)Pull Requestresolved:#150325Approved by:https://github.com/zou3519

Copy link

CollaboratorAuthor

justinchuby commentedApr 9, 2025

@pytorchbot rebase -b main

Copy link

Collaborator

pytorchmergebot commentedApr 9, 2025

@pytorchbot started a rebase job ontorefs/remotes/origin/main. Check the current statushere

Copy link

Collaborator

pytorchmergebot commentedApr 9, 2025

Successfully rebasedjustinchu/ort-bfloat16 ontorefs/remotes/origin/main, please pull locally before adding more changes (for example, viagit checkout justinchu/ort-bfloat16 && git pull --rebase)

pytorchmergebot force-pushed thejustinchu/ort-bfloat16 branch from475444b to014a924Compare

April 9, 2025 16:46

justinchuby requested a review fromtitaiwangms

April 9, 2025 17:09

titaiwangms approved these changes

Apr 10, 2025

View reviewed changes

Copy link

CollaboratorAuthor

justinchuby commentedApr 11, 2025

@pytorchbot rebase -b main

Copy link

Collaborator

pytorchmergebot commentedApr 11, 2025

@pytorchbot started a rebase job ontorefs/remotes/origin/main. Check the current statushere

[ONNX] Support running bfloat16 models with ONNX Runtime

44f4b07

justinchuby added4 commits

April 11, 2025 00:21

Support bfloat16

20fa83b

Reorder import statements

63f5e2f

Hide import

abca01f

Handle errors

27dc679

Copy link

Collaborator

pytorchmergebot commentedApr 11, 2025

pytorchmergebot force-pushed thejustinchu/ort-bfloat16 branch from810bba3 to27dc679Compare

April 11, 2025 00:21

Copy link

CollaboratorAuthor

justinchuby commentedApr 11, 2025

@pytorchbot merge

pytorch-botbot added the ciflow/trunkTrigger trunk jobs on your pull request label

Apr 11, 2025

pytorchmergebot added the merging label

Apr 11, 2025

Copy link

Collaborator

pytorchmergebot commentedApr 11, 2025

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in thewiki.

Questions? Feedback? Please reach out to thePyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot added the Merged label

Apr 11, 2025

pytorchmergebot closed this in75162aa

Apr 11, 2025

pytorchmergebot removed the merging label

Apr 11, 2025

timocafe pushed a commit to timocafe/pytorch that referenced this pull request

Apr 16, 2025

Avoid circular imports in tracing_state_functions (pytorch#150325)

98b252a

tracing_state_functions references some torch functions from submodules like `torch.onnx.is_in_onnx_export` that could trigger module initialization & circular imports. I turned the mapping into a function so that the dictionary is not initialized at torch import.(discovered inpytorch#149646)Pull Requestresolved:pytorch#150325Approved by:https://github.com/zou3519

timocafe pushed a commit to timocafe/pytorch that referenced this pull request

Apr 16, 2025

[ONNX] Support running bfloat16 models with ONNX Runtime (pytorch#149646

8410782

)Use ORTValue objects to support bfloat16 and other dtypes as inputs. This only supports cuda as ort only implements bfloat16 on cuda.Pull Requestresolved:pytorch#149646Approved by:https://github.com/titaiwangms

amathewc pushed a commit to amathewc/pytorch that referenced this pull request

Apr 17, 2025

Avoid circular imports in tracing_state_functions (pytorch#150325)

df21a5f

tracing_state_functions references some torch functions from submodules like `torch.onnx.is_in_onnx_export` that could trigger module initialization & circular imports. I turned the mapping into a function so that the dictionary is not initialized at torch import.(discovered inpytorch#149646)Pull Requestresolved:pytorch#150325Approved by:https://github.com/zou3519

amathewc pushed a commit to amathewc/pytorch that referenced this pull request

Apr 17, 2025

[ONNX] Support running bfloat16 models with ONNX Runtime (pytorch#149646

734d370

)Use ORTValue objects to support bfloat16 and other dtypes as inputs. This only supports cuda as ort only implements bfloat16 on cuda.Pull Requestresolved:pytorch#149646Approved by:https://github.com/titaiwangms