- Notifications
You must be signed in to change notification settings - Fork288
ROCm support#295
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
base:main
Are you sure you want to change the base?
ROCm support#295
Uh oh!
There was an error while loading.Please reload this page.
Conversation
HuggingFaceDocBuilderDev commentedJun 19, 2024
The docs for this PR livehere. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Thanks!
| @@ -0,0 +1,34 @@ | |||
| ## Testing | |||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
I think this is equivalent to the integration snapshoting logic inrouter/tests?
| model = AutoModel.from_pretrained(model_path).to(dtype).to(device) | ||
| self.hidden_size = model.config.hidden_size | ||
| self.pooling_mode = pooling_mode |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Can you add mean pooling / CLS pooling forking L46?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
CLS pooling was already there, I added mean pooling.
What is L46?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
@OlivierDehaene, I created apr to fix this
Uh oh!
There was an error while loading.Please reload this page.
Uh oh!
There was an error while loading.Please reload this page.
…-inference into rocm-support
almersawi commentedJul 2, 2024
any chance this MR is merged soon? |
@almersawi it is in good shape to me. |
baddoub commentedJul 19, 2024 • edited
Loading Uh oh!
There was an error while loading.Please reload this page.
edited
Uh oh!
There was an error while loading.Please reload this page.
Hey guys! thanks for this MR. We have been waiting for it for some time now. Any idea when this MR will be merged ? |
ocnimesh commentedAug 22, 2024
Can please anyone provide build steps for docker image with this branch. /opt/conda/lib/python3.10/site-packages/torch/include/ATen/hip/HIPContextLight.h:20:10: fatal error:'hipsolver/hipsolver.h' file not found Command I executed :sudo docker build -f Dockerfile-rocm -t nims123/tei_amd . |
mht-sharma commentedSep 11, 2024
Applied minor fixes to successfully build the Docker image. PR#403 |
sauravsit commentedSep 17, 2024
any ETA on this merge? |
nbroad1881 commentedOct 1, 2024
Is there anything that I can do to help unblock this? |
Co-authored-by: root <root@tw022.pit.tensorwave.lan>
almersawi commentedApr 8, 2025
any updates? |
Uh oh!
There was an error while loading.Please reload this page.
As per title. Support AMD GPUs through TEI backend.
For now, only embedding model with cls/mean pooling is tested.
MI210/MI250/MI300 can dispatch on CK flash attention 2, but other GPUs will default to manual attention implem (or SDPA). Only bert looks to be supported in the python backend.