- Notifications
You must be signed in to change notification settings - Fork527
Estimation of running time?#541
-
Hi, I just set up ASR and SV downstreams, but I found that running evaluation (which involves training) on each task took more than 24 hours (with one V100 GPU). Is this normal? May I get an estimation on how it it usually takes to finish evaluation on the whole benchmark? Or I could have missed something. I wrote my own upstream model. Do I need to freeze the base model weights somewhere, or will this library do this automatically? Thanks! |
BetaWas this translation helpful?Give feedback.
All reactions
Replies: 2 comments
-
Yes, it is normal. |
BetaWas this translation helpful?Give feedback.
All reactions
Uh oh!
There was an error while loading.Please reload this page.
Uh oh!
There was an error while loading.Please reload this page.
-
Hi. I also had a query regarding training time. I am training on top of fbank features using default ASR task config described here:https://github.com/s3prl/s3prl/blob/main/s3prl/downstream/docs/superb.md#asr-automatic-speech-recognition I am training on one A100 gpu. It has been 24 hours since training, but the logs show that overall is only 18% complete. This seems very slow to me. Is this accurate? I also noticed that for ASR, the default gradient_accumulate_steps=1. This would suggest that using DDP to speed training would produce behavior that differs from default. |
BetaWas this translation helpful?Give feedback.