 | |
| Original author(s) | Microsoft Research |
|---|---|
| Developer(s) | Microsoft |
| Initial release | May 18, 2020; 4 years ago (2020-05-18) |
| Stable release | v0.16.2 / December 18, 2024; 2 months ago (2024-12-18) |
| Repository | github |
| Written in | Python,CUDA,C++ |
| Type | Software library |
| License | Apache License 2.0 |
| Website | deepspeed |
DeepSpeed is anopen sourcedeep learning optimization library forPyTorch.[1]
The library is designed to reduce computing power andmemory use and to train largedistributed models with betterparallelism on existingcomputer hardware.[2][3] DeepSpeed is optimized for low latency, high throughput training. It includes the Zero Redundancy Optimizer (ZeRO) for training models with 1 trillion or more parameters.[4] Features include mixed precision training, single-GPU, multi-GPU, and multi-node training as well as custom model parallelism. The DeepSpeed source code is licensed underMIT License and available onGitHub.[5]
The team claimed to achieve up to a 6.2x throughput improvement, 2.8x faster convergence, and 4.6x less communication.[6]
| Open source | |
|---|---|
| Proprietary |
|
| Main projects |
| ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| MSR Labs applied research |
| ||||||||||||||
