Microsoft's updated DeepSpeed can train trillion-parameter AI models with fewer GPUs

Very impressive! The human brain has only about 86 billion neurons. Until very recently training a billion parameters was considered huge and was achieved by only a few models.



"The company claims the technique, dubbed 3D parallelism, adapts to the varying needs of workload requirements to power extremely large models while balancing scaling efficiency. ... The enhanced DeepSpeed leverages three techniques to enable “trillion-scale” model training: data parallel training, model parallel training, and pipeline parallel training. "



Microsoft's updated DeepSpeed can train trillion-parameter AI models with fewer GPUs | VentureBeat Microsoft today released an updated version of its DeepSpeed library that introduces a new approach to training AI models containing trillions of parameters