A Production System for Training Large Language Models at Scale

Harnessing the Power of Over 10,000 GPUs for Efficient and Stable Training

In the realm of artificial intelligence, large language models (LLMs) have emerged as a transformative technology with the potential to revolutionize a wide range of domains. To achieve state-of-the-art model capability, training LLMs requires enormous computation resources and presents unprecedented challenges in terms of training efficiency and stability. In this article, we delve into the design, implementation, and engineering experience of MegaScale, a production system for training LLMs at scale.

MegaScale: A Specialized System for LLM Training

MegaScale is a specialized system tailored for LLM training that applies two systems principles: algorithm-system co-design and in-depth observability. By taking a full-stack approach, MegaScale spans all important system components, including modifications to the model architecture, parallelism strategies, data pipeline optimization, and network performance tuning.

Algorithm-System Co-Design

To maximize performance, MegaScale incorporates several modifications and optimization techniques to the model architecture. These include a parallel transformer block, sliding window attention, and the LAMB optimizer. Additionally, MegaScale leverages mixed parallelism strategies that combine data parallelism, pipeline parallelism, tensor parallelism, and sequence parallelism. To maximize the overlapping between communication and computation, custom techniques are designed based on the pattern of each parallelism strategy.

In-Depth Observability

To diagnose and fix stability problems, including failures and stragglers, MegaScale applies the principle of in-depth observability. This comprehensive monitoring and visualization strategy gathers detailed, granular data across every component of the system stack, aiming to create a multidimensional view of system performance. The set of tools allows for diagnosing the system, identifying root causes, and automating fault localization and recovery.

Training Efficiency and Stability

MegaScale is deployed in datacenters to train LLMs for various products. The system achieves impressive training efficiency, with a Model FLOPs Utilization (MFU) of 55.2% when training a standard 175B transformer model on 12,288 GPUs. This represents an improvement of 1.34× compared to the state-of-the-art open-source training framework Megatron-LM. In terms of model convergence and stability, MegaScale demonstrates consistent loss convergence and the ability to repair and recover the training process in the presence of failures.

Conclusion

MegaScale is a production-grade system designed to train large language models at scale, harnessing the power of over 10,000 GPUs. By applying algorithm-system co-design and in-depth observability principles, MegaScale achieves high training efficiency and stability. The system offers practical insights for those working on LLM training and paves the way for future research in this rapidly evolving field.

Source: Research Paper