GSPMD: General and Scalable Parallelization for ML Computation Graphs
원문 초록
We present GSPMD, an automatic, compiler-based parallelization system for common machine learning computations. It allows users to write programs in the same way as for a single device, then give hints through a few annotations on how to distribute tensors, based on which GSPMD will parallelize the computation. Its representation of partitioning is simple yet general, allowing it to express different or mixed paradigms of parallelism on a wide variety of models. GSPMD infers the partitioning for every operator based on limited user annotations, making it convenient to scale existing single-device programs. It solves several technical challenges for production usage, allowing GSPMD to achieve 50% to 62% compute utilization on up to 2048 Cloud TPUv3 cores for models with up to one trillion parameters.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Perceptual Similarity for Measuring Decision-Making Style and Policy Diversity in Games
PFGM++: Unlocking the Potential of Physics-Inspired Generative Models
Parallel Context Windows for Large Language Models
On Reinforcement Learning for the Game of 2048
Efficient Human Pose Estimation via 3D Event Point Cloud
Characterizing the Efficiency vs. Accuracy Trade-off for Long-Context NLP Models
토론
이 논문과 연결된 포럼 글이 아직 없습니다.