A Survey and Empirical Evaluation of Parallel Deep Learning Frameworks
원문 초록
The field of deep learning has witnessed a remarkable shift towards extremely compute- and memory-intensive neural networks. These newer larger models have enabled researchers to advance state-of-the-art tools across a variety of fields. This phenomenon has spurred the development of algorithms for distributed training of neural networks over a larger number of hardware accelerators. In this paper, we discuss and compare current state-of-the-art frameworks for large scale distributed deep learning. First, we survey current practices in distributed learning and identify the different types of parallelism used. Then, we present empirical results comparing their performance on large image and language training tasks. Additionally, we address their statistical efficiency and memory consumption behavior. Based on our results, we discuss algorithmic and implementation portions of each framework which hinder performance.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
A Survey on Prompt Tuning
A Survey on Latent Reasoning
A Survey on Vision-Language-Action Models for Autonomous Driving
From Tiny Machine Learning to Tiny Deep Learning: A Survey
A Comprehensive Survey on Continual Learning in Generative Models
A Survey on World Models Grounded in Acoustic Physical Information
토론
이 논문과 연결된 포럼 글이 아직 없습니다.