Gap Aware Mitigation of Gradient Staleness
원문 초록
Cloud computing is becoming increasingly popular as a platform for distributed training of deep neural networks. Synchronous stochastic gradient descent (SSGD) suffers from substantial slowdowns due to stragglers if the environment is non-dedicated, as is common in cloud computing. Asynchronous SGD (ASGD) methods are immune to these slowdowns but are scarcely used due to gradient staleness, which encumbers the convergence process. Recent techniques have had limited success mitigating the gradient staleness when scaling up to many workers (computing nodes). In this paper we define the Gap as a measure of gradient staleness and propose Gap-Aware (GA), a novel asynchronous-distributed method that penalizes stale gradients linearly to the Gap and performs well even when scaling to large numbers of workers. Our evaluation on the CIFAR, ImageNet, and WikiText-103 datasets shows that GA outperforms the currently acceptable gradient penalization method, in final test accuracy. We also provide convergence rate proof for GA. Despite prior beliefs, we show that if GA is applied, momentum becomes beneficial in asynchronous environments, even when the number of workers scales up.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
GATES: Cost-aware Dynamic Workflow Scheduling via Graph Attention Networks and Evolution Strategy
Enhanced Outsourced and Secure Inference for Tall Sparse Decision Trees
InTec: integrated things-edge computing: a framework for distributing machine learning pipelines in edge AI systems
Persistence of Backdoor-based Watermarks for Neural Networks: A Comprehensive Evaluation
Toward Intelligent and Secure Cloud: Large Language Model Empowered Proactive Defense
All-in-one foundational models learning across quantum chemical levels
토론
이 논문과 연결된 포럼 글이 아직 없습니다.