AdaComp : Adaptive Residual Gradient Compression for Data-Parallel Distributed Training
원문 초록
Highly distributed training of Deep Neural Networks (DNNs) on future compute platforms (offering 100 of TeraOps/s of computational capacity) is expected to be severely communication constrained. To overcome this limitation, new gradient compression techniques are needed that are computationally friendly, applicable to a wide variety of layers seen in Deep Neural Networks and adaptable to variations in network architectures as well as their hyper-parameters. In this paper we introduce a novel technique - the Adaptive Residual Gradient Compression (AdaComp) scheme. AdaComp is based on localized selection of gradient residues and automatically tunes the compression rate depending on local activity. We show excellent results on a wide spectrum of state of the art Deep Learning models in multiple domains (vision, speech, language), datasets (MNIST, CIFAR10, ImageNet, BN50, Shakespeare), optimizers (SGD with momentum, Adam) and network parameters (number of learners, minibatch-size etc.). Exploiting both sparsity and quantization, we demonstrate end-to-end compression rates of ~200X for fully-connected and recurrent layers, and ~40X for convolutional layers, without any noticeable degradation in model accuracies.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
MGVQ: Could VQ-VAE Beat VAE? A Generalizable Tokenizer with Multi-group Quantization
Compress Any Segment Anything Model (SAM)
OpenDPDv2: A Unified Learning and Optimization Framework for Neural Network Digital Predistortion
EdgeCodec: Onboard Lightweight High Fidelity Neural Compressor with Residual Vector Quantization
any4: Learned 4-bit Numeric Representation for LLMs
CycleVAR: Repurposing Autoregressive Model for Unsupervised One-Step Image Translation
토론
이 논문과 연결된 포럼 글이 아직 없습니다.