Harnessing Manycore Processors with Distributed Memory for Accelerated Training of Sparse and Recurrent Models
원문 초록
Current AI training infrastructure is dominated by single instruction multiple data (SIMD) and systolic array architectures, such as Graphics Processing Units (GPUs) and Tensor Processing Units (TPUs), that excel at accelerating parallel workloads and dense vector matrix multiplications. Potentially more efficient neural network models utilizing sparsity and recurrence cannot leverage the full power of SIMD processor and are thus at a severe disadvantage compared to today's prominent parallel architectures like Transformers and CNNs, thereby hindering the path towards more sustainable AI. To overcome this limitation, we explore sparse and recurrent model training on a massively parallel multiple instruction multiple data (MIMD) architecture with distributed local memory. We implement a training routine based on backpropagation through time (BPTT) for the brain-inspired class of Spiking Neural Networks (SNNs) that feature binary sparse activations. We observe a massive advantage in using sparse activation tensors with a MIMD processor, the Intelligence Processing Unit (IPU) compared to GPUs. On training workloads, our results demonstrate 5-10x throughput gains compared to A100 GPUs and up to 38x gains for higher levels of activation sparsity, without a significant slowdown in training convergence or reduction in final model performance. Furthermore, our results show highly promising trends for both single and multi IPU configurations as we scale up to larger model sizes. Our work paves the way towards more efficient, non-standard models via AI training hardware beyond GPUs, and competitive large scale SNN models.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
LightSNN: Lightweight Architecture Search for Sparse and Accurate Spiking Neural Networks
Automatic Joint Structured Pruning and Quantization for Efficient Neural Network Training and Compression
When do they StOP?: A First Step Towards Automatically Identifying Team Communication in the Operating Room
HALO: Hadamard-Assisted Lower-Precision Optimization for LLMs
Learn2Mix: Training Neural Networks Using Adaptive Data Integration
Efficient Neural Network Encoding for 3D Color Lookup Tables
토론
이 논문과 연결된 포럼 글이 아직 없습니다.