Reducing the Model Order of Deep Neural Networks Using Information Theory
원문 초록
Deep neural networks are typically represented by a much larger number of parameters than shallow models, making them prohibitive for small footprint devices. Recent research shows that there is considerable redundancy in the parameter space of deep neural networks. In this paper, we propose a method to compress deep neural networks by using the Fisher Information metric, which we estimate through a stochastic optimization method that keeps track of second-order information in the network. We first remove unimportant parameters and then use non-uniform fixed point quantization to assign more bits to parameters with higher Fisher Information estimates. We evaluate our method on a classification task with a convolutional neural network trained on the MNIST data set. Experimental results show that our method outperforms existing methods for both network pruning and quantization.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Formal Uncertainty Propagation for Stochastic Dynamical Systems with Additive Noise
MGVQ: Could VQ-VAE Beat VAE? A Generalizable Tokenizer with Multi-group Quantization
Compress Any Segment Anything Model (SAM)
OpenDPDv2: A Unified Learning and Optimization Framework for Neural Network Digital Predistortion
EdgeCodec: Onboard Lightweight High Fidelity Neural Compressor with Residual Vector Quantization
any4: Learned 4-bit Numeric Representation for LLMs
토론
이 논문과 연결된 포럼 글이 아직 없습니다.