On Efficient Constructions of Checkpoints
원문 초록
Efficient construction of checkpoints/snapshots is a critical tool for training and diagnosing deep learning models. In this paper, we propose a lossy compression scheme for checkpoint constructions (called LC-Checkpoint). LC-Checkpoint simultaneously maximizes the compression rate and optimizes the recovery speed, under the assumption that SGD is used to train the model. LC-Checkpointuses quantization and priority promotion to store the most crucial information for SGD to recover, and then uses a Huffman coding to leverage the non-uniform distribution of the gradient scales. Our extensive experiments show that LC-Checkpoint achieves a compression rate up to $28\times$ and recovery speedup up to $5.77\times$ over a state-of-the-art algorithm (SCAR).
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
MGVQ: Could VQ-VAE Beat VAE? A Generalizable Tokenizer with Multi-group Quantization
Compress Any Segment Anything Model (SAM)
OpenDPDv2: A Unified Learning and Optimization Framework for Neural Network Digital Predistortion
EdgeCodec: Onboard Lightweight High Fidelity Neural Compressor with Residual Vector Quantization
any4: Learned 4-bit Numeric Representation for LLMs
CycleVAR: Repurposing Autoregressive Model for Unsupervised One-Step Image Translation
토론
이 논문과 연결된 포럼 글이 아직 없습니다.