Efficient 8-Bit Quantization of Transformer Neural Machine Language Translation Model
원문 초록
In this work, we quantize a trained Transformer machine language translation model leveraging INT8/VNNI instructions in the latest Intel$^\circledR$ Xeon$^\circledR$ Cascade Lake processors to improve inference performance while maintaining less than 0.5$\%$ drop in accuracy. To the best of our knowledge, this is the first attempt in the industry to quantize the Transformer model. This has high impact as it clearly demonstrates the various complexities of quantizing the language translation model. We present novel quantization techniques directly in TensorFlow to opportunistically replace 32-bit floating point (FP32) computations with 8-bit integers (INT8) and transform the FP32 computational graph. We also present a bin-packing parallel batching technique to maximize CPU utilization. Overall, our optimizations with INT8/VNNI deliver 1.5X improvement over the best FP32 performance. Furthermore, it reveals the opportunities and challenges to boost performance of quantized deep learning inference and establishes best practices to run inference with high efficiency on Intel CPUs.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
CycleVAR: Repurposing Autoregressive Model for Unsupervised One-Step Image Translation
TailorKV: A Hybrid Framework for Long-Context Inference via Tailored KV Cache Optimization
Efficient Speech Translation through Model Compression and Knowledge Distillation
Implementing Keyword Spotting on the MCXN947 Microcontroller with Integrated NPU
Variational Bayesian Channel Estimation and Data Detection for Cell-Free Massive MIMO with Low-Resolution Quantized Fronthaul Links
MNN-LLM: A Generic Inference Engine for Fast Large Language Model Deployment on Mobile Devices
토론
이 논문과 연결된 포럼 글이 아직 없습니다.