Answer Fast: Accelerating BERT on the Tensor Streaming Processor
원문 초록
Transformers have become a predominant machine learning workload, they are not only the de-facto standard for natural language processing tasks, but they are also being deployed in other domains such as vision and speech recognition. Many of the transformer-based applications are real-time systems such as machine translation and web search. These real time systems often come with strict end-to-end inference latency requirements. Unfortunately, while the majority of the transformer computation comes from matrix multiplications, transformers also include several non-linear components that tend to become the bottleneck during an inference. In this work, we accelerate the inference of BERT models on the tensor streaming processor. By carefully fusing all the nonlinear components with the matrix multiplication components, we are able to efficiently utilize the on-chip matrix multiplication units resulting in a deterministic tail latency of 130 $\mu$s for a batch-1 inference through BERT-base, which is 6X faster than the current state-of-the-art.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
KIT's Low-resource Speech Translation Systems for IWSLT2025: System Enhancement with Synthetic Data and Model Regularization
Building a Functional Machine Translation Corpus for Kpelle
HausaNLP: Current Status, Challenges and Future Directions for Hausa Natural Language Processing
TASK AWARE MULTI-TASK LEARNING FOR SPEECH TO TEXT TASKS
First Steps Towards Voice Anonymization for Code-Switching Speech
MambAttention: Mamba with Multi-Head Attention for Generalizable Single-Channel Speech Enhancement
토론
이 논문과 연결된 포럼 글이 아직 없습니다.