Saturated Transformers are Constant-Depth Threshold Circuits
원문 초록
Transformers have become a standard neural network architecture for many NLP problems, motivating theoretical analysis of their power in terms of formal languages. Recent work has shown that transformers with hard attention are quite limited in power (Hahn, 2020), as they can be simulated by constant-depth AND/OR circuits (Hao et al. 2021). However, hard attention is a strong assumption, which may complicate the relevance of these results in practice. In this work, we analyze the circuit complexity of transformers with saturated attention: a generalization of hard attention that more closely captures the attention patterns learnable in practical transformers. We first show that saturated transformers transcend the known limitations of hard-attention transformers. We then prove saturated transformers with floating-point values can be simulated by constant-depth threshold circuits, giving the class $\mathsf{TC}^0$ as an upper bound on the formal languages they recognize.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Center-guided Classifier for Semantic Segmentation of Remote Sensing Images
Hard-Attention Gates with Gradient Routing for Endoscopic Image Computing
TRIP: Trainable Region-of-Interest Prediction for Hardware-Efficient Neuromorphic Processing on Event-based Vision
Recurrent Alignment with Hard Attention for Hierarchical Text Rating
Mutual Distillation Learning For Person Re-Identification
Vamos: Versatile Action Models for Video Understanding
토론
이 논문과 연결된 포럼 글이 아직 없습니다.