Theoretical Limitations of Self-Attention in Neural Sequence Models
원문 초록
Transformers are emerging as the new workhorse of NLP, showing great success across tasks. Unlike LSTMs, transformers process input sequences entirely through self-attention. Previous work has suggested that the computational capabilities of self-attention to process hierarchical structures are limited. In this work, we mathematically investigate the computational power of self-attention to model formal languages. Across both soft and hard attention, we show strong theoretical limitations of the computational abilities of self-attention, finding that it cannot model periodic finite-state languages, nor hierarchical structure, unless the number of layers or heads increases with input length. These limitations seem surprising given the practical success of self-attention and the prominent role assigned to hierarchical structure in linguistics, suggesting that natural language can be approximated well with models that are too weak for the formal languages typically assumed in theoretical linguistics.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Center-guided Classifier for Semantic Segmentation of Remote Sensing Images
Hard-Attention Gates with Gradient Routing for Endoscopic Image Computing
TRIP: Trainable Region-of-Interest Prediction for Hardware-Efficient Neuromorphic Processing on Event-based Vision
Recurrent Alignment with Hard Attention for Hierarchical Text Rating
Mutual Distillation Learning For Person Re-Identification
Vamos: Versatile Action Models for Video Understanding
토론
이 논문과 연결된 포럼 글이 아직 없습니다.