Self-attention in Vision Transformers Performs Perceptual Grouping, Not Attention
원문 초록
Recently, a considerable number of studies in computer vision involves deep neural architectures called vision transformers. Visual processing in these models incorporates computational models that are claimed to implement attention mechanisms. Despite an increasing body of work that attempts to understand the role of attention mechanisms in vision transformers, their effect is largely unknown. Here, we asked if the attention mechanisms in vision transformers exhibit similar effects as those known in human visual attention. To answer this question, we revisited the attention formulation in these models and found that despite the name, computationally, these models perform a special class of relaxation labeling with similarity grouping effects. Additionally, whereas modern experimental findings reveal that human visual attention involves both feed-forward and feedback mechanisms, the purely feed-forward architecture of vision transformers suggests that attention in these models will not have the same effects as those known in humans. To quantify these observations, we evaluated grouping performance in a family of vision transformers. Our results suggest that self-attention modules group figures in the stimuli based on similarity in visual features such as color. Also, in a singleton detection experiment as an instance of saliency detection, we studied if these models exhibit similar effects as those of feed-forward visual salience mechanisms utilized in human visual attention. We found that generally, the transformer-based attention modules assign more salience either to distractors or the ground. Together, our study suggests that the attention mechanisms in vision transformers perform similarity grouping and not attention.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
A Deep Learning Framework for Visual Attention Prediction and Analysis of News Interfaces
Copy-Move Detection in Optical Microscopy: A Segmentation Network and A Dataset
Unlocking Comics: The AI4VA Dataset for Visual Understanding
AIM 2024 Challenge on Video Saliency Prediction: Methods and Results
SUM: Saliency Unification through Mamba for Visual Attention Modeling
ViDSOD-100: A New Dataset and a Baseline Model for RGB-D Video Salient Object Detection
토론
이 논문과 연결된 포럼 글이 아직 없습니다.