Category-Selective Neurons in Deep Networks: Comparing Purely Visual and Visual-Language Models
원문 초록
Category-selective regions in the human brain, such as the fusiform face area (FFA), extrastriate body area (EBA), parahippocampal place area (PPA), and visual word form area (VWFA), play a crucial role in high-level visual processing. Here, we investigate whether artificial neural networks (ANNs) exhibit similar category-selective neurons and how these neurons vary across model layers and between purely visual and vision-language models. Inspired by fMRI functional localizer experiments, we presented images from different categories (faces, bodies, scenes, words, scrambled scenes, and scrambled words) to deep networks and identified category-selective neurons using statistical criteria. Comparing ResNet and the structurally controlled ResNet-based CLIP model, we found that both models contain category-selective neurons, with their proportion increasing across layers, mirroring category selectivity in higher-level visual brain regions. However, CLIP exhibited a higher proportion but lower specificity of category-selective neurons compared to ResNet. Additionally, CLIP's category-selective neurons were more evenly distributed across feature maps and demonstrated greater representational consistency across layers. These findings suggest that language learning increases the number of category-selective neurons while reducing their selectivity strength, reshaping visual representations in deep networks. Our study provides insights into how ANNs mirror biological vision and how multimodal learning influences category-selective representations.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
RadiomicsRetrieval: A Customizable Framework for Medical Image Retrieval Using Radiomics Features
IMC-PINN-FE: A Physics-Informed Neural Network for Patient-Specific Left Ventricular Finite Element Modeling with Image Motion Consistency and Biomechanical Parameter Estimation
Visual-Semantic Knowledge Conflicts in Operating Rooms: Synthetic Data Curation for Surgical Risk Perception in Multimodal Large Language Models
A Modular Taxonomy for Hate Speech Definitions and Its Impact on Zero-Shot LLM Classification Performance
OmniESI: A unified framework for enzyme-substrate interaction prediction with progressive conditional deep learning
Single-cell metabolic flux analysis reveals coexisting optimal sub-groups, cross-feeding, and mixotrophy in a cyanobacterial population
토론
이 논문과 연결된 포럼 글이 아직 없습니다.