Beyond Black Boxes: Enhancing Interpretability of Transformers Trained on Neural Data
원문 초록
Transformer models have become state-of-the-art in decoding stimuli and behavior from neural activity, significantly advancing neuroscience research. Yet greater transparency in their decision-making processes would substantially enhance their utility in scientific and clinical contexts. Sparse autoencoders offer a promising solution by producing hidden units that respond selectively to specific variables, enhancing interpretability. Here, we introduce SAEs into a neural decoding framework by augmenting a transformer trained to predict visual stimuli from calcium imaging in the mouse visual cortex. The enhancement of the transformer model with an SAE preserved its original performance while yielding hidden units that selectively responded to interpretable features, such as stimulus orientation and genetic background. Furthermore, ablating units associated with a given variable impaired the model's ability to process that variable, revealing how specific internal representations support downstream computations. Together, these results demonstrate that integrating SAEs with transformers combines the power of modern deep learning with the interpretability essential for scientific understanding and clinical translation.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Graph-Structured Data Analysis of Component Failure in Autonomous Cargo Ships Based on Feature Fusion
Guiding LLM Decision-Making with Fairness Reward Models
InvestAlign: Overcoming Data Scarcity in Aligning Large Language Models with Investor Decision-Making Processes under Herd Behavior
VERITAS: Verification and Explanation of Realness in Images for Transparency in AI Systems
Enhancing LLM Agent Safety via Causal Influence Prompting
NavMorph: A Self-Evolving World Model for Vision-and-Language Navigation in Continuous Environments
토론
이 논문과 연결된 포럼 글이 아직 없습니다.