Mimetic Initialization Helps State Space Models Learn to Recall
원문 초록
Recent work has shown that state space models such as Mamba are significantly worse than Transformers on recall-based tasks due to the fact that their state size is constant with respect to their input sequence length. But in practice, state space models have fairly large state sizes, and we conjecture that they should be able to perform much better at these tasks than previously reported. We investigate whether their poor copying and recall performance could be due in part to training difficulties rather than fundamental capacity constraints. Based on observations of their "attention" maps, we propose a structured initialization technique that allows state space layers to more readily mimic attention. Across a variety of architecture settings, our initialization makes it substantially easier for Mamba to learn to copy and do associative recall from scratch.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
MambaFusion: Height-Fidelity Dense Global Fusion for Multi-modal 3D Object Detection
Sparsified State-Space Models are Efficient Highway Networks
Hierarchical Mamba Meets Hyperbolic Geometry: A New Paradigm for Structured Language Embeddings
Hybrid-Emba3D: Geometry-Aware and Cross-Path Feature Hybrid Enhanced State Space Model for Point Cloud Classification
RD-UIE: Relation-Driven State Space Modeling for Underwater Image Enhancement
Vision Mamba in Remote Sensing: A Comprehensive Survey of Techniques, Applications and Outlook
토론
이 논문과 연결된 포럼 글이 아직 없습니다.