Fixed-Point RNNs: From Diagonal to Dense in a Few Iterations
원문 초록
Linear recurrent neural networks (RNNs) and state-space models (SSMs) such as Mamba have become promising alternatives to softmax-attention as sequence mixing layers in Transformer architectures. Current models, however, do not exhibit the full state-tracking expressivity of RNNs because they rely on channel-wise (i.e. diagonal) sequence mixing. In this paper, we propose to compute a dense linear RNN as the fixed-point of a parallelizable diagonal linear RNN in a single layer. We explore mechanisms to improve its memory and state-tracking abilities in practice, and achieve state-of-the-art results on the commonly used toy tasks $A_5$, $S_5$, copying, and modular arithmetics. We hope our results will open new avenues to more expressive and efficient sequence mixers.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
MambaFusion: Height-Fidelity Dense Global Fusion for Multi-modal 3D Object Detection
Sparsified State-Space Models are Efficient Highway Networks
Hierarchical Mamba Meets Hyperbolic Geometry: A New Paradigm for Structured Language Embeddings
Hybrid-Emba3D: Geometry-Aware and Cross-Path Feature Hybrid Enhanced State Space Model for Point Cloud Classification
RD-UIE: Relation-Driven State Space Modeling for Underwater Image Enhancement
Vision Mamba in Remote Sensing: A Comprehensive Survey of Techniques, Applications and Outlook
토론
이 논문과 연결된 포럼 글이 아직 없습니다.