Parameter Efficient Mamba Tuning via Projector-targeted Diagonal-centric Linear Transformation
원문 초록
Despite the growing interest in Mamba architecture as a potential replacement for Transformer architecture, parameter-efficient fine-tuning (PEFT) approaches for Mamba remain largely unexplored. In our study, we introduce two key insights-driven strategies for PEFT in Mamba architecture: (1) While state-space models (SSMs) have been regarded as the cornerstone of Mamba architecture, then expected to play a primary role in transfer learning, our findings reveal that Projectors -- not SSMs -- are the predominant contributors to transfer learning, and (2) Based on our observation that adapting pretrained Projectors to new tasks can be effectively approximated through a near-diagonal linear transformation, we propose a novel PEFT method specialized to Mamba architecture: Projector-targeted Diagonal-centric Linear Transformation (ProDiaL). ProDiaL focuses on optimizing only diagonal-centric linear transformation matrices, without directly fine-tuning the pretrained Projector weights. This targeted approach allows efficient task adaptation, utilizing less than 1% of the total parameters, and exhibits strong performance across both vision and language Mamba models, highlighting its versatility and effectiveness.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Memba: Membrane-driven Parameter-Efficient Fine-Tuning for Mamba
MambaFusion: Height-Fidelity Dense Global Fusion for Multi-modal 3D Object Detection
Sparsified State-Space Models are Efficient Highway Networks
Adaptive Gate-Aware Mamba Networks for Magnetic Resonance Fingerprinting
EAGLE: An Efficient Global Attention Lesion Segmentation Model for Hepatic Echinococcosis
State-Space Models in Efficient Whispered and Multi-dialect Speech Recognition
토론
이 논문과 연결된 포럼 글이 아직 없습니다.