SSM2Mel: State Space Model to Reconstruct Mel Spectrogram from the EEG
원문 초록
Decoding speech from brain signals is a challenging research problem that holds significant importance for studying speech processing in the brain. Although breakthroughs have been made in reconstructing the mel spectrograms of audio stimuli perceived by subjects at the word or letter level using noninvasive electroencephalography (EEG), there is still a critical gap in precisely reconstructing continuous speech features, especially at the minute level. To address this issue, this paper proposes a State Space Model (SSM) to reconstruct the mel spectrogram of continuous speech from EEG, named SSM2Mel. This model introduces a novel Mamba module to effectively model the long sequence of EEG signals for imagined speech. In the SSM2Mel model, the S4-UNet structure is used to enhance the extraction of local features of EEG signals, and the Embedding Strength Modulator (ESM) module is used to incorporate subject-specific information. Experimental results show that our model achieves a Pearson correlation of 0.069 on the SparrKULee dataset, which is a 38% improvement over the previous baseline.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Spec2VolCAMU-Net: A Spectrogram-to-Volume Model for EEG-to-fMRI Reconstruction based on Multi-directional Time-Frequency Convolutional Attention Encoder and Vision-Mamba U-Net
LangMamba: A Language-driven Mamba Framework for Low-dose CT Denoising with Vision-language Models
Differential Mamba
eegFloss: A Python package for refining sleep EEG recordings using machine learning models
FindRec: Stein-Guided Entropic Flow for Multi-Modal Sequential Recommendation
MambaFusion: Height-Fidelity Dense Global Fusion for Multi-modal 3D Object Detection
토론
이 논문과 연결된 포럼 글이 아직 없습니다.