RespLLM: Unifying Audio and Text with Multimodal LLMs for Generalized Respiratory Health Prediction
원문 초록
The high incidence and mortality rates associated with respiratory diseases underscores the importance of early screening. Machine learning models can automate clinical consultations and auscultation, offering vital support in this area. However, the data involved, spanning demographics, medical history, symptoms, and respiratory audio, are heterogeneous and complex. Existing approaches are insufficient and lack generalizability, as they typically rely on limited training data, basic fusion techniques, and task-specific models. In this paper, we propose RespLLM, a novel multimodal large language model (LLM) framework that unifies text and audio representations for respiratory health prediction. RespLLM leverages the extensive prior knowledge of pretrained LLMs and enables effective audio-text fusion through cross-modal attentions. Instruction tuning is employed to integrate diverse data from multiple sources, ensuring generalizability and versatility of the model. Experiments on five real-world datasets demonstrate that RespLLM outperforms leading baselines by an average of 4.6% on trained tasks, 7.9% on unseen datasets, and facilitates zero-shot predictions for new tasks. Our work lays the foundation for multimodal models that can perceive, listen to, and understand heterogeneous data, paving the way for scalable respiratory health diagnosis.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
ShareGPT-4o-Image: Aligning Multimodal Models with GPT-4o-Level Image Generation
VIS-Shepherd: Constructing Critic for LLM-based Data Visualization Generation
KptLLM++: Towards Generic Keypoint Comprehension with Large Language Model
TalkFashion: Intelligent Virtual Try-On Assistant Based on Multimodal Large Language Model
ASCD: Attention-Steerable Contrastive Decoding for Reducing Hallucination in MLLM
CFBenchmark-MM: Chinese Financial Assistant Benchmark for Multimodal Large Language Model
토론
이 논문과 연결된 포럼 글이 아직 없습니다.