Uncertainty-aware Multi-modal Learning via Cross-modal Random Network Prediction
원문 초록
Multi-modal learning focuses on training models by equally combining multiple input data modalities during the prediction process. However, this equal combination can be detrimental to the prediction accuracy because different modalities are usually accompanied by varying levels of uncertainty. Using such uncertainty to combine modalities has been studied by a couple of approaches, but with limited success because these approaches are either designed to deal with specific classification or segmentation problems and cannot be easily translated into other tasks, or suffer from numerical instabilities. In this paper, we propose a new Uncertainty-aware Multi-modal Learner that estimates uncertainty by measuring feature density via Cross-modal Random Network Prediction (CRNP). CRNP is designed to require little adaptation to translate between different prediction tasks, while having a stable training process. From a technical point of view, CRNP is the first approach to explore random network prediction to estimate uncertainty and to combine multi-modal data. Experiments on two 3D multi-modal medical image segmentation tasks and three 2D multi-modal computer vision classification tasks show the effectiveness, adaptability and robustness of CRNP. Also, we provide an extensive discussion on different fusion functions and visualization to validate the proposed model.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
MetaFormer and CNN Hybrid Model for Polyp Image Segmentation
LSegDiff: A Latent Diffusion Model for Medical Image Segmentation
DEU-Net: Dual-Encoder U-Net for Automated Skin Lesion Segmentation
Rethinking Dual-Stream Super-Resolution Semantic Learning in Medical Image Segmentation
PLN: Parasitic-Like Network for Barely Supervised Medical Image Segmentation
Crosslink-Net: Double-Branch Encoder Network via Fusing Vertical and Horizontal Convolutions for Medical Image Segmentation
토론
이 논문과 연결된 포럼 글이 아직 없습니다.