Any2Any: Incomplete Multimodal Retrieval with Conformal Prediction
원문 초록
Autonomous agents perceive and interpret their surroundings by integrating multimodal inputs, such as vision, audio, and LiDAR. These perceptual modalities support retrieval tasks, such as place recognition in robotics. However, current multimodal retrieval systems encounter difficulties when parts of the data are missing due to sensor failures or inaccessibility, such as silent videos or LiDAR scans lacking RGB information. We propose Any2Any-a novel retrieval framework that addresses scenarios where both query and reference instances have incomplete modalities. Unlike previous methods limited to the imputation of two modalities, Any2Any handles any number of modalities without training generative models. It calculates pairwise similarities with cross-modal encoders and employs a two-stage calibration process with conformal prediction to align the similarities. Any2Any enables effective retrieval across multimodal datasets, e.g., text-LiDAR and text-time series. It achieves a Recall@5 of 35% on the KITTI dataset, which is on par with baseline models with complete modalities.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Response Quality Assessment for Retrieval-Augmented Generation via Conditional Conformal Factuality
Trustworthy Prediction with Gaussian Process Knowledge Scores
Foundation models for time series forecasting: Application in conformal prediction
Graph-Structured Feedback Multimodel Ensemble Online Conformal Prediction
Valid Selection among Conformal Sets
When Can We Reuse a Calibration Set for Multiple Conformal Predictions?
토론
이 논문과 연결된 포럼 글이 아직 없습니다.