Beyond Bounding Box: Multimodal Knowledge Learning for Object Detection
원문 초록
Multimodal supervision has achieved promising results in many visual language understanding tasks, where the language plays an essential role as a hint or context for recognizing and locating instances. However, due to the defects of the human-annotated language corpus, multimodal supervision remains unexplored in fully supervised object detection scenarios. In this paper, we take advantage of language prompt to introduce effective and unbiased linguistic supervision into object detection, and propose a new mechanism called multimodal knowledge learning (\textbf{MKL}), which is required to learn knowledge from language supervision. Specifically, we design prompts and fill them with the bounding box annotations to generate descriptions containing extensive hints and context for instances recognition and localization. The knowledge from language is then distilled into the detection model via maximizing cross-modal mutual information in both image- and object-level. Moreover, the generated descriptions are manipulated to produce hard negatives to further boost the detector performance. Extensive experiments demonstrate that the proposed method yields a consistent performance gain by 1.6\% $\sim$ 2.1\% and achieves state-of-the-art on MS-COCO and OpenImages datasets.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Towards Reliable Detection of Empty Space: Conditional Marked Point Processes for Object Detection
Multiple Object Stitching for Unsupervised Representation Learning
Domain Randomization for Object Detection in Manufacturing Applications using Synthetic Data: A Comprehensive Study
Edge-Enabled Collaborative Object Detection for Real-Time Multi-Vehicle Perception
Multispectral Detection Transformer with Infrared-Centric Sensor Fusion
A Real-Time System for Egocentric Hand-Object Interaction Detection in Industrial Domains
토론
이 논문과 연결된 포럼 글이 아직 없습니다.