Open-vocabulary object detection focusing on detecting novel categories guided by natural language. In this report, we propose Open-Vocabulary Light-Weighted Detection Transformer (OVLW-DETR), a deployment friendly open-vocabulary detector with strong performance and low latency. Building upon OVLW-DETR, we provide an end-to-end training recipe that transferring knowledge from vision-language model (VLM) to object detector with simple alignment. We align detector with the text encoder from VLM by replacing the fixed classification layer weights in detector with the class-name embeddings extracted from the text encoder. Without additional fusing module, OVLW-DETR is flexible and deployment friendly, making it easier to implement and modulate. improving the efficiency of interleaved attention computation. Experimental results demonstrate that the proposed approach is superior over existing real-time open-vocabulary detectors on standard Zero-Shot LVIS benchmark. Source code and pre-trained models are available at [https://github.com/Atten4Vis/LW-DETR].
Multi-object tracking (MOT) plays a pivotal role in various computer vision domains with recent tracking-by detection algorithms that treat MOT as distinct detection and association tasks. However, the existing trackers often rely on sensitive thresholds to associate previous tracks and current detection results while forming complete trajectories across a video. These thresholds are crucial for tracking performance and require manual tuning for each dataset or even sequence, limiting the adaptability in real-world applications. To tackle this problem, in this paper, we introduce AdapTrack, a novel MOT algorithm designed to enable adaption on varying scenarios without handcrafted threshold configuration. With a carefully designed matching strategy, our tracker can adaptively select proper thresholds for each frame and correctly associate detected objects. Consequently, AdapTrack shows outperforming results on standard MOT benchmarks, MOT17 and MOT20, compared to existing state-of-the-art methods. Every source code is available at https://github.com/kamkyu94/AdapTrack.
Model efficiency has become increasingly important in computer vision. In this paper, we systematically study neural network architecture design choices for object detection and propose several key optimizations to improve efficiency. First, we propose a weighted bi-directional feature pyramid network (BiFPN), which allows easy and fast multiscale feature fusion; Second, we propose a compound scaling method that uniformly scales the resolution, depth, and width for all backbone, feature network, and box/class prediction networks at the same time. Based on these optimizations and better backbones, we have developed a new family of object detectors, called EfficientDet, which consistently achieve much better efficiency than prior art across a wide spectrum of resource constraints. In particular, with single model and single-scale, our EfficientDet-D7 achieves state-of-the-art 55.1 AP on COCO test-dev with 77M parameters and 410B FLOPs, being 4x - 9x smaller and using 13x - 42x fewer FLOPs than previous detectors. Code is available at https://github.com/google/automl/tree/master/efficientdet.
3D sensors, also known as RGB-D sensors, utilize depth images where each pixel measures the distance from the camera to objects, using principles like structured light or time-of-flight. Advances in artificial vision have led to affordable 3D cameras capable of real-time object detection without object movement, surpassing 2D cameras in information depth. These cameras can identify objects of varying colors and reflectivities and are less affected by lighting changes. The described prototype uses RGB-D sensors for bidirectional people counting in venues, aiding security and surveillance in spaces like stadiums or airports. It determines real-time occupancy and checks against maximum capacity, crucial during emergencies. The system includes a RealSense D415 depth camera and a mini-computer running object detection algorithms to count people and a 2D camera for identity verification. The system supports statistical analysis and uses C++, Python, and PHP with OpenCV for image processing, demonstrating a comprehensive approach to monitoring venue occupancy.
Few-shot object detection (FSOD) identifies objects from extremely few annotated samples. Most existing FSOD methods, recently, apply the two-stage learning paradigm, which transfers the knowledge learned from abundant base classes to assist the few-shot detectors by learning the global features. However, such existing FSOD approaches seldom consider the localization of objects from local to global. Limited by the scarce training data in FSOD, the training samples of novel classes typically capture part of objects, resulting in such FSOD methods cannot detect the completely unseen object during testing. To tackle this problem, we propose an Extensible Co-Existing Attention (ECEA) module to enable the model to infer the global object according to the local parts. Essentially, the proposed module continuously learns the extensible ability on the base stage with abundant samples and transfers it to the novel stage, which can assist the few-shot model to quickly adapt in extending local regions to co-existing regions. Specifically, we first devise an extensible attention mechanism that starts with a local region and extends attention to co-existing regions that are similar and adjacent to the given local region. We then implement the extensible attention mechanism in different feature scales to progressively discover the full object in various receptive fields. Extensive experiments on the PASCAL VOC and COCO datasets show that our ECEA module can assist the few-shot detector to completely predict the object despite some regions failing to appear in the training samples and achieve the new state of the art compared with existing FSOD methods.
객체 탐지를 위한 도메인 일반화(DG)는 보지 못한 대상 도메인에서의 모델 성능을 향상시키는 것을 목표로 한다. 그러나 기존 탐지기는 시각적으로 크게 다른 미지의 도메인에서 상당한 성능 저하를 겪는데, 이는 제한된 학습 스타일에 갇힌 모델이 객체 수준의 의미론보다 표면적인 통계적 단서에 과적합되는 경향이 있기 때문이다. 우리는 의미적으로 관련된 특징에 주로 주목하고 무관한 스타일 단서는 무시하는 인간의 지각에서 착안하여, 다중 스타일 데이터 합성과 스타일 간 일관성 학습을 결합해 스타일에 민감한 지름길 학습을 억제하는 소스 전용 프레임워크인 Origin-Anchored Multi-Style Consistency(OAMSC)를 제안한다. 데이터 준비 단계에서는 기성 확산 모델을 활용하여 소스 데이터셋을 20가지의 서로 다른 스타일로 확장하며, 생성된 모든 이미지를 수작업으로 검증하여 스타일 변형 간에 객체 인스턴스와 주석이 일관되게 유지되도록 한다. 학습 중에는 장면 수준과 객체 수준에서 원본 기준(Origin-anchored) 일관성 제약을 부과하여, 스타일에 민감한 단서의 영향을 억제하면서 도메인에 안정적인 특징의 학습을 강화한다. 여러 도전적인 DG 벤치마크에서 수행한 광범위한 실험과 절제 연구는 다양한 도메인 변화에 걸쳐 OAMSC의 일관된 효과와 강력한 일반화 성능을 입증한다. 특히 우리의 접근법은 단일 단계 탐지기와 2단계 탐지기 모두에 적용되며, 데이터 중심 설계는 원칙적으로 다른 DG 방법과 상호 보완적이다. 학습 코드와 실험을 위해 구축한 두 개의 스타일 변환 데이터셋은 총 약 180K개의 이미지로 구성되며 GitHub에 공개되어 있다.
Advances in deep learning recognition have led to accurate object detection with 2D images. However, these 2D perception methods are insufficient for complete 3D world information. Concurrently, advanced 3D shape estimation approaches focus on the shape itself, without considering metric scale. These methods cannot determine the accurate location and orientation of objects. To tackle this problem, we propose a framework that jointly estimates a metric scale shape and pose from a single RGB image. Our framework has two branches: the Metric Scale Object Shape branch (MSOS) and the Normalized Object Coordinate Space branch (NOCS). The MSOS branch estimates the metric scale shape observed in the camera coordinates. The NOCS branch predicts the normalized object coordinate space (NOCS) map and performs similarity transformation with the rendered depth map from a predicted metric scale mesh to obtain 6d pose and size. Additionally, we introduce the Normalized Object Center Estimation (NOCE) to estimate the geometrically aligned distance from the camera to the object center. We validated our method on both synthetic and real-world datasets to evaluate category-level object pose and shape.
Recent progress on salient object detection (SOD) mainly benefits from multi-scale learning, where the high-level and low-level features collaborate in locating salient objects and discovering fine details, respectively. However, most efforts are devoted to low-level feature learning by fusing multi-scale features or enhancing boundary representations. High-level features, which although have long proven effective for many other tasks, yet have been barely studied for SOD. In this paper, we tap into this gap and show that enhancing high- level features is essential for SOD as well. To this end, we introduce an Extremely-Downsampled Network (EDN), which employs an extreme downsampling technique to effectively learn a global view of the whole image, leading to accurate salient object localization. To accomplish better multi-level feature fusion, we construct the Scale-Correlated Pyramid Convolution (SCPC) to build an elegant decoder for recovering object details from the above extreme downsampling. Extensive experiments demonstrate that EDN achieves state-of-the-art performance with real-time speed. Our efficient EDN-Lite also achieves competitive performance with a speed of 316fps. Hence, this work is expected to spark some new thinking in SOD. Code is available at https://github.com/yuhuan-wu/EDN.
In this paper, we propose the first self-distillation framework for general object detection, termed LGD (Label-Guided self-Distillation). Previous studies rely on a strong pretrained teacher to provide instructive knowledge that could be unavailable in real-world scenarios. Instead, we generate an instructive knowledge based only on student representations and regular labels. Our framework includes sparse label-appearance encoder, inter-object relation adapter and intra-object knowledge mapper that jointly form an implicit teacher at training phase, dynamically dependent on labels and evolving student representations. They are trained end-to-end with detector and discarded in inference. Experimentally, LGD obtains decent results on various detectors, datasets, and extensive tasks like instance segmentation. For example in MS-COCO dataset, LGD improves RetinaNet with ResNet-50 under 2x single-scale training from 36.2% to 39.0% mAP (+ 2.8%). It boosts much stronger detectors like FCOS with ResNeXt-101 DCN v2 under 2x multi-scale training from 46.1% to 47.9% (+ 1.8%). Compared with a classical teacher-based method FGFI, LGD not only performs better without requiring pretrained teacher but also reduces 51% training cost beyond inherent student learning. Codes are available at https://github.com/megvii-research/LGD.
3D object detection is one of the most important tasks in 3D vision perceptual system of autonomous vehicles. In this paper, we propose a novel two stage 3D object detection method aimed at get the optimal solution of object location in 3D space based on regressing two additional 3D object properties by a deep convolutional neural network and combined with cascaded geometric constraints between the 2D and 3D boxes. First, we modify the existing 3D properties regressing network by adding two additional components, viewpoints classification and the center projection of the 3D bounding box s bottom face. Second, we use the predicted center projection combined with similar triangle constraint to acquire an initial 3D bounding box by a closed-form solution. Then, the location predicted by previous step is used as the initial value of the over-determined equations constructed by 2D and 3D boxes fitting constraint with the configuration determined with the classified viewpoint. Finally, we use the recovered physical world information by the 3D detections to filter out the false detection and false alarm in 2D detections. We compare our method with the state-of-the-arts on the KITTI dataset show that although conceptually simple, our method outperforms more complex and computational expensive methods not only by improving the overall precision of 3D detections, but also increasing the orientation estimation precision. Furthermore our method can deal with the truncated objects to some extent and remove the false alarm and false detections in both 2D and 3D detections.
We present a novel approach to perform the unsupervised domain adaptation for object detection through forward-backward cyclic (FBC) training. Recent adversarial training based domain adaptation methods have shown their effectiveness on minimizing domain discrepancy via marginal feature distributions alignment. However, aligning the marginal feature distributions does not guarantee the alignment of class conditional distributions. This limitation is more evident when adapting object detectors as the domain discrepancy is larger compared to the image classification task, e.g. various number of objects exist in one image and the majority of content in an image is the background. This motivates us to learn domain invariance for category level semantics via gradient alignment. Intuitively, if the gradients of two domains point in similar directions, then the learning of one domain can improve that of another domain. To achieve gradient alignment, we propose Forward-Backward Cyclic Adaptation, which iteratively computes adaptation from source to target via backward hopping and from target to source via forward passing. In addition, we align low-level features for adapting holistic color/texture via adversarial training. However, the detector performs well on both domains is not ideal for target domain. As such, in each cycle, domain diversity is enforced by maximum entropy regularization on the source domain to penalize confident source-specific learning and minimum entropy regularization on target domain to intrigue target-specific learning. Theoretical analysis of the training process is provided, and extensive experiments on challenging cross-domain object detection datasets have shown the superiority of our approach over the state-of-the-art.
포도 잎 병해 인식은 조기 진단, 적시 개입, 포도원 관리 개선을 가능하게 하므로 정밀 농업에 중요하다. 심층 학습이 우수한 성능을 달성했지만, 많은 연구는 통제된 조건에서 수집된 소수의 데이터셋에 의존하며, 복잡한 배경, 가변 조명, 가림(occlusion), 잎 자세, 병 중증도, 기기 차이와 같은 실제 포도원의 어려움을 반영하지 못할 수 있다. 본 논문은 포도 잎 병해 분류 및 검출을 위한 데이터셋 중심 심층 학습 방법 벤치마크를 제시한다. 공개 데이터셋을 병 카테고리, 어노테이션 유형, 촬영 조건, 이미지 특성, 클래스 분포, 출처(provenance), 과제 적합성 측면에서 분석한다. 대표 모델을 이미지 수준 분류, 영역 수준 분류, 객체 검출의 세 가지 설정에서 평가한다. 분류는 정확도(accuracy)로, 검출은 mAP@50 및 mAP@50:95로 평가한다. 교차 데이터셋 실험은 호환되는 병 카테고리를 공유하지만 시각적·어노테이션 특성이 다른 데이터셋 간 전이(transfer)를 추가로 검토한다. 결과는 여러 통제되거나 파생(derivative) 데이터셋에서 분류 성능이 거의 포화에 가깝고, 이질적인 데이터셋에서는 난이도가 더 크며, 어노테이션 설정에 따라 검출 성능 변동이 크다는 것을 보여 준다. 교차 데이터셋 성능은 특히 객체 검출에서 급격히 하락하여, 공유된 병 라벨이 반드시 동등한 인식 과제를 정의하지는 않음을 시사한다. 본 벤치마크는 신뢰할 수 있는 포도원 병해 인식을 위해 데이터셋 출처, 현실적인 현장 평가, 어노테이션 호환성, 외부 검증을 강조한다.
Existing object detection models assume both the training and test data are sampled from the same source domain. This assumption does not hold true when these detectors are deployed in real-world applications, where they encounter new visual domain. Unsupervised Domain Adaptation (UDA) methods are generally employed to mitigate the adverse effects caused by domain shift. Existing UDA methods operate in an offline manner where the model is first adapted towards the target domain and then deployed in real-world applications. However, this offline adaptation strategy is not suitable for real-world applications as the model frequently encounters new domain shifts. Hence, it becomes critical to develop a feasible UDA method that generalizes to these domain shifts encountered during deployment time in a continuous online manner. To this end, we propose a novel unified adaptation framework that adapts and improves generalization on the target domain in online settings. In particular, we introduce MemXformer - a cross-attention transformer-based memory module where items in the memory take advantage of domain shifts and record prototypical patterns of the target distribution. Further, MemXformer produces strong positive and negative pairs to guide a novel contrastive loss, which enhances target specific representation learning. Experiments on diverse detection benchmarks show that the proposed strategy can produce state-of-the-art performance in both online and offline settings. To the best of our knowledge, this is the first work to address online and offline adaptation settings for object detection. Code at https://github.com/Vibashan/memXformer-online-da
객체 탐지기는 정상적인 촬영 조건에서는 강한 성능을 보이지만, 블러, 잡음, 압축, 악천후 또는 해상도 변화에 노출되면 겉으로 드러나지 않게 실패할 수 있다. 따라서 안전이 중요한 환경에서는 입력이 탐지기의 정상 동작 영역 안에 있는지를 평가하지 않고 예측만 내놓는 것은 충분하지 않으며, 우리는 이를 자기 인식 객체 탐지라고 부른다. 우리는 의미적 내용이 아니라 영상 열화에 따라 탐지기의 특징 공간을 구조화하는 열화 매니폴드에 기반한 열화 인식 프레임워크를 제안한다. 우리의 방법은 표준 탐지 백본에 다층 대조 학습으로 학습되는 경량 임베딩 헤드를 추가하며, 동일한 열화 조합을 공유하는 영상은 가깝게 모으고 서로 다른 구성은 분리한다. 기존의 대조 IQA 접근법과 달리, 우리의 방법은 탐지기와 정렬된 입력 충실도 모니터링을 수행한다. 즉, 독립된 표현에서 지각적 영상 품질을 추정하는 대신, 과제 백본 특징과 정상 프로토타입을 사용하여 열화로 인한 변화를 탐지한다. 표현을 고정하기 위해 깨끗한 학습 임베딩으로부터 원본 프로토타입을 추정하여 특징 공간에서 정상 동작점을 정의한다. 자기 인식은 이 기준으로부터의 기하학적 편차로 도출되며, 탐지 신뢰도와 무관하게 열화로 인한 변화에 대한 내재적인 영상 수준 신호를 제공한다. 합성 왜곡, 데이터셋 간 제로샷 전이, 자연 기상으로 인한 변화에 대한 실험은 원본과 열화 영상 간의 강한 분리 가능성, 탐지기 아키텍처 전반에 걸친 일관된 거동, 그리고 내용 차이가 큰 데이터셋 전반에 걸친 견고한 일반화를 보여 준다. 이러한 결과는 열화 인식 탐지기 표현이 입력 충실도 변화를 모니터링하는 신뢰할 만한 신호를 제공함을 시사한다.
In this paper, we propose a novel top-down instance segmentation framework based on explicit shape encoding, named \textbf{ESE-Seg}. It largely reduces the computational consumption of the instance segmentation by explicitly decoding the multiple object shapes with tensor operations, thus performs the instance segmentation at almost the same speed as the object detection. ESE-Seg is based on a novel shape signature Inner-center Radius (IR), Chebyshev polynomial fitting and the strong modern object detectors. ESE-Seg with YOLOv3 outperforms the Mask R-CNN on Pascal VOC 2012 at [email omitted] while 7 times faster.
오픈월드 객체 검출(open-world object detection)은 모델이 알려진 범주를 인식하고, 낯선 객체를 거부하며, 시간이 지남에 따라 새로운 클래스를 통합할 것을 요구한다. 이는 희귀 범주가 소수의 주석 예제만 가질 수 있고 세분화된(fine-grained) 클래스가 미묘한 형태학적 차이로 구분되는 생의학 및 과학 영상과 같은 데이터가 부족한 환경에서 특히 어렵다. 프로토타입 기반 검출기는 이러한 환경에 적합하지만, 일반적으로 클래스 프로토타입을 독립적인 앵커로 학습하여 클래스 간의 관계적 구조를 무시한다. 본 논문에서는 학습 데이터로부터 추정된 시각적 또는 의미적 클래스 간 비유사도를 보존하도록, 학습된 프로토타입 또는 클래스 표현 공간을 제약하는 일반 프레임워크인 클래스 기하 지도(class-geometry supervision, CGS)를 제안한다. CGS는 표준 태스크 손실을 유지하면서, 학습된 클래스 표현들 간의 쌍별 거리를 목표 클래스 기하 행렬과 정렬하는 비유사도 보존 목적 함수를 도입한다. 우리는 동일한 목적 함수를 프로토타입 인식, 퓨샷 생의학 객체 검출, 오픈셋 검출, 새로운 클래스 삽입, 그리고 COCO에서의 OWOD 적응에 걸쳐 구현한다. 실험 결과, CGS는 인식 및 ova 검출에서 표본 효율성을 향상시키고, 새로운 클래스 삽입을 크게 강화하며, 알려진 클래스 검출 성능의 상당 부분을 유지하면서 COCO에서 미지 클래스 재현율을 향상시키는 것으로 나타났다. 절제 실험(ablation)은 의미 있는 시각적 기하가 가장 안정적인 성능 향상을 제공하는 반면, 무작위 기하는 새로운 클래스 분리에는 도움이 될 수 있으나 퓨샷 검출에서는 일관성이 떨어짐을 보여준다. 이러한 결과는 관계적 클래스 기하가 제한된 지도 하에서 보정되고 확장 가능한 오픈월드 검출기를 구축하기 위한 효과적인 지도 신호임을 시사한다.
Human-Object Interaction (HOI) detection is important to human-centric scene understanding tasks. Existing works tend to assume that the same verb has similar visual characteristics in different HOI categories, an approach that ignores the diverse semantic meanings of the verb. To address this issue, in this paper, we propose a novel Polysemy Deciphering Network (PD-Net) that decodes the visual polysemy of verbs for HOI detection in three distinct ways. First, we refine features for HOI detection to be polysemyaware through the use of two novel modules: namely, Language Prior-guided Channel Attention (LPCA) and Language Prior-based Feature Augmentation (LPFA). LPCA highlights important elements in human and object appearance features for each HOI category to be identified; moreover, LPFA augments human pose and spatial features for HOI detection using language priors, enabling the verb classifiers to receive language hints that reduce intra-class variation for the same verb. Second, we introduce a novel Polysemy-Aware Modal Fusion module (PAMF), which guides PD-Net to make decisions based on feature types deemed more important according to the language priors. Third, we propose to relieve the verb polysemy problem through sharing verb classifiers for semantically similar HOI categories. Furthermore, to expedite research on the verb polysemy problem, we build a new benchmark dataset named HOI-VerbPolysemy (HOIVP), which includes common verbs (predicates) that have diverse semantic meanings in the real world. Finally, through deciphering the visual polysemy of verbs, our approach is demonstrated to outperform state-of-the-art methods by significant margins on the HICO-DET, V-COCO, and HOI-VP databases. Code and data in this paper are available at https://github.com/MuchHair/PD-Net.
Object-oriented SLAM is a popular technology in autonomous driving and robotics. In this paper, we propose a stereo visual SLAM with a robust quadric landmark representation method. The system consists of four components, including deep learning detection, object-oriented data association, dual quadric landmark initialization and object-based pose optimization. State-of-the-art quadric-based SLAM algorithms always face observation related problems and are sensitive to observation noise, which limits their application in outdoor scenes. To solve this problem, we propose a quadric initialization method based on the decoupling of the quadric parameters method, which improves the robustness to observation noise. The sufficient object data association algorithm and object-oriented optimization with multiple cues enables a highly accurate object pose estimation that is robust to local observations. Experimental results show that the proposed system is more robust to observation noise and significantly outperforms current state-of-the-art methods in outdoor environments. In addition, the proposed system demonstrates real-time performance.
Despite the impressive progress brought by deep network in visual object recognition, robot vision is still far from being a solved problem. The most successful convolutional architectures are developed starting from ImageNet, a large scale collection of images of object categories downloaded from the Web. This kind of images is very different from the situated and embodied visual experience of robots deployed in unconstrained settings. To reduce the gap between these two visual experiences, this paper proposes a simple yet effective data augmentation layer that zooms on the object of interest and simulates the object detection outcome of a robot vision system. The layer, that can be used with any convolutional deep architecture, brings to an increase in object recognition performance of up to 7\%, in experiments performed over three different benchmark databases. Upon acceptance of the paper, our robot data augmentation layer will be made publicly available.
Large Vision Language Models (VLMs), such as CLIP, have significantly contributed to various computer vision tasks, including object recognition and object detection. Their open vocabulary feature enhances their value. However, their black-box nature and lack of explainability in predictions make them less trustworthy in critical domains. Recently, some work has been done to force VLMs to provide reasonable rationales for object recognition, but this often comes at the expense of classification accuracy. In this paper, we first propose a mathematical definition of explainability in the object recognition task based on the joint probability distribution of categories and rationales, then leverage this definition to fine-tune CLIP in an explainable manner. Through evaluations of different datasets, our method demonstrates state-of-the-art performance in explainable classification. Notably, it excels in zero-shot settings, showcasing its adaptability. This advancement improves explainable object recognition, enhancing trust across diverse applications. The code will be made available online upon publication.