Most previous 3D object detection methods that leverage the multi-modality of LiDAR and cameras utilize the Bird's Eye View (BEV) space for intermediate feature representation. However, this space uses a low x, y-resolution and sacrifices z-axis information to reduce the overall feature resolution, which may result in declined accuracy. To tackle the problem of using low-resolution features, this paper focuses on the sparse nature of LiDAR point cloud data. From our observation, the number of occupied cells in the 3D voxels constructed from a LiDAR data can be even fewer than the number of total cells in the BEV map, despite the voxels' significantly higher resolution. Based on this, we introduce a novel sparse voxel-based transformer network for 3D object detection, dubbed as SparseVoxFormer. Instead of performing BEV feature extraction, we directly leverage sparse voxel features as the input for a transformer-based detector. Moreover, with regard to the camera modality, we introduce an explicit modality fusion approach that involves projecting 3D voxel coordinates onto 2D images and collecting the corresponding image features. Thanks to these components, our approach can leverage geometrically richer multi-modal features while even reducing the computational cost. Beyond the proof-of-concept level, we further focus on facilitating better multi-modal fusion and flexible control over the number of sparse features. Finally, thorough experimental results demonstrate that utilizing a significantly smaller number of sparse features drastically reduces computational costs in a 3D object detector while enhancing both overall and long-range performance.
단안 3D 객체 탐지는 고정된 범주 어휘 내에서 동작하는 폐쇄 집합 탐지기와, 3D 기하를 위해 깊이 파운데이션 모델을 활용하여 임의의 범주를 위치 추정하는 개방 어휘 탐지기라는 두 가지 체제에 걸쳐 있다. 우리는 현재의 깊이 파운데이션 모델이 강력한 제로샷 일반화 능력에도 불구하고 3D 탐지가 요구하는 객체 수준의 정밀도가 부족함을 발견했다. 강력한 탐지기가 예측한 깊이를 최첨단 깊이 파운데이션 모델로 대체하면 정확도가 저하된다. 탐지기나 깊이 모델을 종단간으로 더 정확하게 만드는 대신, 우리는 객체 수준 깊이 정제를 독립된 과제로 다루고, 수치 값을 전혀 예측하지 않고 깊이를 보정하는 비전-언어 모델인 RefineAny3D를 제시한다. 핵심 통찰은 깊이 오차가 영상 공간에서 직접적인 시각적 흔적을 갖는다는 것이다. 영상에 투영했을 때 올바르게 배치된 박스는 객체를 꼭 맞게 둘러싸는 반면, 너무 먼 박스는 지나치게 작게, 너무 가까운 박스는 지나치게 크게 투영된다. 따라서 깊이 정제는 지표 회귀 문제가 아니라 시각적 정렬 문제로 환원되며, 우리는 이를 VLM의 어휘를 수치 깊이 출력 대신 범주형 결정을 내리는 행동 토큰으로 확장하고, 각 결정을 명시적인 시각적 근거에 기반하게 하는 대규모 사고 연쇄 데이터셋으로 모델을 지도함으로써 구현한다. 단일 사후 단계로 적용되는 RefineAny3D는 폐쇄 집합 탐지기, 개방 어휘 탐지기, 3D 자동 레이블링 도구 전반에서 일관된 성능 향상을 제공하며, 재학습 없이 새로운 범주, 장면, 카메라로 일반화된다. 코드는 https://github.com/alanzhangcs/RefineAny3D 에서 이용할 수 있다.
As the rapid development of depth learning, object detection in aviatic remote sensing images has become increasingly popular in recent years. Most of the current Anchor Free detectors based on key point detection sampling directly regression and classification features, with the design of object loss function based on the horizontal bounding box. It is more challenging for complex and diverse aviatic remote sensing object. In this paper, we propose an Anchor Free aviatic remote sensing object detector (BWP-Det) to detect rotating and multi-scale object. Specifically, we design a interactive double-branch(IDB) up-sampling network, in which one branch gradually up-sampling is used for the prediction of Heatmap, and the other branch is used for the regression of boundary box parameters. We improve a weighted multi-scale convolution (WmConv) in order to highlight the difference between foreground and background. We extracted Pixel level attention features from the middle layer to guide the two branches to pay attention to effective object information in the sampling process. Finally, referring to the calculation idea of horizontal IoU, we design a rotating IoU based on the split polar coordinate plane, namely JIoU, which is expressed as the intersection ratio following discretization of the inner ellipse of the rotating bounding box, to solve the correlation between angle and side length in the regression process of the rotating bounding box. Ultimately, BWP-Det, our experiments on DOTA, UCAS-AOD and NWPU VHR-10 datasets show, achieves advanced performance with simpler models and fewer regression parameters.
3D Multi-Object Tracking (MOT) obtains significant performance improvements with the rapid advancements in 3D object detection, particularly in cost-effective multi-camera setups. However, the prevalent end-to-end training approach for multi-camera trackers results in detector-specific models, limiting their versatility. Moreover, current generic trackers overlook the unique features of multi-camera detectors, i.e., the unreliability of motion observations and the feasibility of visual information. To address these challenges, we propose RockTrack, a 3D MOT method for multi-camera detectors. Following the Tracking-By-Detection framework, RockTrack is compatible with various off-the-shelf detectors. RockTrack incorporates a confidence-guided preprocessing module to extract reliable motion and image observations from distinct representation spaces from a single detector. These observations are then fused in an association module that leverages geometric and appearance cues to minimize mismatches. The resulting matches are propagated through a staged estimation process, forming the basis for heuristic noise modeling. Additionally, we introduce a novel appearance similarity metric for explicitly characterizing object affinities in multi-camera settings. RockTrack achieves state-of-the-art performance on the nuScenes vision-only tracking leaderboard with 59.1% AMOTA while demonstrating impressive computational efficiency.
Vision-language models (VLMs) are prone to object hallucinations, where they erroneously indicate the presenceof certain objects in an image. Existing benchmarks quantify hallucinations using relatively small, labeled datasets. However, this approach is i) insufficient to assess hallucinations that arise in open-world settings, where VLMs are widely used, and ii) inadequate for detecting systematic errors in VLMs. We propose DASH (Detection and Assessment of Systematic Hallucinations), an automatic, large-scale pipeline designed to identify systematic hallucinations of VLMs on real-world images in an open-world setting. A key component is DASH-OPT for image-based retrieval, where we optimize over the ''natural image manifold'' to generate images that mislead the VLM. The output of DASH consists of clusters of real and semantically similar images for which the VLM hallucinates an object. We apply DASH to PaliGemma and two LLaVA-NeXT models across 380 object classes and, in total, find more than 19k clusters with 950k images. We study the transfer of the identified systematic hallucinations to other VLMs and show that fine-tuning PaliGemma with the model-specific images obtained with DASH mitigates object hallucinations. Code and data are available at https://YanNeu.github.io/DASH.
도구를 사용하거나 직접적으로 상호작용하는 손과 객체를 이해하는 것은 행동 인식부터 3D 재구성 및 로봇공학에 이르는 작업들의 핵심 단계입니다. 본 논문은 손-객체 상호작용(HOI) 이해 연구 분야에 다음과 같은 여러 기여를 제공합니다: (1) Co-DETR 아키텍처에 손-객체 및 객체-객체 상호작용을 도입하여 최첨단 방법론을 구현한 새로운 프레임워크인 HOI-DETR; (2) HD-EPIC 데이터셋에서 파생된 비디오 벤치마크와 Hands23 벤치마크를 개선하는 새로운 주석을 포함하여 4가지 다양한 데이터셋으로 구성된 포괄적인 HOI 평가 스위트, 그리고 (3) Hands23과 FineBio에서 20퍼센트 포인트 이상의 mAP 향상을 포함하여 Hands23, HOIST, FineBio 및 HD-EPIC 전반에서 최첨단 성능을 크게 향상시키는 학습된 체크포인트입니다. 우리의 소거 연구는 각 모델 구성 요소의 기여를 확인해 줍니다.
The existing state-of-the-art (SOTA) video salient object detection (VSOD) models have widely followed short-term methodology, which dynamically determines the balance between spatial and temporal saliency fusion by solely considering the current consecutive limited frames. However, the short-term methodology has one critical limitation, which conflicts with the real mechanism of our visual system -- a typical long-term methodology. As a result, failure cases keep showing up in the results of the current SOTA models, and the short-term methodology becomes the major technical bottleneck. To solve this problem, this paper proposes a novel VSOD approach, which performs VSOD in a complete long-term way. Our approach converts the sequential VSOD, a sequential task, to a data mining problem, i.e., decomposing the input video sequence to object proposals in advance and then mining salient object proposals as much as possible in an easy-to-hard way. Since all object proposals are simultaneously available, the proposed approach is a complete long-term approach, which can alleviate some difficulties rooted in conventional short-term approaches. In addition, we devised an online updating scheme that can grasp the most representative and trustworthy pattern profile of the salient objects, outputting framewise saliency maps with rich details and smoothing both spatially and temporally. The proposed approach outperforms almost all SOTA models on five widely used benchmark datasets.
Traditional object recognition approaches apply feature extraction, part deformation handling, occlusion handling and classification sequentially while they are independent from each other. Ouyang and Wang proposed a model for jointly learning of all of the mentioned processes using one deep neural network. We utilized, and manipulated their toolbox in order to apply it in car detection scenarios where it had not been tested. Creating a single deep architecture from these components, improves the interaction between them and can enhance the performance of the whole system. We believe that the approach can be used as a general purpose object detection toolbox. We tested the algorithm on UIUC car dataset, and achieved an outstanding result. The accuracy of our method was 97 % while the previously reported results showed an accuracy of up to 91 %. We strongly believe that having an experiment on a larger dataset can show the advantage of using deep models over shallow ones.
Conventional detection networks usually need abundant labeled training samples, while humans can learn new concepts incrementally with just a few examples. This paper focuses on a more challenging but realistic class-incremental few-shot object detection problem (iFSD). It aims to incrementally transfer the model for novel objects from only a few annotated samples without catastrophically forgetting the previously learned ones. To tackle this problem, we propose a novel method LEAST, which can transfer with Less forgetting, fEwer training resources, And Stronger Transfer capability. Specifically, we first present the transfer strategy to reduce unnecessary weight adaptation and improve the transfer capability for iFSD. On this basis, we then integrate the knowledge distillation technique using a less resource-consuming approach to alleviate forgetting and propose a novel clustering-based exemplar selection process to preserve more discriminative features previously learned. Being a generic and effective method, LEAST can largely improve the iFSD performance on various benchmarks.
Neural network quantization aims to accelerate and trim full-precision neural network models by using low bit approximations. Methods adopting the quantization aware training (QAT) paradigm have recently seen a rapid growth, but are often conceptually complicated. This paper proposes a novel and highly effective QAT method, quantized feature distillation (QFD). QFD first trains a quantized (or binarized) representation as the teacher, then quantize the network using knowledge distillation (KD). Quantitative results show that QFD is more flexible and effective (i.e., quantization friendly) than previous quantization methods. QFD surpasses existing methods by a noticeable margin on not only image classification but also object detection, albeit being much simpler. Furthermore, QFD quantizes ViT and Swin-Transformer on MS-COCO detection and segmentation, which verifies its potential in real world deployment. To the best of our knowledge, this is the first time that vision transformers have been quantized in object detection and image segmentation tasks.
End-to-end production of object tracklets from high resolution video in real-time and with high accuracy remains a challenging problem due to the cost of object detection on each frame. In this work we present Localization-based Tracking (LBT), an extension to any tracker that follows the tracking by detection or joint detection and tracking paradigms. Localization-based Tracking focuses only on regions likely to contain objects to boost detection speed and avoid matching errors. We evaluate LBT as an extension to two example trackers (KIOU and SORT) on the UA-DETRAC and MOT20 datasets. LBT-extended trackers outperform all other reported algorithms in terms of PR-MOTA, PR-MOTP, and mostly tracked objects on the UA-DETRAC benchmark, establishing a new state-of-the art. relative to tracking by detection with KIOU, LBT-extended KIOU achieves a 25% higher frame-rate and is 1.1% more accurate in terms of PR-MOTA on the UA-DETRAC dataset. LBT-extended SORT achieves a 62% speedup and a 3.2% increase in PR-MOTA on the UA-DETRAC dataset. On MOT20, LBT-extended KIOU has a 50% higher frame-rate than tracking by detection and is 0.4% more accurate in terms of MOTA. As of submission time, our LBT-extended KIOU tracker places 10th overall on the MOT20 benchmark.
Modern driver assistance systems rely on a wide range of sensors (RADAR, LIDAR, ultrasound and cameras) for scene understanding and prediction. These sensors are typically used for detecting traffic participants and scene elements required for navigation. In this paper we argue that relying on camera based systems, specifically Around View Monitoring (AVM) system has great potential to achieve these goals in both parking and driving modes with decreased costs. The contributions of this paper are as follows: we present a new end-to-end solution for delimiting the safe drivable area for each frame by means of identifying the closest obstacle in each direction from the driving vehicle, we use this approach to calculate the distance to the nearest obstacles and we incorporate it into a unified end-to-end architecture capable of joint object detection, curb detection and safe drivable area detection. Furthermore, we describe the family of networks for both a high accuracy solution and a low complexity solution. We also introduce further augmentation of the base architecture with 3D object detection.
To avoid the exhaustive search over locations and scales, current state-of-the-art object detection systems usually involve a crucial component generating a batch of candidate object proposals from images. In this paper, we present a simple yet effective approach for segmenting object proposals via a deep architecture of recursive neural networks (ReNNs), which hierarchically groups regions for detecting object candidates over scales. Unlike traditional methods that mainly adopt fixed similarity measures for merging regions or finding object proposals, our approach adaptively learns the region merging similarity and the objectness measure during the process of hierarchical region grouping. Specifically, guided by a structured loss, the ReNN model jointly optimizes the cross-region similarity metric with the region merging process as well as the objectness prediction. During inference of the object proposal generation, we introduce randomness into the greedy search to cope with the ambiguity of grouping regions. Extensive experiments on standard benchmarks, e.g., PASCAL VOC and ImageNet, suggest that our approach is capable of producing object proposals with high recall while well preserving the object boundaries and outperforms other existing methods in both accuracy and efficiency.
Autonomous driving systems require huge amounts of data to train. Manual annotation of this data is time-consuming and prohibitively expensive since it involves human resources. Therefore, active learning emerged as an alternative to ease this effort and to make data annotation more manageable. In this paper, we introduce a novel active learning approach for object detection in videos by exploiting temporal coherence. Our active learning criterion is based on the estimated number of errors in terms of false positives and false negatives. The detections obtained by the object detector are used to define the nodes of a graph and tracked forward and backward to temporally link the nodes. Minimizing an energy function defined on this graphical model provides estimates of both false positives and false negatives. Additionally, we introduce a synthetic video dataset, called SYNTHIA-AL, specially designed to evaluate active learning for video object detection in road scenes. Finally, we show that our approach outperforms active learning baselines tested on two datasets.
State-of-the-art multi-object tracking~(MOT) methods follow the tracking-by-detection paradigm, where object trajectories are obtained by associating per-frame outputs of object detectors. In crowded scenes, however, detectors often fail to obtain accurate detections due to heavy occlusions and high crowd density. In this paper, we propose a new MOT paradigm, tracking-by-counting, tailored for crowded scenes. Using crowd density maps, we jointly model detection, counting, and tracking of multiple targets as a network flow program, which simultaneously finds the global optimal detections and trajectories of multiple targets over the whole video. This is in contrast to prior MOT methods that either ignore the crowd density and thus are prone to errors in crowded scenes, or rely on a suboptimal two-step process using heuristic density-aware point-tracks for matching targets.Our approach yields promising results on public benchmarks of various domains including people tracking, cell tracking, and fish tracking.
We consider the problem of referring camouflaged object detection (Ref-COD), a new task that aims to segment specified camouflaged objects based on a small set of referring images with salient target objects. We first assemble a large-scale dataset, called R2C7K, which consists of 7K images covering 64 object categories in real-world scenarios. Then, we develop a simple but strong dual-branch framework, dubbed R2CNet, with a reference branch embedding the common representations of target objects from referring images and a segmentation branch identifying and segmenting camouflaged objects under the guidance of the common representations. In particular, we design a Referring Mask Generation module to generate pixel-level prior mask and a Referring Feature Enrichment module to enhance the capability of identifying specified camouflaged objects. Extensive experiments show the superiority of our Ref-COD methods over their COD counterparts in segmenting specified camouflaged objects and identifying the main body of target objects. Our code and dataset are publicly available at https://github.com/zhangxuying1004/RefCOD.
장기 객체 탐색(long-term object search)은 변화하는 환경을 반복적이지만 불균일하게 관측하면서 객체가 주로 어디에 나타나는지 학습해야 합니다. 본 논문에서는 회고적 개방형 어휘 메모리(retrospective open-vocabulary memory)를 중도 절단된 관측값(censored observations)으로부터의 확률적 추론(probabilistic inference) 문제로 정의하며, 핵심 아이디어는 기회당 증거(evidence per opportunity)를 바탕으로 추론하는 것입니다. 즉, 탐지 여부는 로봇이 해당 위치를 관측할 기회에 비례해서만 신념(belief)에 영향을 주어야 합니다. 본 연구에서는 이 원리를 활용하여 질의 시점에만 지정되는 개념에 대한 장기 유병률(long-term prevalence)을 추정하고, 도출된 신념을 능동적 탐색 사전 확률(active-search prior)로 직접 변환하는 ECROM을 제안합니다. 이 문제를 평가하기 위해 반복적인 탐색 경로 전반에 걸쳐 객체 배치와 관측 기회를 독립적으로 변화시키는 10개 HM3D 주택 환경 기반의 통제된 장기 벤치마크(benchmark)를 도입합니다. ECROM은 보류된(held-out) 질의에 대해 각 지표별 가장 강력한 경쟁 메모리 대비 지지 수준 AP(support-level AP)를 4.5점, 탐색 SPL을 4.2점 향상시킵니다. 벤치마크, 데이터셋, 코드는 오픈소스로 공개될 예정입니다. 프로젝트 페이지: https://jiaming.im/ecrom/
단일 기준 이미지로부터 복잡한 장면 속의 특정 객체 인스턴스를 찾고, 그것이 없을 때 없다고 말하는 일은 보통 대규모 시각-언어 모델에 맡겨진다. 본 연구는 세계 모델의 표현이 가장 강력한 자기지도 이미지 특징에 없는 무언가를 더해 주는지 묻는다. 본 연구는 단일 기준 이미지로부터 복잡한 실제 장면에서 특정 객체 인스턴스를 찾는 원샷 탐지기 WALDO를 소개한다. 인스턴스 탐지는 보통 객체별 학습을 필요로 해 왔다. 대안은 범주 수준의 개방형 어휘 탐지기나 시각-언어 모델이지만, 이들은 머그잔은 찾을 수 있어도 바로 이 머그잔은 찾지 못한다. 본 연구는 고정된 자기지도 특징 위의 경량 조건부 헤드만으로도 객체별 적응 없이 보지 못한 인스턴스를 위치 추정하기에 충분함을 보인다. 또한 이미지 특징과 세계 모델 특징이 상호 보완적임을 발견하였다. DINOv3와 비디오 세계 모델 V-JEPA 2.1은 서로 다른 질의에서 성공한다. 한 가지 문제가 있다. 비디오 모델을 정지 이미지에 적용하는 통상적인 방식으로 사용하면 V-JEPA의 밀집 특징은 성능이 떨어진다. 모델의 이미지 경로를 통해 읽으면 위치 추정이 개선된다. 검증 데이터로 조정한 후기 융합으로 두 백본을 결합하면, 객체가 장면에 있는지 판별하는 DINOv3의 능력을 해치지 않으면서 보지 못한 인스턴스에 대한 DINOv3의 위치 추정이 소폭 향상된다. 강력한 시각-언어 모델과 이미지 유도 개방형 어휘 탐지기는 특정 인스턴스를 거의 위치 추정하지 못하며, 질의당 훨씬 오랜 시간이 걸린다. 세계 모델 특징과의 융합은 이미지 특징만 사용할 때보다 위치 추정을 강화한다.
This paper introduces an approach to produce accurate 3D detection boxes for objects on the ground using single monocular images. We do so by merging 2D visual cues, 3D object dimensions, and ground plane constraints to produce boxes that are robust against small errors and incorrect predictions. First, we train a single-shot convolutional neural network (CNN) that produces multiple visual and geometric cues of interest: 2D bounding boxes, 2D keypoints of interest, coarse object orientations and object dimensions. Subsets of these cues are then used to poll probable ground planes from a pre-computed database of ground planes, to identify the "best fit" plane with highest consensus. Once identified, the "best fit" plane provides enough constraints to successfully construct the desired 3D detection box, without directly predicting the 6DoF pose of the object. The entire ground plane polling (GPP) procedure is constructed as a non-parametrized layer of the CNN that outputs the desired "best fit" plane and the corresponding 3D keypoints, which together define the final 3D bounding box. Doing so allows us to poll thousands of different ground plane configurations without adding considerable overhead, while also creating a single CNN that directly produces the desired output without the need for post processing. We evaluate our method on the 2D detection and orientation estimation benchmark from the challenging KITTI dataset, and provide additional comparisons for 3D metrics of importance. This single-stage, single-pass CNN results in superior localization and orientation estimation compared to more complex and computationally expensive monocular approaches.
Current Zero-Shot Learning (ZSL) approaches are restricted to recognition of a single dominant unseen object category in a test image. We hypothesize that this setting is ill-suited for real-world applications where unseen objects appear only as a part of a complex scene, warranting both the `recognition' and `localization' of an unseen category. To address this limitation, we introduce a new \emph{`Zero-Shot Detection'} (ZSD) problem setting, which aims at simultaneously recognizing and locating object instances belonging to novel categories without any training examples. We also propose a new experimental protocol for ZSD based on the highly challenging ILSVRC dataset, adhering to practical issues, e.g., the rarity of unseen objects. To the best of our knowledge, this is the first end-to-end deep network for ZSD that jointly models the interplay between visual and semantic domain information. To overcome the noise in the automatically derived semantic descriptions, we utilize the concept of meta-classes to design an original loss function that achieves synergy between max-margin class separation and semantic space clustering. Furthermore, we present a baseline approach extended from recognition to detection setting. Our extensive experiments show significant performance boost over the baseline on the imperative yet difficult ZSD problem.