슬라이싱 보조 하이퍼 추론(SAHI)은 고해상도 영상에서 소형 객체 검출을 개선하지만, 종종 배경 타일에 상당한 연산을 소비한다. 본 연구는 관심 영역(ROI)-게이트 SAHI를 제안한다. 이는 추론 시점 프레임워크로, 경량 proposer를 도입해 전경 영역을 국소화하고 슬라이스 기반 정제를 정보가 있는 영역으로만 제한한다. 프레임워크를 두 가지 설정에서 평가한다. 128장 영상으로 구성된 COCO128 full split 데이터셋에서 정적 ROI 게이팅은 평균적으로 Full SAHI보다 느리며 속도 비율 0.88을 기록하고, [email omitted] Full SAHI의 0.7569에 비해 0.6602로 더 낮다. τ=0.4의 단순 적응형 라우팅 정책은 평균 지연을 줄여 Full SAHI 대비 1.02배의 소폭 이득을 달성한다. 3장 영상 sparse-to-dense 사례 연구에서는 ROI 게이팅이 0.96배에서 6.90배까지의 속도 향상을 보이며 평균 3.41배이다. 이 결과는 ROI 게이팅이 sparse 장면에서 가장 유리하고, 견고한 평균 동작을 위해서는 정책 기반 라우팅이 필요함을 보여준다.
Recent advances in self-supervised visual representation learning have paved the way for unsupervised methods tackling tasks such as object discovery and instance segmentation. However, discovering objects in an image with no supervision is a very hard task; what are the desired objects, when to separate them into parts, how many are there, and of what classes? The answers to these questions depend on the tasks and datasets of evaluation. In this work, we take a different approach and propose to look for the background instead. This way, the salient objects emerge as a by-product without any strong assumption on what an object should be. We propose FOUND, a simple model made of a single $conv1\times1$ initialized with coarse background masks extracted from self-supervised patch-based representations. After fast training and refining these seed masks, the model reaches state-of-the-art results on unsupervised saliency detection and object discovery benchmarks. Moreover, we show that our approach yields good results in the unsupervised semantic segmentation retrieval task. The code to reproduce our results is available at https://github.com/valeoai/FOUND.
Transfer learning based approaches have recently achieved promising results on the few-shot detection task. These approaches however suffer from ``catastrophic forgetting'' issue due to finetuning of base detector, leading to sub-optimal performance on the base classes. Furthermore, the slow convergence rate of stochastic gradient descent (SGD) results in high latency and consequently restricts real-time applications. We tackle the aforementioned issues in this work. We pose few-shot detection as a hierarchical learning problem, where the novel classes are treated as the child classes of existing base classes and the background class. The detection heads for the novel classes are then trained using a specialized optimization strategy, leading to significantly lower training times compared to SGD. Our approach obtains competitive novel class performance on few-shot MS-COCO benchmark, while completely retaining the performance of the initial model on the base classes. We further demonstrate the application of our approach to a new class-refined few-shot detection task.
The purpose of this study is to successfully train our vehicle detector using R-CNN, Faster R-CNN deep learning methods on a sample vehicle data sets and to optimize the success rate of the trained detector by providing efficient results for vehicle detection by testing the trained vehicle detector on the test data. The working method consists of six main stages. These are respectively; loading the data set, the design of the convolutional neural network, configuration of training options, training of the Faster R-CNN object detector and evaluation of trained detector. In addition, in the scope of the study, Faster R-CNN, R-CNN deep learning methods were mentioned and experimental analysis comparisons were made with the results obtained from vehicle detection.
We perform fast vehicle detection from traffic surveillance cameras. A novel deep learning framework, namely Evolving Boxes, is developed that proposes and refines the object boxes under different feature representations. Specifically, our framework is embedded with a light-weight proposal network to generate initial anchor boxes as well as to early discard unlikely regions; a fine-turning network produces detailed features for these candidate boxes. We show intriguingly that by applying different feature fusion techniques, the initial boxes can be refined for both localization and recognition. We evaluate our network on the recent DETRAC benchmark and obtain a significant improvement over the state-of-the-art Faster RCNN by 9.5% mAP. Further, our network achieves 9-13 FPS detection speed on a moderate commercial GPU.
With the recent development of Semi-Supervised Object Detection (SS-OD) techniques, object detectors can be improved by using a limited amount of labeled data and abundant unlabeled data. However, there are still two challenges that are not addressed: (1) there is no prior SS-OD work on anchor-free detectors, and (2) prior works are ineffective when pseudo-labeling bounding box regression. In this paper, we present Unbiased Teacher v2, which shows the generalization of SS-OD method to anchor-free detectors and also introduces Listen2Student mechanism for the unsupervised regression loss. Specifically, we first present a study examining the effectiveness of existing SS-OD methods on anchor-free detectors and find that they achieve much lower performance improvements under the semi-supervised setting. We also observe that box selection with centerness and the localization-based labeling used in anchor-free detectors cannot work well under the semi-supervised setting. On the other hand, our Listen2Student mechanism explicitly prevents misleading pseudo-labels in the training of bounding box regression; we specifically develop a novel pseudo-labeling selection mechanism based on the Teacher and Student's relative uncertainties. This idea contributes to favorable improvement in the regression branch in the semi-supervised setting. Our method, which works for both anchor-free and anchor-based methods, consistently performs favorably against the state-of-the-art methods in VOC, COCO-standard, and COCO-additional.
정확한 3D 검출은 이동형 체화 에이전트(embodied agent)에게 필수적이며, 비전 파운데이션 모델(VFM)은 전이 가능한 시각적 및 기하학적 사전 지식(prior)을 제공합니다. 그러나 기존의 VFM 기반 3D 검출기는 좁은 시야의 단안 영상이나 불연속적인 원근 뷰에 의존하므로 일관된 전방위 인식이 제한됩니다. 반면 정거원통도법(ERP)은 단일 이미지에 연속적인 360도 장면을 인코딩합니다. 하지만 ERP는 기하학 및 시각 정보를 다르게 구성하여 객체 관련 단서를 모델링, 위치 파악 및 보존하기 어렵게 만들기 때문에 직접적인 전이는 여전히 어렵습니다. 본 논문에서는 전이 가능한 VFM 사전 지식을 보존하면서 원근 뷰로 학습된 VFM 검출기를 ERP에 적응시키는 프레임워크인 OmniAct3D를 제안합니다. 기하학적 불일치를 해결하기 위해 ERP-Ray 기하 어댑터(ERGA-Ray)는 구면 시선(viewing ray)과 주기적 공간 구조를 모델링합니다. 전체 장면 맥락에서 증거를 국소화하기 위해 시각-행동 추론 체인(VARC)은 각 가설을 관련 파노라마 증거에 그라운딩(grounding)하고 이를 구조화된 기하학적 행동으로 변환합니다. 고정된 토큰(token) 예산 하에서 손실된 국소 단서를 복구하기 위해 외형 유도 방위 전문가(AGHE)는 방위각 추정을 위해 객체 영역을 더 높은 해상도로 재인코딩합니다. 실험 결과, OmniAct3D는 Spheriverse에서 이전 최고 3D 검출기 대비 2.96 NDS 포인트, PanoMMOcc에서 적응되지 않은 VFM 기준선(baseline) 대비 24.87 mAP 포인트 향상된 성능을 보였습니다. 타깃별 기하 적응을 통해 VARC는 동일 구성 mAP의 95~98%를 유지하여 센싱 구성 전반에서 재사용 가능한 객체 수준 3D 추론 능력을 보여줍니다. 소스 코드는 https://github.com/FeiT-FeiTeng/OmniAct3D 에서 공개될 예정입니다.
When an object detector is deployed in a novel setting it often experiences a drop in performance. This paper studies how an embodied agent can automatically fine-tune a pre-existing object detector while exploring and acquiring images in a new environment without relying on human intervention, i.e., a fully self-supervised approach. In our setting, an agent initially learns to explore the environment using a pre-trained off-the-shelf detector to locate objects and associate pseudo-labels. By assuming that pseudo-labels for the same object must be consistent across different views, we learn the exploration policy Look Around to mine hard samples, and we devise a novel mechanism called Disagreement Reconciliation for producing refined pseudo-labels from the consensus among observations. We implement a unified benchmark of the current state-of-the-art and compare our approach with pre-existing exploration policies and perception mechanisms. Our method is shown to outperform existing approaches, improving the object detector by 6.2% in a simulated scenario, a 3.59% advancement over other state-of-the-art methods, and by 9.97% in the real robotic test without relying on ground-truth. Code for the proposed approach and baselines are available at https://iit-pavis.github.io/Look_Around_And_Learn/.
센서 높이와 시점의 변화는 객체 수준의 포인트 분포를 변경하여 크로스 플랫폼 LiDAR 비지도 도메인 적응(UDA)을 어렵게 만듭니다. 자기 학습(self-training)은 레이블이 지정된 소스 스캔과 레이블이 지정되지 않은 타깃 스캔을 사용하지만, 유지된 예측은 유용한 타깃 위치를 제공하면서도 희소한 전경 반사 신호, 배경 클러터, 또는 예측된 박스와 불일치하는 포인트를 포함할 수 있습니다. 본 논문에서는 이러한 불일치를 박스-포인트 불일치(box-point inconsistency)라고 부릅니다. 우리는 타깃 배치를 보존하고 레이블이 지정된 소스 스캔의 측정된 기하학적 구조를 사용하여 연관된 의사 객체(pseudo-object)를 복원하는 SimFuse3D를 제안합니다. 객체 메모리(Object Memory)는 호환 가능한 레이블 지정 소스 인스턴스를 검색합니다. 타깃 시뮬레이션(Target Simulation)은 해당 인스턴스의 정답(ground-truth) 박스를 타깃 위치에 배치하고, 포인트를 타깃 시점 기하 구조와 정렬한 뒤, 정렬된 크롭을 필터링하여 타깃 관측치를 근사합니다. 신뢰도 유도형 다단계 위치 추정 재가중치(CMLR) 기법은 각 타깃 의사 객체의 신뢰도 점수를 RPN 위치 추정 및 R-CNN 박스 회귀에서 공유하는 유계 가중치로 매핑합니다. 모든 구성 요소는 적응 과정에서만 동작하며, 검출기 아키텍처와 추론 그래프는 변경하지 않습니다. 6개의 크로스 플랫폼 전이 실험에서 SimFuse3D는 보고된 모든 AP 지표에서 Pi3DET-Net을 능가하였으며, 비교된 적응 방법들 중 거의 모든 지표에서 1위를 차지했습니다. nuScenes-to-KITTI 환경에서는 평가된 두 검출기 모두에서 비교된 적응 방법 중 1위를 기록했습니다.
3D object trackers usually require training on large amounts of annotated data that is expensive and time-consuming to collect. Instead, we propose leveraging vast unlabeled datasets by self-supervised metric learning of 3D object trackers, with a focus on data association. Large scale annotations for unlabeled data are cheaply obtained by automatic object detection and association across frames. We show how these self-supervised annotations can be used in a principled manner to learn point-cloud embeddings that are effective for 3D tracking. We estimate and incorporate uncertainty in self-supervised tracking to learn more robust embeddings, without needing any labeled data. We design embeddings to differentiate objects across frames, and learn them using uncertainty-aware self-supervised training. Finally, we demonstrate their ability to perform accurate data association across frames, towards effective and accurate 3D tracking. Project videos and code are at https://jianrenw.github.io/Self-Supervised-3D-Data-Association.
In this paper, we demonstrate the ability to discriminate between cultivated maize plant and grass or grass-like weed image segments using the context surrounding the image segments. While convolutional neural networks have brought state of the art accuracies within object detection, errors arise when objects in different classes share similar features. This scenario often occurs when objects in images are viewed at too small of a scale to discern distinct differences in features, causing images to be incorrectly classified or localized. To solve this problem, we will explore using context when classifying image segments. This technique involves feeding a convolutional neural network a central square image along with a border of its direct surroundings at train and test times. This means that although images are labelled at a smaller scale to preserve accurate localization, the network classifies the images and learns features that include the wider context. We demonstrate the benefits of this context technique in the object detection task through a case study of grass (foxtail) and grass-like (yellow nutsedge) weed detection in maize fields. In this standard situation, adding context alone nearly halved the error of the neural network from 7.1% to 4.3%. After only one epoch with context, the network also achieved a higher accuracy than the network without context did after 50 epochs. The benefits of using the context technique are likely to particularly evident in agricultural contexts in which parts (such as leaves) of several plants may appear similar when not taking into account the context in which those parts appear.
본 보고서는 UrbanTwin V2X-Real Sim2Real LiDAR 챌린지에 대해 합성 포인트 클라우드 생성과 3D 객체 탐지를 모두 다루는 완전한 기술 솔루션을 제시한다. 데이터 생성에는 축별 분위수 보정을 채택한다. 공개된 V2X-Real 학습 데이터로 실제 기준 분포를 구성하고, 기본 합성 포인트 클라우드의 x, y, z 축 각각에 대해 독립적인 1차원 분위수 매핑을 추정한다. 이후 보정된 포인트 클라우드를 고정된 난수 시드로 재샘플링하고 유효 공간 범위로 클리핑하여, 프레임당 2,000개의 float32 XYZ 점으로 이루어진 50개 프레임을 생성한다. 탐지에는 RangeLDM으로 강화한 합성 포인트 클라우드 데이터, 프레임당 5만 점으로 다운샘플링한 포인트 클라우드, 그리고 재구성한 보행자 데이터로 학습한 2단계 Voxel R-CNN(VoxelRCNN) 탐지기를 사용한다. 학습은 NVIDIA GeForce RTX 4090 GPU 4장에서 전역 배치 크기 16으로 40 에폭 수행한다. 최종 테스트는 원래의 VoxelRCNN 모델을 이용한 단일 모델 직접 추론으로 수행한다. 공식 평가 서버에서 정규화된 Realism 점수는 0.7905, 정규화된 Detection 점수는 0.5583이며, mAP@0.5는 0.3908, BEV mAP@0.5는 0.4219, mAP@0.7은 0.1047이다. 공식 가중치 규칙에 따른 최종 종합 점수는 0.6 x 0.5583 + 0.4 x 0.7905 = 0.6512이다. 본 보고서는 데이터 구성, 모델 학습, 추론, 제출 형식, 공식 평가에 이르는 전체 구현 세부 사항을 기록한다.
Modern object detection methods can be divided into one-stage approaches and two-stage ones. One-stage detectors are more efficient owing to straightforward architectures, but the two-stage detectors still take the lead in accuracy. Although recent work try to improve the one-stage detectors by imitating the structural design of the two-stage ones, the accuracy gap is still significant. In this paper, we propose MimicDet, a novel and efficient framework to train a one-stage detector by directly mimic the two-stage features, aiming to bridge the accuracy gap between one-stage and two-stage detectors. Unlike conventional mimic methods, MimicDet has a shared backbone for one-stage and two-stage detectors, then it branches into two heads which are well designed to have compatible features for mimicking. Thus MimicDet can be end-to-end trained without the pre-train of the teacher network. And the cost does not increase much, which makes it practical to adopt large networks as backbones. We also make several specialized designs such as dual-path mimicking and staggered feature pyramid to facilitate the mimicking process. Experiments on the challenging COCO detection benchmark demonstrate the effectiveness of MimicDet. It achieves 46.1 mAP with ResNeXt-101 backbone on the COCO test-dev set, which significantly surpasses current state-of-the-art methods.
Camouflaged object detection and segmentation is a new and challenging research topic in computer vision. There is a serious issue of lacking data on concealed objects such as camouflaged animals in natural scenes. In this paper, we address the problem of few-shot learning for camouflaged object detection and segmentation. To this end, we first collect a new dataset, CAMO-FS, for the benchmark. As camouflaged instances are challenging to recognize due to their similarity compared to the surroundings, we guide our models to obtain camouflaged features that highly distinguish the instances from the background. In this work, we propose FS-CDIS, a framework to efficiently detect and segment camouflaged instances via two loss functions contributing to the training process. Firstly, the instance triplet loss with the characteristic of differentiating the anchor, which is the mean of all camouflaged foreground points, and the background points are employed to work at the instance level. Secondly, to consolidate the generalization at the class level, we present instance memory storage with the scope of storing camouflaged features of the same category, allowing the model to capture further class-level information during the learning process. The extensive experiments demonstrated that our proposed method achieves state-of-the-art performance on the newly collected dataset. Code is available at https://github.com/danhntd/FS-CDIS.
Object detection is of paramount importance in biomedical image analysis, particularly for lesion identification. While current methodologies are proficient in identifying and pinpointing lesions, they often lack the precision needed to detect minute biomedical entities (e.g., abnormal cells, lung nodules smaller than 3 mm), which are critical in blood and lung pathology. To address this challenge, we propose CAF-YOLO, based on the YOLOv8 architecture, a nimble yet robust method for medical object detection that leverages the strengths of convolutional neural networks (CNNs) and transformers. To overcome the limitation of convolutional kernels, which have a constrained capacity to interact with distant information, we introduce an attention and convolution fusion module (ACFM). This module enhances the modeling of both global and local features, enabling the capture of long-term feature dependencies and spatial autocorrelation. Additionally, to improve the restricted single-scale feature aggregation inherent in feed-forward networks (FFN) within transformer architectures, we design a multi-scale neural network (MSNN). This network improves multi-scale information aggregation by extracting features across diverse scales. Experimental evaluations on widely used datasets, such as BCCD and LUNA16, validate the rationale and efficacy of CAF-YOLO. This methodology excels in detecting and precisely locating diverse and intricate micro-lesions within biomedical imagery. Our codes are available at https://github.com/xiaochen925/CAF-YOLO.
Combining multiple object detection datasets offers a path to improved generalisation but is hindered by inconsistencies in class semantics and bounding box annotations. Some methods to address this assume shared label taxonomies and address only spatial inconsistencies; others require manual relabelling, or produce a unified label space, which may be unsuitable when a fixed target label space is required. We propose Label-Aligned Transfer (LAT), a label transfer framework that systematically projects annotations from diverse source datasets into the label space of a target dataset. LAT begins by training dataset-specific detectors to generate pseudo-labels, which are then combined with ground-truth annotations via a Privileged Proposal Generator (PPG) that replaces the region proposal network in two-stage detectors. To further refine region features, a Semantic Feature Fusion (SFF) module injects class-aware context and features from overlapping proposals using a confidence-weighted attention mechanism. This pipeline preserves dataset-specific annotation granularity while enabling many-to-one label space transfer across heterogeneous datasets, resulting in a semantically and spatially aligned representation suitable for training a downstream detector. LAT thus jointly addresses both class-level misalignments and bounding box inconsistencies without relying on shared label spaces or manual annotations. Across multiple benchmarks, LAT demonstrates consistent improvements in target-domain detection performance, achieving gains of up to +4.8AP over semi-supervised baselines.
Every day around the world, interminable terabytes of data are being captured for surveillance purposes. A typical 1-2MP CCTV camera generates around 7-12GB of data per day. Frame-by-frame processing of such enormous amount of data requires hefty computational resources. In recent years, compressive sensing approaches have shown impressive results in signal processing by reducing the sampling bandwidth. Different sampling mechanisms were developed to incorporate compressive sensing in image and video acquisition. Pixel-wise coded exposure is one among the promising sensing paradigms for capturing videos in the compressed domain, which was also realized into an all-CMOS sensor \cite{Xiong2017}. Though cameras that perform compressive sensing save a lot of bandwidth at the time of sampling and minimize the memory required to store videos, we cannot do much in terms of processing until the videos are reconstructed to the original frames. But, the reconstruction of compressive-sensed (CS) videos still takes a lot of time and is also computationally expensive. In this work, we show that object detection and localization can be possible directly on the CS frames (easily upto 20x compression). To our knowledge, this is the first time that the problem of object detection and localization on CS frames has been attempted. Hence, we also created a dataset for training in the CS domain. We were able to achieve a good accuracy of 46.27\% mAP(Mean Average Precision) with the proposed model with an inference time of 23ms directly on the compressed frames(approx. 20 original domain frames), this facilitated for real-time inference which was verified on NVIDIA TX2 embedded board. Our framework will significantly reduce the communication bandwidth, and thus reduction in power as the video compression will be done at the image sensor processing core.
Modern pre-trained architectures struggle to retain previous information while undergoing continuous fine-tuning on new tasks. Despite notable progress in continual classification, systems designed for complex vision tasks such as detection or segmentation still struggle to attain satisfactory performance. In this work, we introduce a memory-based detection transformer architecture to adapt a pre-trained DETR-style detector to new tasks while preserving knowledge from previous tasks. We propose a novel localized query function for efficient information retrieval from memory units, aiming to minimize forgetting. Furthermore, we identify a fundamental challenge in continual detection referred to as background relegation. This arises when object categories from earlier tasks reappear in future tasks, potentially without labels, leading them to be implicitly treated as background. This is an inevitable issue in continual detection or segmentation. The introduced continual optimization technique effectively tackles this challenge. Finally, we assess the performance of our proposed system on continual detection benchmarks and demonstrate that our approach surpasses the performance of existing state-of-the-art resulting in 5-7% improvements on MS-COCO and PASCAL-VOC on the task of continual detection.
In this report, we introduce our real-time 2D object detection system for the realistic autonomous driving scenario. Our detector is built on a newly designed YOLO model, called YOLOX. On the Argoverse-HD dataset, our system achieves 41.0 streaming AP, which surpassed second place by 7.8/6.1 on detection-only track/fully track, respectively. Moreover, equipped with TensorRT, our model achieves the 30FPS inference speed with a high-resolution input size (e.g., 1440-2304). Code and models will be available at https://github.com/Megvii-BaseDetection/YOLOX
Neural networks have been employed for a wide range of processing applications like image processing, motor control, object detection and many others. Living neural networks offer advantages of lower power consumption, faster processing, and biological realism. Optogenetics offers high spatial and temporal control over biological neurons and presents potential in training live neural networks. This work proposes a simulated living neural network trained indirectly by backpropagating STDP based algorithms using precision activation by optogenetics achieving accuracy comparable to traditional neural network training algorithms.