Object detection models rely on large-scale annotated datasets that require substantial manual annotation effort. The labeling costs further increase when annotation uncertainty is captured through soft labels, as multiple annotations must be collected for each object. We address this challenge by introducing a semi-supervised framework for the uncertainty-aware annotation of object detection datasets. Our approach leverages open-vocabulary object detectors to generate region proposals and semantically meaningful object representations. Based on these representations, we construct an object-level similarity graph and propagate probabilistic label information. Our framework assigns soft labels to previously unlabeled objects, quantifying both aleatoric and epistemic uncertainty. We evaluate our method on four object detection benchmarks with soft-label annotations and demonstrate that high-quality annotations can be obtained with only a small fraction of manually labeled objects. Furthermore, we show that detectors trained on semi-automatically annotated data achieve performance comparable to models trained on fully manually annotated data. We achieve state-of-the-art downstream detection performance for the task of semi-supervised object detection on the COCO dataset, with 40.5% AP virtually matching the performance of the model trained on fully manually constructed data (40.9%) at only 1% of labeling costs. These results highlight the potential of open-vocabulary models and graph-based label propagation for efficient, accurate and uncertainty-aware pseudo-labeling.
Object detection constitutes the primary task within the domain of computer vision. It is utilized in numerous domains. Nonetheless, object detection continues to encounter the issue of catastrophic forgetting. The model must be retrained whenever new products are introduced, utilizing not only the new products dataset but also the entirety of the previous dataset. The outcome is obvious: increasing model training expenses and significant time consumption. In numerous sectors, particularly retail checkout, the frequent introduction of new products presents a great challenge. This study introduces Zero-Retraining Based Recognition and Object Detection (ZeBROD), a methodology designed to address the issue of catastrophic forgetting by integrating YOLO11n for object localization with DeIT and Proxy Anchor Loss for feature extraction and metric learning. For classification, we utilize cosine similarity between the embedding features of the target product and those in the Qdrant vector database. In a case study conducted in a retail store with 140 products, the experimental results demonstrate that our proposed framework achieves encouraging accuracy, whether for detecting new or existing products. Furthermore, without retraining, the training duration difference is significant. We achieve almost 3 times the training time efficiency compared to classical object detection approaches. This efficiency escalates as additional new products are added to the product database. The average inference time is 580 ms per image containing multiple products, on an edge device, validating the proposed framework's feasibility for practical use.
Medical object detection suffers when a single detector is trained on mixed medical modalities (e.g., CXR, CT, MRI) due to heterogeneous statistics and disjoint representation spaces. To address this challenge, we turn to representation alignment, an approach that has proven effective for bringing features from different sources into a shared space. Specifically, we target the representations of DETR-style object queries and propose a simple, detector-agnostic framework to align them with modality context. First, we define modality tokens: compact, text-derived embeddings encoding imaging modality that are lightweight and require no extra annotations. We integrate the modality tokens into the detection process via Multimodality Context Attention (MoCA), mixing object-query representations via self-attention to propagate modality context within the query set. This preserves DETR-style architectures and adds negligible latency while injecting modality cues into object queries. We further introduce QueryREPA, a short pretraining stage that aligns query representations to their modality tokens using a task-specific contrastive objective with modality-balanced batches. Together, MoCA and QueryREPA produce modality-aware, class-faithful queries that transfer effectively to downstream training. Across diverse modalities trained altogether, the proposed approach consistently improves AP with minimal overhead and no architectural modifications, offering a practical path toward robust multimodality medical object detection.
Detection transformers have recently shown promising object detection results and attracted increasing attention. However, how to develop effective domain adaptation techniques to improve its cross-domain performance remains unexplored and unclear. In this paper, we delve into this topic and empirically find that direct feature distribution alignment on the CNN backbone only brings limited improvements, as it does not guarantee domain-invariant sequence features in the transformer for prediction. To address this issue, we propose a novel Sequence Feature Alignment (SFA) method that is specially designed for the adaptation of detection transformers. Technically, SFA consists of a domain query-based feature alignment (DQFA) module and a token-wise feature alignment (TDA) module. In DQFA, a novel domain query is used to aggregate and align global context from the token sequence of both domains. DQFA reduces the domain discrepancy in global feature representations and object relations when deploying in the transformer encoder and decoder, respectively. Meanwhile, TDA aligns token features in the sequence from both domains, which reduces the domain gaps in local and instance-level feature representations in the transformer encoder and decoder, respectively. Besides, a novel bipartite matching consistency loss is proposed to enhance the feature discriminability for robust object detection. Experiments on three challenging benchmarks show that SFA outperforms state-of-the-art domain adaptive object detection methods. Code has been made available at: https://github.com/encounter1997/SFA.
Underwater object detection is strongly affected by domain shift, where performance can vary significantly across different locations, habitats, and deployment conditions. However, detector performance is typically evaluated using aggregate metrics that hide failures in specific environments, while existing domain generalization benchmarks often rely on synthetic variations that do not reflect real-world conditions. To address these limitations we introduce a framework that characterizes underwater images by appearance, scene composition, and acquisition geometry to assign domain labels. Using this framework, we perform the first systematic study of how domain factors influence both human annotation quality in underwater object detection datasets and deep learning-based detector performance, revealing substantial domain-dependent discrepancies. By incorporating physically meaningful domain labels, domain shift becomes something we can characterize, measure, benchmark, and act on. We highlight how our framework can be used to guide data collection and annotation, design more informative benchmarks, and assess detector robustness across diverse underwater environments.
Surgical augmented reality (AR) can provide contextual guidance by overlaying virtual annotations, tool cues, and procedural information onto the surgical workspace. However, the virtual content may obstruct task-relevant real-world information, such as surgical instruments, and interfere with users' perception during time-sensitive surgical tasks. In this paper, we investigate visual obstruction detection for surgical AR and present a latency-aware pipeline that combines vision-language model (VLM)-based surgical-object recognition with segmentation-based obstruction reasoning. To reduce inference overhead, the system adopts a cascaded small-to-large VLM architecture with segmentation-guided early exiting and attention-based visual token pruning. The small VLM handles easy frames when its key-object prediction is supported by segmentation consistency, while difficult frames are forwarded to a large VLM with pruned visual tokens. We construct a pseudo-AR surgical obstruction detection benchmark by overlaying virtual content onto surgical-tool images and labeling whether the virtual content obstructs task-relevant instruments. Evaluation results show that the proposed system achieves 87.43% obstruction detection accuracy with an average end-to-end latency of 479 ms, reducing latency by 62.90% compared with a cloud large-model baseline. These results demonstrate the feasibility of latency-aware obstruction detection for surgical AR and motivate future work on dynamic surgical videos, multi-object scenes, and clinically grounded AR guidance content.
Large vision-language models can produce object hallucinations in image descriptions, highlighting the need for effective detection and mitigation strategies. Prior work commonly relies on the model's attention weights on visual tokens as a detection signal. We reveal that coarse-grained attention-based analysis is unreliable due to hidden confounders, specifically token position and object repetition in a description. This leads to Simpson's paradox: the attention trends reverse or disappear when statistics are aggregated. Based on this observation, we introduce HaloProbe, a Bayesian framework that factorizes external description statistics and internal decoding signals to estimate token-level hallucination probabilities. HaloProbe uses balanced training to encourage learning from internal evidence and combines it with a learned prior over external features to recover the true posterior. While intervention-based mitigation methods often degrade utility or fluency by modifying models' internals, we use HaloProbe as an external scoring signal for non-invasive mitigation. Our experiments show that HaloProbe-guided decoding reduces hallucinations more effectively than state-of-the-art intervention-based methods while preserving caption utility and linguistic fluency.
Strong data augmentation is a fundamental component of state-of-the-art mean teacher-based Source-Free domain adaptive Object Detection (SFOD) methods, enabling consistency-based self-supervised optimization along weak augmentation. However, our theoretical analysis and empirical observations reveal a critical limitation: strong augmentation can inadvertently erase class-relevant components, leading to artificial inter-category confusion. To address this issue, we introduce Weak-to-strong Semantics Compensation (WSCo), a novel remedy that leverages weakly augmented images, which preserve full semantics, as anchors to enrich the feature space of their strongly augmented counterparts. Essentially, this compensates for the class-relevant semantics that may be lost during strong augmentation on the fly. Notably, WSCo can be implemented as a generic plug-in, easily integrable with any existing SFOD pipelines. Extensive experiments validate the negative impact of strong augmentation on detection performance, and the effectiveness of WSCo in enhancing the performance of previous detection models on standard benchmarks.
Benchmark suites are crucial for advancing computer vision research, providing not only curated datasets but also standardized evaluation protocols for reproducible comparison. The railway domain lacks such resources, hindering progress in automated train operation. To address this gap, we present RAIL-BENCH, the first perception benchmark suite designed specifically for railways. RAIL-BENCH comprises five challenges - rail track detection, object detection, vegetation segmentation, multi-object tracking, and monocular visual odometry - each tailored to railway-specific characteristics and including a curated dataset from diverse real-world scenarios, evaluation metrics, and public scoreboards; we demonstrate the first three challenges in this paper. We further introduce LineAP, a novel segment-based average precision metric for rail track detection that decouples geometric accuracy from instance-level grouping and employs a track-width-based distance threshold approximating a fixed real-world distance, addressing key limitations of existing line detection metrics. Our benchmark suite is publicly available at https://www.mrt.kit.edu/railbench/.
Monocular 3D object detection spans two regimes: closed-set detectors operating within a fixed category vocabulary, and open-vocabulary detectors that localize arbitrary categories by leveraging depth foundation models for 3D geometry. We find that current depth foundation models, despite their strong zero-shot generalization, lack the object-level precision 3D detection demands: substituting a state-of-the-art depth foundation model for a strong detector's predicted depth degrades accuracy. Rather than pushing detectors or depth models to be more accurate end-to-end, we treat object-level depth refinement as a stand-alone task and present RefineAny3D, a vision-language model that corrects depth without ever predicting a numerical value. Our key insight is that depth error has a direct visual signature in image space: when projected onto the image, a correctly placed box tightly encloses the object, while a too-far box projects too small and a too-close box projects too large. Depth refinement thus reduces to a visual alignment problem rather than a metric regression problem, which we instantiate by extending the VLM's vocabulary with action tokens that replace numerical depth output with categorical decisions, and by supervising the model on a large-scale chain-of-thought dataset that grounds each decision in explicit visual evidence. Applied as a single post-hoc step, RefineAny3D delivers consistent gains across closed-set detectors, open-vocabulary detectors, and 3D auto-labeling tools, and generalizes to novel categories, scenes, and cameras without retraining. Code is available at https://github.com/alanzhangcs/RefineAny3D.
Infrastructure-based perception plays a crucial role in intelligent transportation systems by providing global situational awareness and enabling cooperative autonomy. However, camera-based 3D object detection remains challenging under multi-view infrastructure setups, heterogeneous camera configurations, degraded sensing conditions, diverse road layouts, and adverse environmental conditions. We introduce MIC-BEV, a Transformer-based bird's-eye-view (BEV) perception framework for infrastructure-based multi-camera 3D object detection. MIC-BEV supports a variable number of cameras with heterogeneous intrinsic and extrinsic parameters and uses a graph-enhanced fusion module to adaptively integrate multi-view image features by modeling geometric relationships between cameras and the BEV grid. To support training and evaluation, we introduce M2I, a synthetic dataset featuring diverse camera layouts, road geometries, environmental conditions, and sensor-degradation cases. Experiments on the simulated M2I dataset and the real-world RoScenes dataset show that MIC-BEV achieves strong 3D detection performance and improved robustness under challenging conditions. Ablation studies validate the effectiveness of its key design choices, while weight analyses show that the learned fusion weights are associated with view-reliability factors and adapt to degraded camera inputs. We further report computational cost and deployment-related factors, providing additional evidence of practical feasibility and real-world deployment potential. These results indicate that MIC-BEV is a promising framework for infrastructure-based perception. The dataset and source code are available at: https://github.com/HandsomeYun/MIC-BEV.
Object detectors achieve strong performance under nominal imaging conditions but can fail silently when exposed to blur, noise, compression, adverse weather, or resolution changes. In safety-critical settings, it is therefore insufficient to produce predictions without assessing whether the input remains within the detector's nominal operating regime, which we refer to as self-aware object detection. We introduce a degradation-aware framework based on degradation manifolds, which structure a detector's feature space according to image degradation rather than semantic content. Our method augments a standard detection backbone with a lightweight embedding head trained via multi-layer contrastive learning, where images sharing the same degradation composition are pulled together while differing configurations are separated. Unlike prior contrastive IQA approaches, our method performs detector-aligned input-fidelity monitoring: it uses task-backbone features and a nominal prototype to detect degradation-induced shifts, rather than estimating perceptual image quality in a standalone representation. To anchor the representation, we estimate a pristine prototype from clean training embeddings, defining a nominal operating point in feature space. Self-awareness is then derived as geometric deviation from this reference, providing an intrinsic image-level signal of degradation-induced shift that is independent of detection confidence. Experiments on synthetic corruptions, cross-dataset zero-shot transfer, and natural weather-induced shifts show strong pristine-degraded separability, consistent behavior across detector architectures, and robust generalization across datasets with substantial content variation. These results suggest that degradation-aware detector representations provide a reliable signal for monitoring input-fidelity shifts.
How can a robot quickly identify and recognize new objects shown to it during a human demonstration? Existing closed-set object detectors frequently fail at this because the objects are out-of-distribution. While open-set detectors (e.g., VLMs) sometimes succeed, they often require expensive and tedious human-in-the-loop prompt engineering to uniquely recognize novel object instances. In this paper, we present a self-supervised system that eliminates the need for tedious language descriptions and expensive prompt engineering by training a bespoke object detector on an automatically created dataset, supervised by the human demonstration itself. In our approach, "Show, Don't Tell," we show the detector the specific objects of interest during the demonstration, rather than telling the detector about these objects via complex language descriptions. By bypassing language altogether, this paradigm enables us to quickly train bespoke detectors tailored to the relevant objects observed in human task demonstrations. We develop an integrated on-robot system to deploy our "Show, Don't Tell" paradigm of automatic dataset creation and novel object-detection on a real-world robot. Empirical results demonstrate that our pipeline significantly outperforms state-of-the-art detection and recognition methods for manipulated objects, leading to improved task completion for the robot.
Open-vocabulary detection (OVD) recognizes categories unseen during training through textual category queries, yet achieving strong generalization with real-time efficiency remains challenging. Beyond vocabulary scaling, zero-shot generalization may benefit from reusable visual--semantic cues learned from seen data, including attributes, actions, states, and contextual relations. Existing real-time OVD methods primarily emphasize vocabulary coverage and efficient region/query--text matching; under strict efficiency constraints, compact detectors may struggle to absorb rich instance semantics and scene context. We propose RT-DETR-World, a compact DETR-style detector that transfers the rich semantics conveyed by descriptions during training while retaining lightweight query--text matching at inference. We construct GroundingCapv2 with three levels of supervision: category names for standard OVD, object descriptions conveying instance-level semantics, and image descriptions conveying object relations and scene context. These descriptions serve only as training-time semantic supervision. To help the compact detector absorb these semantics, we propose Dual-Path Description Alignment (DDA), combining a deployment-consistent MiniLM pathway with a training-only LLM teacher. MiniLM provides query--category supervision and object-description alignment, while offline teacher features supervise matched queries and global visual representations at the object and image levels, respectively. All teacher features are precomputed, and the teacher-side modules are removed after training. We further propose Relation-Aware Negative Relaxation (RNR), which uses teacher-derived semantic similarities to relax related negatives while preserving exact positives. Experiments demonstrate competitive zero-shot accuracy and a favorable accuracy--efficiency trade-off. The code will be released.
6D pose estimation of novel objects without CAD models remains an open challenge: model-free methods must build the object representation from scratch during onboarding and have long trailed their model-based counterparts. We present NovaSplat, a model-free pipeline that closes this gap. From a short smartphone video captured at onboarding, we train a 3D Gaussian Splatting (3DGS) model that serves both as a template source for detection and coarse pose estimation and as a differentiable backbone for pose refinement. Our refiner optimizes the object-to-camera pose with combined photometric and geometric losses under a continuous rotation parametrization free of gradient singularities. On a household-object benchmark, NovaSplat surpasses all model-based methods. On a challenging industrial benchmark with textureless and specular surfaces, it outperforms all model-free competitors while substantially narrowing the gap to model-based ones. We further introduce NovaSplat-Lite, an inference variant that is $3.1\times$ faster at negligible accuracy cost. Code will be made available at https://github.com/POSE-Lab/novasplat .
Long-tailed human-object interaction (HOI) detection remains challenging because rare interactions are sparsely represented in training data. Although test observations provide additional visual evidence, existing cache-based methods typically assign uniform memory capacity across classes, limiting their ability to capture informative features of underrepresented interactions. To address this issue, we propose Adaptive Diversity Cache (ADC), a training-free, plug-and-play framework that adaptively constructs test-time caches through frequency-aware capacity allocation and confidence-diversity joint sample selection. Specifically, Frequency-Aware Cache Adaptation assigns larger capacities to underrepresented interactions using frequency-dependent priorities and a weighted logarithmic allocation objective, and fills remaining slots with selected augmented views. Confidence-Diversity Joint Cache Selection further combines prediction confidence with local feature diversity to retain informative cache entries. Cache retrieval then refines the predictions of the frozen detector. Experiments on HICO-DET and V-COCO across three base detectors demonstrate consistent improvements. On HICO-DET, ADC improves mean average precision (mAP) for rare interactions by 4.02--8.57 percentage points, while also improving performance on non-rare interactions. Online adaptation further achieves gains of 2.87--7.36 percentage points on rare interactions. These results demonstrate that adaptive test-time memory effectively complements existing HOI detectors and improves long-tailed detection performance.
Illegal and unsustainable wildlife trade remains a major conservation challenge and is increasingly shifting to online marketplaces where monitoring is difficult. To address this gap, we used real-world images from online platforms, enforcement seizures, and market inspections to develop a deep-learning object detection framework to identify products derived from three highly traded species: elephant, pangolin, and tiger. Our dataset of 39,941 images included both wildlife products and visually similar non-wildlife items, reflecting real enforcement complexity. We used cross-entropy and asymmetric focal loss functions to train Faster Region-based convolutional neural network (Faster R-CNN) models and compared species-specific and multispecies training strategies. The best-performing model, trained with cross-entropy on all species combined, achieved an overall accuracy of 84.2%, with high performance for pangolin (93.6%) and tiger (90.3%) and moderate performance for elephant (71.1%). Although asymmetric focal loss increased recall, it substantially reduced precision. We deployed the optimized model in a prototype smartphone application, potentially usable in real-world settings for monitoring online images. With an independent test set, the application achieved 91.3% accuracy and an F-score of 0.86. These results demonstrate the potential of object detection models to support scalable, real-time monitoring of wildlife trade and enable the rapid triage of suspicious posts, regulatory compliance, and conservation decision-making.
Salient and camouflaged object detection (SOD and COD) both recover foreground support, yet most joint methods output only a binary mask and therefore cannot distinguish the type. USC12K distinguishes the two types, but its baseline USCNet still binds foreground existence to the type within a single normalization, so the failure source cannot be attributed. Our cross-task study shows that foreground localization transfers between SOD and COD, whereas type discrimination does not: a COD-only model recovers foreground on SOD images but its type judgement follows the training distribution rather than the foreground-background relation, so the two decisions require separate mechanisms. SegDis factorizes the output into an explicit foreground probability and a conditional type probability given foreground: FLR provides high-quality task-agnostic support, and RTA refines the SOD/COD assignment from the foreground-background relation without a task identifier. Because support is explicit, each ground-truth foreground pixel is classified as correct-type, type-swapped, or missed, so foreground-recovery and type-assignment errors are diagnosed separately. With PVT-v2-B2, SegDis attains 68.98% FG-mIoU and 78.35% mIoU on USC12K at about one-eighth of USCNet's parameters; CSCS is 5.75.
While large models demonstrate the strong representational power of vanilla attention, this core mechanism cannot be directly applied to Dense Object Tracking: its quadratic all-to-all interactions are computationally prohibitive for dense motion estimation on high-resolution features. This mismatch prevents Dense Object Tracking from fully leveraging attention-based modeling in crowded and occlusion-heavy scenes. To address this challenge, we introduce GateMOT, an online tracking framework centered on Q-Gated Attention (Q-Attention), an efficient and spatially aware attention variant. Our key idea is to repurpose the Query from a similarity-conditioning term into a learnable gating unit. This Gating-Query (Gating-Q) produces a probabilistic gate that modulates Key features in an element-wise manner, enabling explicit relevance selection instead of costly global aggregation. Built on this mechanism, parallel Q-Attention heads transform one shared feature map into task-specific yet consistent representations for detection, motion, and re-identification, yielding a tightly coupled multi-task decoder with linear-complexity gating operations. GateMOT achieves state-of-the-art HOTA of 48.4, MOTA of 67.8, and IDF1 of 64.5 on BEE24, and demonstrates strong performance on additional Dense Object Tracking benchmarks. These results show that Q-Attention is a simple, effective, and transferable building block for attention-based tracking in dense tracking scenarios.
Single-Domain Generalized Object Detection (S-DGOD) aims to generalize from a single source domain to unseen domains under distribution shifts. Existing efforts mainly focus on simulating unseen domains or suppressing domain-specific factors. However, they largely overlook the intrinsic generalization capability embedded in Vision Foundation Models (VFMs). In this work, we show that unconstrained fine-tuning disrupts the geometric structure of pre-trained representations, which encodes domain-invariant semantics. Motivated by this, we propose Geometric-Invariance Fine-Tuning (GIFT), a geometric-aware fine-tuning method for VFMs. Specifically, we introduce a series of learnable Householder reflections to the pre-trained weights to encourage geometry-consistent adaptation under domain shift. In addition, we propose a lightweight channel-wise scaling to facilitate effective task-specific adaptation. Extensive experiments demonstrate that GIFT consistently outperforms state-of-the-art (SOTA) approaches on two S-DGOD benchmarks. Notably, by introducing fewer than 1% additional trainable parameters into the frozen backbone, the proposed method achieves improvements of +7.9% and +2.9% in mPC over the previous SOTA on the Cityscapes-C and Diverse Weather Dataset, respectively. This finding highlights that preserving representation geometry is crucial for robust cross-domain generalization, establishing GIFT as a principled and efficient solution to domain shift.