Remote sensing images pose distinct challenges for downstream tasks due to their inherent complexity. While a considerable amount of research has been dedicated to remote sensing classification, object detection and semantic segmentation, most of these studies have overlooked the valuable prior knowledge embedded within remote sensing scenarios. Such prior knowledge can be useful because remote sensing objects may be mistakenly recognized without referencing a sufficiently long-range context, which can vary for different objects. This paper considers these priors and proposes a lightweight Large Selective Kernel Network (LSKNet) backbone. LSKNet can dynamically adjust its large spatial receptive field to better model the ranging context of various objects in remote sensing scenarios. To our knowledge, large and selective kernel mechanisms have not been previously explored in remote sensing images. Without bells and whistles, our lightweight LSKNet sets new state-of-the-art scores on standard remote sensing classification, object detection and semantic segmentation benchmarks. Our comprehensive analysis further validated the significance of the identified priors and the effectiveness of LSKNet. The code is available at https://github.com/zcablii/LSKNet.
A self-driving vehicle must understand its environment to determine the appropriate action. Traditional autonomy systems rely on object detection to find the agents in the scene. However, object detection assumes a discrete set of objects and loses information about uncertainty, so any errors compound when predicting the future behavior of those agents. Alternatively, dense occupancy grid maps have been utilized to understand free-space. However, predicting a grid for the entire scene is wasteful since only certain spatio-temporal regions are reachable and relevant to the self-driving vehicle. We present a unified, interpretable, and efficient autonomy framework that moves away from cascading modules that first perceive, then predict, and finally plan. Instead, we shift the paradigm to have the planner query occupancy at relevant spatio-temporal points, restricting the computation to those regions of interest. Exploiting this representation, we evaluate candidate trajectories around key factors such as collision avoidance, comfort, and progress for safety and interpretability. Our approach achieves better highway driving quality than the state-of-the-art in high-fidelity closed-loop simulations.
Existing camouflaged object detection~(COD) methods depend heavily on large-scale pixel-level annotations.However, acquiring such annotations is laborious due to the inherent camouflage characteristics of the objects.Semi-supervised learning offers a promising solution to this challenge.Yet, its application in COD is hindered by significant pseudo-label noise, both pixel-level and instance-level.We introduce CamoTeacher, a novel semi-supervised COD framework, utilizing Dual-Rotation Consistency Learning~(DRCL) to effectively address these noise issues.Specifically, DRCL minimizes pseudo-label noise by leveraging rotation views' consistency in pixel-level and instance-level.First, it employs Pixel-wise Consistency Learning~(PCL) to deal with pixel-level noise by reweighting the different parts within the pseudo-label.Second, Instance-wise Consistency Learning~(ICL) is used to adjust weights for pseudo-labels, which handles instance-level noise.Extensive experiments on four COD benchmark datasets demonstrate that the proposed CamoTeacher not only achieves state-of-the-art compared with semi-supervised learning methods, but also rivals established fully-supervised learning methods.Our code will be available soon.
신뢰할 수 있는 환경 인식은 자율주행 차량의 안전한 운행을 위한 주요 장애 요인 중 하나로 남아 있다. 의도된 기능의 안전성(Safety of the Intended Functionality, SOTIF)은 인식 부족으로 인한 안전 위험, 특히 기존 검출기가 종종 실패하는 불리한 조건에서의 위험과 관련된다. 대규모 비전-언어 모델(Large Vision-Language Models, LVLM)은 유망한 의미론적 추론 능력을 보여주지만, 안전이 중요한 2D 객체 검출에 대한 정량적 효과는 아직 충분히 탐구되지 않았다. 본 논문은 장꼬리 교통 시나리오와 환경 열화를 위해 특별히 구축된 벤치마크인 PeSOTIF 데이터셋을 사용하여 10개의 대표적인 LVLM에 대한 체계적 평가를 제시한다. 성능은 두 가지 전문 검출기인 앵커 기반 YOLOv5와 트랜스포머 기반 RT-DETRv4와 정량적으로 비교한다. 실험 결과는 중요한 트레이드오프를 드러낸다. 최고 성능 LVLM(예: Gemini 3)은 재현율에서 YOLOv5보다 25% 이상 높으며, 자연적 시각 열화 조건에서 RT-DETRv4와 거의 대등한 수준을 보이지만, 전문 검출기는 수작업 교란에 대한 기하학적 정밀도에서 우위를 유지한다. 이러한 결과는 의미론적 추론과 기하학적 회귀의 상보적 강점을 부각하며, SOTIF 지향 자율주행 시스템에서 LVLM을 고수준 안전 검증기로 활용하는 것을 뒷받침한다.
자연 환경에서의 조류 모니터링은 장면에 비해 일부 종의 조류 크기가 작고, 배경 혼잡, 조명 변화, 관찰자의 시점 문제 등으로 인해 어렵습니다. 또한 행동 패턴을 이해하는 데 필수적인 대규모의 현실적인 데이터셋이 부족하여 연구 진척이 더욱 제한되고 있습니다. 이러한 격차를 해소하기 위해, 본 연구는 41개 녹화 세션에서 수집된 6,000장 이상의 레이블링된 이미지로 구성된 서부파랑새(Sialia Mexicana) 탐지 및 분할을 위한 새로운 벤치마크 데이터셋을 소개합니다. 이 데이터셋은 조류가 이미지의 극히 일부만을 차지하는 고해상도(4K) 야생 이미지를 특징으로 합니다. 본 논문에서는 지도학습 기반 탐지기, 제로샷(zero-shot) 및 미세조정 설정 하에서의 개방형 어휘(open-vocabulary) 모델, 분할 접근 방식을 평가했습니다. 전반적으로 지도학습 탐지기가 여전히 가장 신뢰할 만한 성능을 보였으며, Faster R-CNN이 가장 높은 탐지 mAP를 기록했고 RT-DETR은 최적의 정밀도-재현율 상충관계(precision-recall trade-off)를 제공했습니다. 개방형 어휘 모델은 제로샷 설정에서는 성능이 저조했으나 미세조정을 통해 성능이 크게 향상되어, YOLO-World는 지도학습 방식과 경쟁할 만한 수준에 도달하며 가장 높은 정밀도, F1-점수, [email omitted] 달성했습니다. 분할의 경우 지도학습 방식이 Grounded-SAM 및 SAM 3을 크게 능가했습니다. Mask R-CNN이 가장 높은 마스크 mAP를 기록한 반면, YOLOv8-Seg는 가장 우수한 정밀도와 가장 빠른 추론 속도를 제공했습니다. 진단 분석 결과 실패 원인은 객체 크기만으로 설명되지 않으며 겉보기 스케일, 밝기, 대비, 혼잡도, 흐림(blur), 밀집도, 녹화 세션별 변동의 조합에 기인하는 것으로 나타났습니다. 종합적으로 본 연구 결과는 복잡한 생태적 장면에서 제로샷 조류 탐지의 어려움을 강조하며, 소형 객체 환경에서 도메인 적응(domain adaptation)의 중요성을 부각합니다.
On-road obstacle detection is an important field of research that falls in the scope of intelligent transportation infrastructure systems. The use of vision-based approaches results in an accurate and cost-effective solution to such systems. In this research paper, we propose a threat detection mechanism for autonomous self-driving cars using dashcam videos to ensure the presence of any unwanted obstacle on the road that falls within its visual range. This information can assist the vehicle's program to en route safely. There are four major components, namely, YOLO to identify the objects, advanced lane detection algorithm, multi regression model to measure the distance of the object from the camera, the two-second rule for measuring the safety, and limiting speed. In addition, we have used the Car Crash Dataset(CCD) for calculating the accuracy of the model. The YOLO algorithm gives an accuracy of around 93%. The final accuracy of our proposed Threat Detection Model (TDM) is 82.65%.
Change detection is one of the fundamental applications of synthetic aperture radar (SAR) images. However, speckle noise presented in SAR images has a much negative effect on change detection. In this research, a novel two-phase object-based deep learning approach is proposed for multi-temporal SAR image change detection. Compared with traditional methods, the proposed approach brings two main innovations. One is to classify all pixels into three categories rather than two categories: unchanged pixels, changed pixels caused by strong speckle (false changes), and changed pixels formed by real terrain variation (real changes). The other is to group neighboring pixels into segmented into superpixel objects (from pixels) such as to exploit local spatial context. Two phases are designed in the methodology: 1) Generate objects based on the simple linear iterative clustering algorithm, and discriminate these objects into changed and unchanged classes using fuzzy c-means (FCM) clustering and a deep PCANet. The prediction of this Phase is the set of changed and unchanged superpixels. 2) Deep learning on the pixel sets over the changed superpixels only, obtained in the first phase, to discriminate real changes from false changes. SLIC is employed again to achieve new superpixels in the second phase. Low rank and sparse decomposition are applied to these new superpixels to suppress speckle noise significantly. A further clustering step is applied to these new superpixels via FCM. A new PCANet is then trained to classify two kinds of changed superpixels to achieve the final change maps. Numerical experiments demonstrate that, compared with benchmark methods, the proposed approach can distinguish real changes from false changes effectively with significantly reduced false alarm rates, and achieve up to 99.71% change detection accuracy using multi-temporal SAR imagery.
Video object segmentation (VOS) describes the task of segmenting a set of objects in each frame of a video. In the semi-supervised setting, the first mask of each object is provided at test time. Following the one-shot principle, fine-tuning VOS methods train a segmentation model separately on each given object mask. However, recently the VOS community has deemed such a test time optimization and its impact on the test runtime as unfeasible. To mitigate the inefficiencies of previous fine-tuning approaches, we present efficient One-Shot Video Object Segmentation (e-OSVOS). In contrast to most VOS approaches, e-OSVOS decouples the object detection task and predicts only local segmentation masks by applying a modified version of Mask R-CNN. The one-shot test runtime and performance are optimized without a laborious and handcrafted hyperparameter search. To this end, we meta learn the model initialization and learning rates for the test time optimization. To achieve optimal learning behavior, we predict individual learning rates at a neuron level. Furthermore, we apply an online adaptation to address the common performance degradation throughout a sequence by continuously fine-tuning the model on previous mask predictions supported by a frame-to-frame bounding box propagation. e-OSVOS provides state-of-the-art results on DAVIS 2016, DAVIS 2017, and YouTube-VOS for one-shot fine-tuning methods while reducing the test runtime substantially. Code is available at https://github.com/dvl-tum/e-osvos.
Airplane detection from satellite imagery is a challenging task due to the complex backgrounds in the images and differences in data acquisition conditions caused by the sensor geometry and atmospheric effects. Deep learning methods provide reliable and accurate solutions for automatic detection of airplanes; however, huge amount of training data is required to obtain promising results. In this study, we create a novel airplane detection dataset called High Resolution Planes (HRPlanes) by using images from Google Earth (GE) and labeling the bounding box of each plane on the images. HRPlanes include GE images of several different airports across the world to represent a variety of landscape, seasonal and satellite geometry conditions obtained from different satellites. We evaluated our dataset with two widely used object detection methods namely YOLOv4 and Faster R-CNN. Our preliminary results show that the proposed dataset can be a valuable data source and benchmark data set for future applications. Moreover, proposed architectures and results of this study could be used for transfer learning of different datasets and models for airplane detection.
Glomerulus detection is a key step in histopathological evaluation of microscopy images of kidneys. However, the task of automatic detection of glomeruli poses challenges due to the disparity in sizes and shapes of glomeruli in renal sections. Moreover, extensive variations of their intensities due to heterogeneity in immunohistochemistry staining are also encountered. Despite being widely recognized as a powerful descriptor for general object detection, the rectangular histogram of oriented gradients (Rectangular HOG) suffers from many false positives due to the aforementioned difficulties in the context of glomerulus detection. A new descriptor referred to as Segmental HOG is developed to perform a comprehensive detection of hundreds of glomeruli in images of whole kidney sections. The new descriptor possesses flexible blocks that can be adaptively fitted to input images to acquire robustness to deformations of glomeruli. Moreover, the novel segmentation technique employed herewith generates high quality segmentation outputs and the algorithm is assured to converge to an optimal solution. Consequently, experiments using real world image data reveal that Segmental HOG achieves significant improvements in detection performance compared to Rectangular HOG. The proposed descriptor and method for glomeruli detection present promising results and is expected to be useful in pathological evaluation.
6D pose estimation is the task of predicting the translation and orientation of objects in a given input image, which is a crucial prerequisite for many robotics and augmented reality applications. Lately, the Transformer Network architecture, equipped with a multi-head self-attention mechanism, is emerging to achieve state-of-the-art results in many computer vision tasks. DETR, a Transformer-based model, formulated object detection as a set prediction problem and achieved impressive results without standard components like region of interest pooling, non-maximal suppression, and bounding box proposals. In this work, we propose T6D-Direct, a real-time single-stage direct method with a transformer-based architecture built on DETR to perform 6D multi-object pose direct estimation. We evaluate the performance of our method on the YCB-Video dataset. Our method achieves the fastest inference time, and the pose estimation accuracy is comparable to state-of-the-art methods.
The great progress of 3D object detectors relies on large-scale data and 3D annotations. The annotation cost for 3D bounding boxes is extremely expensive while the 2D ones are easier and cheaper to collect. In this paper, we introduce a hybrid training framework, enabling us to learn a visual 3D object detector with massive 2D (pseudo) labels, even without 3D annotations. To break through the information bottleneck of 2D clues, we explore a new perspective: Temporal 2D Supervision. We propose a temporal 2D transformation to bridge the 3D predictions with temporal 2D labels. Two steps, including homography wraping and 2D box deduction, are taken to transform the 3D predictions into 2D ones for supervision. Experiments conducted on the nuScenes dataset show strong results (nearly 90% of its fully-supervised performance) with only 25% 3D annotations. We hope our findings can provide new insights for using a large number of 2D annotations for 3D perception.
Instance segmentation is of great importance for many biological applications, such as study of neural cell interactions, plant phenotyping, and quantitatively measuring how cells react to drug treatment. In this paper, we propose a novel box-based instance segmentation method. Box-based instance segmentation methods capture objects via bounding boxes and then perform individual segmentation within each bounding box region. However, existing methods can hardly differentiate the target from its neighboring objects within the same bounding box region due to their similar textures and low-contrast boundaries. To deal with this problem, in this paper, we propose an object-guided instance segmentation method. Our method first detects the center points of the objects, from which the bounding box parameters are then predicted. To perform segmentation, an object-guided coarse-to-fine segmentation branch is built along with the detection branch. The segmentation branch reuses the object features as guidance to separate target object from the neighboring ones within the same bounding box region. To further improve the segmentation quality, we design an auxiliary feature refinement module that densely samples and refines point-wise features in the boundary regions. Experimental results on three biological image datasets demonstrate the advantages of our method. The code will be available at https://github.com/yijingru/ObjGuided-Instance-Segmentation.
We study adapting trained object detectors to unseen domains manifesting significant variations of object appearance, viewpoints and backgrounds. Most current methods align domains by either using image or instance-level feature alignment in an adversarial fashion. This often suffers due to the presence of unwanted background and as such lacks class-specific alignment. A common remedy to promote class-level alignment is to use high confidence predictions on the unlabelled domain as pseudo labels. These high confidence predictions are often fallacious since the model is poorly calibrated under domain shift. In this paper, we propose to leverage model predictive uncertainty to strike the right balance between adversarial feature alignment and class-level alignment. Specifically, we measure predictive uncertainty on class assignments and the bounding box predictions. Model predictions with low uncertainty are used to generate pseudo-labels for self-supervision, whereas the ones with higher uncertainty are used to generate tiles for an adversarial feature alignment stage. This synergy between tiling around the uncertain object regions and generating pseudo-labels from highly certain object regions allows us to capture both the image and instance level context during the model adaptation stage. We perform extensive experiments covering various domain shift scenarios. Our approach improves upon existing state-of-the-art methods with visible margins.
Detecting objects in a two-dimensional setting is often insufficient in the context of real-life applications where the surrounding environment needs to be accurately recognized and oriented in three-dimension (3D), such as in the case of autonomous driving vehicles. Therefore, accurately and efficiently detecting objects in the three-dimensional setting is becoming increasingly relevant to a wide range of industrial applications, and thus is progressively attracting the attention of researchers. Building systems to detect objects in 3D is a challenging task though, because it relies on the multi-modal fusion of data derived from different sources. In this paper, we study the effects of anchoring using the current state-of-the-art 3D object detector and propose Class-specific Anchoring Proposal (CAP) strategy based on object sizes and aspect ratios based clustering of anchors. The proposed anchoring strategy significantly increased detection accuracy's by 7.19%, 8.13% and 8.8% on Easy, Moderate and Hard setting of the pedestrian class, 2.19%, 2.17% and 1.27% on Easy, Moderate and Hard setting of the car class and 12.1% on Easy setting of cyclist class. We also show that the clustering in anchoring process also enhances the performance of the regional proposal network in proposing regions of interests significantly. Finally, we propose the best cluster numbers for each class of objects in KITTI dataset that improves the performance of detection model significantly.
Label assignment in object detection aims to assign targets, foreground or background, to sampled regions in an image. Unlike labeling for image classification, this problem is not well defined due to the object's bounding box. In this paper, we investigate the problem from a perspective of distillation, hence we call Label Assignment Distillation (LAD). Our initial motivation is very simple, we use a teacher network to generate labels for the student. This can be achieved in two ways: either using the teacher's prediction as the direct targets (soft label), or through the hard labels dynamically assigned by the teacher (LAD). Our experiments reveal that: (i) LAD is more effective than soft-label, but they are complementary. (ii) Using LAD, a smaller teacher can also improve a larger student significantly, while soft-label can't. We then introduce Co-learning LAD, in which two networks simultaneously learn from scratch and the role of teacher and student are dynamically interchanged. Using PAA-ResNet50 as a teacher, our LAD techniques can improve detectors PAA-ResNet101 and PAA-ResNeXt101 to $46 \rm AP$ and $47.5\rm AP$ on the COCO test-dev set. With a stronger teacher PAA-SwinB, we improve the students PAA-ResNet50 to $43.7\rm AP$ by only 1x schedule training and standard setting, and PAA-ResNet101 to $47.9\rm AP$, significantly surpassing the current methods. Our source code and checkpoints are released at https://git.io/JrDZo.
개방형 어휘 객체 탐지(OVD)는 기본 클래스로 학습한 모델이 추론 시 기본 클래스와 신규 클래스를 모두 인식하기를 기대한다. 대표적인 방법들은 CLIP 방식의 비전-언어 임베딩에 의존하지만, 학습된 공간은 흔히 기본 범주에 편향되어 신규 개념을 위해 보존되는 유효 용량이 제한적이다. 이처럼 기본 범주가 지배하는 표현은 신규 의미를 압축하여 추론 시 의미 붕괴와 잘못 보정된 신뢰도를 초래할 수 있다. 이 문제를 해결하기 위해 본 논문은 신규 범주를 위한 용량을 명시적으로 확보하고 범주 분리를 강화하면서 다양한 탐지기 구조에 원활하게 통합될 수 있는 분할-정제 프레임워크인 PR-OVD를 제안한다. 구체적으로 PR-OVD는 먼저 시각 및 텍스트 양식 모두에서 기본, 신규, 그리고 명시적인 배경 분할에 대한 전역적 구분을 구축하는 거친 분할을 수행하여, 객체와 무관한 임베딩이 범주별 표현으로 유입되는 것을 방지한다. 이후 각 분할 내에서 범주 수준의 정제를 적용하여 결정 경계를 선명하게 하고 CLIP에서 세밀한 영역 수준 사전학습이 부족한 문제를 완화한다. 신규 범주를 위한 신뢰할 수 있는 시각적 기준점을 추가로 제공하기 위해, 자기중심적 속성과 타자중심적 맥락을 모두 부호화하는 인지적으로 영감을 받은 구조 프롬프트(SP)를 설계하여 안정적인 분할-정제 학습을 돕는다. 두 OVD 벤치마크에 대한 폭넓은 실험에서 PR-OVD는 최신 방법 대비 신규 범주에서 평균 6.5%의 비례적 향상을 달성한다. 특히 PR-OVD는 분할 및 분류를 포함한 다른 개방형 어휘 과제에서도 이점을 보여 그 효과성과 일반성을 더욱 확인한다.
Simultaneous Localization And Mapping (SLAM) is a fundamental problem in mobile robotics. While sparse point-based SLAM methods provide accurate camera localization, the generated maps lack semantic information. On the other hand, state of the art object detection methods provide rich information about entities present in the scene from a single image. This work incorporates a real-time deep-learned object detector to the monocular SLAM framework for representing generic objects as quadrics that permit detections to be seamlessly integrated while allowing the real-time performance. Finer reconstruction of an object, learned by a CNN network, is also incorporated and provides a shape prior for the quadric leading further refinement. To capture the dominant structure of the scene, additional planar landmarks are detected by a CNN-based plane detector and modeled as independent landmarks in the map. Extensive experiments support our proposed inclusion of semantic objects and planar structures directly in the bundle-adjustment of SLAM - Semantic SLAM - that enriches the reconstructed map semantically, while significantly improving the camera localization. The performance of our SLAM system is demonstrated in https://youtu.be/UMWXd4sHONw and https://youtu.be/QPQqVrvP0dE .
We propose a text-guided variational image generation method to address the challenge of getting clean data for anomaly detection in industrial manufacturing. Our method utilizes text information about the target object, learned from extensive text library documents, to generate non-defective data images resembling the input image. The proposed framework ensures that the generated non-defective images align with anticipated distributions derived from textual and image-based knowledge, ensuring stability and generality. Experimental results demonstrate the effectiveness of our approach, surpassing previous methods even with limited non-defective data. Our approach is validated through generalization tests across four baseline models and three distinct datasets. We present an additional analysis to enhance the effectiveness of anomaly detection models by utilizing the generated images.
The technology of vehicle and driver detection in Intelligent Transportation System(ITS) is a hot topic in recent years. In particular, the driver detection is still a challenging problem which is conductive to supervising traffic order and maintaining public safety. In this paper, an algorithm based on YOLOv3 is proposed to realize the detection and classification of vehicles, drivers, and people on the highway, so as to achieve the purpose of distinguishing driver and passenger and form a one-to-one correspondence between vehicles and drivers. The proposed model and contrast experiment are conducted on our self-build traffic driver's face database. The effectiveness of our proposed algorithm is validated by extensive experiments and verified under various complex highway conditions. Compared with other advanced vehicle and driver detection technologies, the model has a good performance and is robust to road blocking, different attitudes, and extreme lighting.