The Segment Anything Model 2 (SAM2), a prompt-guided video foundation model, has remarkably performed in video object segmentation, drawing significant attention in the community. Due to the high similarity between camouflaged objects and their surroundings, which makes them difficult to distinguish even by the human eye, the application of SAM2 for automated segmentation in real-world scenarios faces challenges in camouflage perception and reliable prompts generation. To address these issues, we propose CamoSAM2, a motion-appearance prompt inducer (MAPI) and refinement framework to automatically generate and refine prompts for SAM2, enabling high-quality automatic detection and segmentation in VCOD task. Initially, we introduce a prompt inducer that simultaneously integrates motion and appearance cues to detect camouflaged objects, delivering more accurate initial predictions than existing methods. Subsequently, we propose a video-based adaptive multi-prompts refinement (AMPR) strategy tailored for SAM2, aimed at mitigating prompt error in initial coarse masks and further producing good prompts. Specifically, we introduce a novel three-step process to generate reliable prompts by camouflaged object determination, pivotal prompting frame selection, and multi-prompts formation. Extensive experiments conducted on two benchmark datasets demonstrate that our proposed model, CamoSAM2, significantly outperforms existing state-of-the-art methods, achieving increases of 8.0% and 10.1% in mIoU metric. Additionally, our method achieves the fastest inference speed compared to current VCOD models.
멀티모달 대규모 언어 모델(MLLM)의 제로샷(zero-shot) 능력은 돌출 객체 검출(SOD)을 과제 특화형 지도학습을 넘어서는 단계로 끌어올리고 있습니다. 기존의 마스크 기반 평가를 넘어 MLLM을 면밀히 분석하기 위해, 본 연구에서는 SOD를 위치 파악(localization)과 분할(segmentation)로 분해하고 프레이즈(phrase), 바운딩 박스(box), 속성(attribute)을 기반으로 데이터셋을 재구성하여 MLLM의 돌출성 지각 능력을 진단하는 벤치마크인 SaliLLM을 구축합니다. SaliLLM은 주목할 만한 능력의 불일치를 드러냅니다. MLLM은 위치 파악에서는 최고 수준(SOTA) 방법론을 능가하지만, 분할에서는 여전히 현저하게 약한 모습을 보입니다. 추가 분석 결과, 이러한 격차는 주로 전경의 기수(cardinality), 세분성(granularity), 영역 범위(extent)에 대해 MLLM과 주석(annotation) 사이의 불일치에서 기인함을 확인했습니다. 이러한 진단에 착안하여, 우리는 제로샷 SOD를 프로토콜에 정렬된 전경 조직화(Foreground Organization)로 재구성하고, 게슈탈트(Gestalt)에서 영감을 얻은 협력적 어텐션을 활용하는 최초의 학습(training) 불필요 통합 SOD 프레임워크인 FOCUS를 제안합니다. FOCUS는 프로토콜 기반 전경 세분성에 대한 하향식 베이지안 서프라이즈(Bayesian-surprise) 캘리브레이션과, 자기지도 학습 특징으로 유도된 엔티티 중심 지각 매니폴드 상에서 MLLM 근거의 상향식 전파를 결합하여 일관된 객체 영역을 생성하고 이를 범용 분할기의 프롬프트로 활용합니다. 13개의 RGB, RGB-D, RGB-T SOD 벤치마크 전반에서 FOCUS는 추가 학습 없이도 기존 SOTA 방법론들을 대체로 능가하며, 완전 지도, 준지도, 자기지도 학습 방법론 대비 평균 절대 오차(MAE)를 각각 11%, 34%, 48% 줄였습니다. 본 연구의 발견은 SOD가 과제 특화형 지도학습에서 제로샷 전경 조직화로 나아가는 르네상스를 알립니다. 코드는 보충 자료에서 제공됩니다.
본 연구는 손상 평가를 위한 통합 멀티모달 AI 시스템을 제시하며, 검색-증augmented 생성(RAG) 모델, 열 스펙트럼 인식, 비전 파운데이션 모델 파이프라인, 탐색적 무선 신호 센싱을 통합한다. RAG 구성요소는 추론 시 환각을 완화하기 위해 손상 등급 분류 기준을 포함한 프로젝트별 문서에 로컬 호스팅 언어 모델을 그라운딩하도록 개발되었다. 정적 퓨샷(few-shot) 프롬프트에 대한 대조 실험을 통해 동적 검색이 그라운딩과 사실적 일관성을 향상시킴을 보인다. 또한 벡터 기반 RAG와 개체-관계 추출로 구축한 지식 그래프 변형을 비교하였으며, 문서 간 추론이 필요한 손상 평가 질의에 대해 그래프 기반 검색이 더 강한 응답을 생성함을 보여 하이브리드 dense·sparse·그래프 인식 검색의 필요성을 뒷받침한다. 불리한 조명 및 기상 조건에서 EO(전자광학) 영상의 한계를 보완하기 위해 적외선(IR)/열 센싱을 객체 검출 및 분할에 사용한다. 검출기는 후보 검출을 생성하여 광범위한 객체에 대한 분할을 개선한다. IR 대 가시광 스펙트럼 추적의 페어링 실험은 실패 양상을 드러내어, 견고한 객체 검출 및 손상 분석을 위한 멀티모달 퓨전을 촉진한다. 비전 파운데이션 및 비전-언어 모델을 활용해 합성 손상 영상을 생성하고 손상 심각도를 높은 정확도로 분류하여, 하류 손상 평가 모델의 학습과 검증을 지원한다. 마지막으로, 탐색적 무선 기반 센싱은 EO 및 IR 센싱이 효과적이지 않은 경우 존재, 움직임, 사건 이후 환경 변화를 검출할 잠재력을 보여준다.
An edge detection is important for its reliability and security which delivers a better understanding of object recognition in the applications of computer vision, such as pedestrian detection, face detection, and video surveillance. This paper introduced two fundamental limitations encountered in edge detection: edge connectivity and edge thickness, those have been used by various developments in the state-of-the-art. An optimal selection of the threshold for effectual edge detection has constantly been a key challenge in computer vision. Therefore, a robust edge detection algorithm using multiple threshold approaches (B-Edge) is proposed to cover both the limitations. The majorly used canny edge operator focuses on two thresholds selections and still witnesses a few gaps for optimal results. To handle the loopholes of the canny edge operator, our method selects the simulated triple thresholds that target to the prime issues of the edge detection: image contrast, effective edge pixels selection, errors handling, and similarity to the ground truth. The qualitative and quantitative experimental evaluations demonstrate that our edge detection method outperforms competing algorithms for mentioned issues. The proposed approach endeavors an improvement for both grayscale and colored images.
Craters are one of the most prominent features on planetary surfaces, used in applications such as age estimation, hazard detection, and spacecraft navigation. Crater detection is a challenging problem due to various aspects, including complex crater characteristics such as varying sizes and shapes, data resolution, and planetary data types. Similar to other computer vision tasks, deep learning-based approaches have significantly impacted research on crater detection in recent years. This survey aims to assist researchers in this field by examining the development of deep learning-based crater detection algorithms (CDAs). The review includes over 140 research works covering diverse crater detection approaches, including planetary data, craters database, and evaluation metrics. To be specific, we discuss the challenges in crater detection due to the complex properties of the craters and survey the DL-based CDAs by categorizing them into three parts: (a) semantic segmentation-based, (b) object detection-based, and (c) classification-based. Additionally, we have conducted training and testing of all the semantic segmentation-based CDAs on a common dataset to evaluate the effectiveness of each architecture for crater detection and its potential applications. Finally, we have provided recommendations for potential future works.
The field of machine learning has been greatly transformed with the advancement of deep artificial neural networks (ANNs) and the increased availability of annotated data. Spiking neural networks (SNNs) have recently emerged as a low-power alternative to ANNs due to their sparsity nature. In this work, we propose a novel hybrid ANN-SNN co-training framework to improve the performance of converted SNNs. Our approach is a fine-tuning scheme, conducted through an alternating, forward-backward training procedure. We apply our framework to object detection and image segmentation tasks. Experiments demonstrate the effectiveness of our approach in achieving the design goals.
With the emergence of foundation models, deep learning-based object detectors have shown practical usability in closed set scenarios. However, for real-world tasks, object detectors often operate in open environments, where crucial factors (e.g., data distribution, objective) that influence model learning are often changing. The dynamic and intricate nature of the open environment poses novel and formidable challenges to object detectors. Unfortunately, current research on object detectors in open environments lacks a comprehensive analysis of their distinctive characteristics, challenges, and corresponding solutions, which hinders their secure deployment in critical real-world scenarios. This paper aims to bridge this gap by conducting a comprehensive review and analysis of object detectors in open environments. We initially identified limitations of key structural components within the existing detection pipeline and propose the open environment object detector challenge framework that includes four quadrants (i.e., out-of-domain, out-of-category, robust learning, and incremental learning) based on the dimensions of the data / target changes. For each quadrant of challenges in the proposed framework, we present a detailed description and systematic analysis of the overarching goals and core difficulties, systematically review the corresponding solutions, and benchmark their performance over multiple widely adopted datasets. In addition, we engage in a discussion of open problems and potential avenues for future research. This paper aims to provide a fresh, comprehensive, and systematic understanding of the challenges and solutions associated with open-environment object detectors, thus catalyzing the development of more solid applications in real-world scenarios. A project related to this survey can be found at https://github.com/LiangSiyuan21/OEOD_Survey.
외부 리소스와 통합된 대규모 언어 모델(LLM) 에이전트는 복잡한 태스크 수행 능력을 얻지만, 통합된 자연어 컨텍스트 채널 때문에 주입 공격(injection attack)에 취약하다. 신뢰할 수 없는 외부 데이터가 LLM 추론 과정에서 동적으로 행동 유도 지시문(behavior-guiding instruction)으로 파싱되어 에이전트의 의사결정을 전복시킬 수 있다. 기존 방어는 입력·출력 수준에서 악성 콘텐츠를 정적으로 탐지하거나 격리하는 데 초점을 맞추어, 모델 추론 중에 발생하는 이러한 동적 유도를 탐지하기에는 불충분하다. 우리는 도구 호출(tool-calling) 결정에 실질적으로 영향을 미치는 컨텍스트 구간, 즉 행동 유도 지시문을 세밀하게 지역화하는 런타임 프레임워크 Attnlocate를 제안한다. Attnlocate는 이 지역화 문제를 객체 검출(object detection) 과제로 형식화하여, 어텐션(attention) 행렬 내에서 행동 유도 지시문이 유발하는 distinctive activation trace를 검출하는 것을 목표로 한다. 구체적으로, 객체 검출에 맞춘 토큰(token) 수준 특징(feature) 공간을 구성하기 위해 다중 헤드·다층(layer) 어텐션 집계 방식을 설계한다. 이어서 anchor-free 검출 헤드를 갖춘 1-D U-Net을 배치하여 이러한 구간을 검출한다. 마지막으로, 검출된 행동 유도 지시문 구간의 출처 제공자(provider) 권한(authority)에 따라 Attnlocate는 악의적 호출 시도를 동적으로 판정한다. Attnlocate를 5개 LLM 계열의 10가지 에이전트 구성에 걸쳐 평가했으며, 간접 프롬프트 주입(indirect prompt injection) 및 도구 오염(tool poisoning) 시나리오를 포함한다. Attnlocate는 평균 IoU 0.743, 평균 AUROC 0.956, 위양성률(false-positive rate) 0.067에서 참양성률(true-positive rate) 0.934를 달성한다. 또한 보지 못한 모델(model)에도 효과적으로 전이되며, 재학습 없이 권한 정책 적응을 지원한다.
During 3D reconstruction, it is often the case that people cannot scan each individual object from all views, resulting in missing geometry in the captured scan. This missing geometry can be fundamentally limiting for many applications, e.g., a robot needs to know the unseen geometry to perform a precise grasp on an object. Thus, we introduce the task of semantic instance completion: from an incomplete RGB-D scan of a scene, we aim to detect the individual object instances and infer their complete object geometry. This will open up new possibilities for interactions with objects in a scene, for instance for virtual or robotic agents. We tackle this problem by introducing RevealNet, a new data-driven approach that jointly detects object instances and predicts their complete geometry. This enables a semantically meaningful decomposition of a scanned scene into individual, complete 3D objects, including hidden and unobserved object parts. RevealNet is an end-to-end 3D neural network architecture that leverages joint color and geometry feature learning. The fully-convolutional nature of our 3D network enables efficient inference of semantic instance completion for 3D scans at scale of large indoor environments in a single forward pass. We show that predicting complete object geometry improves both 3D detection and instance segmentation performance. We evaluate on both real and synthetic scan benchmark data for the new task, where we outperform state-of-the-art approaches by over 15 in [email omitted] on ScanNet, and over 18 in [email omitted] on SUNCG.
Video surveillance systems have been installed to ensure the student safety in schools. However, discovering dangerous behaviors, such as fighting and falling down, usually depends on untimely human observations. In this paper, we focus on detecting dangerous behaviors of students automatically, which faces numerous challenges, such as insufficient datasets, confusing postures, keyframes detection and prompt response. To address these challenges, we first build a danger behavior dataset with locations and labels from surveillance videos, and transform action recognition of long videos to an object detection task that avoids keyframes detection. Then, we propose a novel end-to-end dangerous behavior detection method, named DangerDet, that combines multi-scale body features and keypoints-based pose features. We could improve the accuracy of behavior classification due to the highly correlation between pose and behavior. On our dataset, DangerDet achieves 71.0\% mAP with about 11 FPS. It keeps a better balance between the accuracy and time cost.
When it comes to addressing the safety/security related needs at different production/construction sites, accurate detection of the presence of workers, vehicles, equipment important and formed an integral part of computer vision-based surveillance systems (CVSS). Traditional CVSS systems focus on the use of different computer vision and pattern recognition algorithms overly reliant on manual extraction of features and small datasets, limiting their usage because of low accuracy, need for expert knowledge and high computational costs. The main objective of this paper is to provide decision makers at sites with a practical yet comprehensive deep learning and IoT based solution to tackle various computer vision related problems such as scene classification, object detection in scenes, semantic segmentation, scene captioning etc. Our overarching goal is to address the central question of What is happening at this site and where is it happening in an automated fashion minimizing the need for human resources dedicated to surveillance. We developed Deep ExxonMobil Eye for Video Analysis (DEEVA) package to handle scene classification, object detection, semantic segmentation and captioning of scenes in a hierarchical approach. The results reveal that transfer learning with the RetinaNet object detector is able to detect the presence of workers, different types of vehicles/construction equipment, safety related objects at a high level of accuracy (above 90%). With the help of deep learning to automatically extract features and IoT technology to automatic capture, transfer and process vast amount of realtime images, this framework is an important step towards the development of intelligent surveillance systems aimed at addressing myriads of open ended problems in the realm of security/safety monitoring, productivity assessments and future decision making.
3D semantic occupancy prediction aims to reconstruct the 3D geometry and semantics of the surrounding environment. With dense voxel labels, prior works typically formulate it as a dense segmentation task, independently classifying each voxel. However, this paradigm neglects critical instance-centric discriminability, leading to instance-level incompleteness and adjacent ambiguities. To address this, we highlight a free lunch of occupancy labels: the voxel-level class label implicitly provides insight at the instance level, which is overlooked by the community. Motivated by this observation, we first introduce a training-free Voxel-to-Instance (VoxNT) trick: a simple yet effective method that freely converts voxel-level class labels into instance-level offset labels. Building on this, we further propose VoxDet, an instance-centric framework that reformulates the voxel-level occupancy prediction as dense object detection by decoupling it into two sub-tasks: offset regression and semantic prediction. Specifically, based on the lifted 3D volume, VoxDet first uses (a) Spatially-decoupled Voxel Encoder to generate disentangled feature volumes for the two sub-tasks, which learn task-specific spatial deformation in the densely projected tri-perceptive space. Then, we deploy (b) Task-decoupled Dense Predictor to address this task via dense detection. Here, we first regress a 4D offset field to estimate distances (6 directions) between voxels and object borders in the voxel space. The regressed offsets are then used to guide the instance-level aggregation in the classification branch, achieving instance-aware prediction. Experiments show that VoxDet can be deployed on both camera and LiDAR input, jointly achieving state-of-the-art results on both benchmarks. VoxDet is not only highly efficient, but also achieves 63.0 IoU on the SemanticKITTI test set, ranking 1st on the online leaderboard.
Relations amongst entities play a central role in image understanding. Due to the complexity of modeling (subject, predicate, object) relation triplets, it is crucial to develop a method that can not only recognize seen relations, but also generalize to unseen cases. Inspired by a previously proposed visual translation embedding model, or VTransE, we propose a context-augmented translation embedding model that can capture both common and rare relations. The previous VTransE model maps entities and predicates into a low-dimensional embedding vector space where the predicate is interpreted as a translation vector between the embedded features of the bounding box regions of the subject and the object. Our model additionally incorporates the contextual information captured by the bounding box of the union of the subject and the object, and learns the embeddings guided by the constraint predicate $\approx$ union (subject, object) $-$ subject $-$ object. In a comprehensive evaluation on multiple challenging benchmarks, our approach outperforms previous translation-based models and comes close to or exceeds the state of the art across a range of settings, from small-scale to large-scale datasets, from common to previously unseen relations. It also achieves promising results for the recently introduced task of scene graph generation.
This work proposes a novel pose estimation model for object categories that can be effectively transferred to previously unseen environments. The deep convolutional network models (CNN) for pose estimation are typically trained and evaluated on datasets specifically curated for object detection, pose estimation, or 3D reconstruction, which requires large amounts of training data. In this work, we propose a model for pose estimation that can be trained with small amount of data and is built on the top of generic mid-level representations \cite{taskonomy2018} (e.g. surface normal estimation and re-shading). These representations are trained on a large dataset without requiring pose and object annotations. Later on, the predictions are refined with a small CNN neural network that exploits object masks and silhouette retrieval. The presented approach achieves superior performance on the Pix3D dataset \cite{pix3d} and shows nearly 35\% improvement over the existing models when only 25\% of the training data is available. We show that the approach is favorable when it comes to generalization and transfer to novel environments. Towards this end, we introduce a new pose estimation benchmark for commonly encountered furniture categories on challenging Active Vision Dataset \cite{Ammirato2017ADF} and evaluated the models trained on the Pix3D dataset.
Multimodal Large Language Models (MLLMs) such as GPT-4V and Gemini Pro face challenges in achieving human-level perception in Visual Question Answering (VQA), particularly in object-oriented perception tasks which demand fine-grained understanding of object identities, locations or attributes, as indicated by empirical findings. This is mainly due to their limited capability to effectively integrate complex visual cues with textual information and potential object hallucinations. In this paper, we present a novel approach, Joint Visual and Text Prompting (VTPrompt), that employs fine-grained visual information to enhance the capability of MLLMs in VQA, especially for object-oriented perception. VTPrompt merges visual and text prompts to extract key concepts from textual questions and employs a detection model to highlight relevant objects as visual prompts in images. The processed images alongside text prompts are subsequently fed into MLLMs to produce more accurate answers. Our experiments with GPT-4V and Gemini Pro, on three benchmarks, i.e., MME , MMB and POPE, demonstrate significant improvements. Particularly, our method led to a score improvement of up to 183.5 for GPT-4V on MME and enhanced MMB performance by 8.17\% for GPT-4V and 15.69\% for Gemini Pro.
Abnormal event detection in video is a complex computer vision problem that has attracted significant attention in recent years. The complexity of the task arises from the commonly-adopted definition of an abnormal event, that is, a rarely occurring event that typically depends on the surrounding context. Following the standard formulation of abnormal event detection as outlier detection, we propose a background-agnostic framework that learns from training videos containing only normal events. Our framework is composed of an object detector, a set of appearance and motion auto-encoders, and a set of classifiers. Since our framework only looks at object detections, it can be applied to different scenes, provided that normal events are defined identically across scenes and that the single main factor of variation is the background. To overcome the lack of abnormal data during training, we propose an adversarial learning strategy for the auto-encoders. We create a scene-agnostic set of out-of-domain pseudo-abnormal examples, which are correctly reconstructed by the auto-encoders before applying gradient ascent on the pseudo-abnormal examples. We further utilize the pseudo-abnormal examples to serve as abnormal examples when training appearance-based and motion-based binary classifiers to discriminate between normal and abnormal latent features and reconstructions. We compare our framework with the state-of-the-art methods on four benchmark data sets, using various evaluation metrics. Compared to existing methods, the empirical results indicate that our approach achieves favorable performance on all data sets. In addition, we provide region-based and track-based annotations for two large-scale abnormal event detection data sets from the literature, namely ShanghaiTech and Subway.
Neural networks have enabled state-of-the-art approaches to achieve incredible results on computer vision tasks such as object detection. However, such success greatly relies on costly computation resources, which hinders people with cheap devices from appreciating the advanced technology. In this paper, we propose Cross Stage Partial Network (CSPNet) to mitigate the problem that previous works require heavy inference computations from the network architecture perspective. We attribute the problem to the duplicate gradient information within network optimization. The proposed networks respect the variability of the gradients by integrating feature maps from the beginning and the end of a network stage, which, in our experiments, reduces computations by 20% with equivalent or even superior accuracy on the ImageNet dataset, and significantly outperforms state-of-the-art approaches in terms of AP50 on the MS COCO object detection dataset. The CSPNet is easy to implement and general enough to cope with architectures based on ResNet, ResNeXt, and DenseNet. Source code is at https://github.com/WongKinYiu/CrossStagePartialNetworks.
Multiple objects tracking (MOT) is a difficult task, as it usually requires special hardware and higher computation complexity. In this work, we present a new framework of MOT by using of equilibrium optimizer (EO) algorithm and reducing the resolution of the bounding boxes of the objects to solve such problems in the detection free framework. First, in the first frame the target objects are initialized and its size is computed, then its resolution is reduced if it is higher than a threshold, and then modeled by their kernel color histogram to establish a feature model. The Bhattacharya distances between the histogram of object models and other candidates are used as the fitness function to be optimized. Multiple agents are generated by EO, according to the number of the target objects to be tracked. EO algorithm is used because of its efficiency and lower computation cost compared to other algorithms in global optimization. Experimental results confirm that EO multi-object tracker achieves satisfying tracking results then other trackers.
Self-training for unsupervised domain adaptive object detection is a challenging task, of which the performance depends heavily on the quality of pseudo boxes. Despite the promising results, prior works have largely overlooked the uncertainty of pseudo boxes during self-training. In this paper, we present a simple yet effective framework, termed as Probabilistic Teacher (PT), which aims to capture the uncertainty of unlabeled target data from a gradually evolving teacher and guides the learning of a student in a mutually beneficial manner. Specifically, we propose to leverage the uncertainty-guided consistency training to promote classification adaptation and localization adaptation, rather than filtering pseudo boxes via an elaborate confidence threshold. In addition, we conduct anchor adaptation in parallel with localization adaptation, since anchor can be regarded as a learnable parameter. Together with this framework, we also present a novel Entropy Focal Loss (EFL) to further facilitate the uncertainty-guided self-training. Equipped with EFL, PT outperforms all previous baselines by a large margin and achieve new state-of-the-arts.
Autonomous vehicles rely on LiDAR sensors to detect obstacles such as pedestrians, other vehicles, and fixed infrastructures. LiDAR spoofing attacks have been demonstrated that either create erroneous obstacles or prevent detection of real obstacles, resulting in unsafe driving behaviors. In this paper, we propose an approach to detect and mitigate LiDAR spoofing attacks by leveraging LiDAR scan data from other neighboring vehicles. This approach exploits the fact that spoofing attacks can typically only be mounted on one vehicle at a time, and introduce additional points into the victim's scan that can be readily detected by comparison from other, non-modified scans. We develop a Fault Detection, Identification, and Isolation procedure that identifies non-existing obstacle, physical removal, and adversarial object attacks, while also estimating the actual locations of obstacles. We propose a control algorithm that guarantees that these estimated object locations are avoided. We validate our framework using a CARLA simulation study, in which we verify that our FDII algorithm correctly detects each attack pattern.