Estimation of 3D motion in a dynamic scene from a temporal pair of images is a core task in many scene understanding problems. In real world applications, a dynamic scene is commonly captured by a moving camera (i.e., panning, tilting or hand-held), increasing the task complexity because the scene is observed from different view points. The main challenge is the disambiguation of the camera motion from scene motion, which becomes more difficult as the amount of rigidity observed decreases, even with successful estimation of 2D image correspondences. Compared to other state-of-the-art 3D scene flow estimation methods, in this paper we propose to \emph{learn} the rigidity of a scene in a supervised manner from a large collection of dynamic scene data, and directly infer a rigidity mask from two sequential images with depths. With the learned network, we show how we can effectively estimate camera motion and projected scene flow using computed 2D optical flow and the inferred rigidity mask. For training and testing the rigidity network, we also provide a new semi-synthetic dynamic scene dataset (synthetic foreground objects with a real background) and an evaluation split that accounts for the percentage of observed non-rigid pixels. Through our evaluation we show the proposed framework outperforms current state-of-the-art scene flow estimation methods in challenging dynamic scenes.
In remote sensing, each sensor can provide complementary or reinforcing information. It is valuable to fuse outputs from multiple sensors to boost overall performance. Previous supervised fusion methods often require accurate labels for each pixel in the training data. However, in many remote sensing applications, pixel-level labels are difficult or infeasible to obtain. In addition, outputs from multiple sensors often have different resolution or modalities. For example, rasterized hyperspectral imagery presents data in a pixel grid while airborne Light Detection and Ranging (LiDAR) generates dense three-dimensional (3D) point clouds. It is often difficult to directly fuse such multi-modal, multi-resolution data. To address these challenges, we present a novel Multiple Instance Multi-Resolution Fusion (MIMRF) framework that can fuse multi-resolution and multi-modal sensor outputs while learning from automatically-generated, imprecisely-labeled data. Experiments were conducted on the MUUFL Gulfport hyperspectral and LiDAR data set and a remotely-sensed soybean and weed data set. Results show improved, consistent performance on scene understanding and agricultural applications when compared to traditional fusion methods.
Objects often occlude each other in scenes; Inferring their appearance beyond their visible parts plays an important role in scene understanding, depth estimation, object interaction and manipulation. In this paper, we study the challenging problem of completing the appearance of occluded objects. Doing so requires knowing which pixels to paint (segmenting the invisible parts of objects) and what color to paint them (generating the invisible parts). Our proposed novel solution, SeGAN, jointly optimizes for both segmentation and generation of the invisible parts of objects. Our experimental results show that: (a) SeGAN can learn to generate the appearance of the occluded parts of objects; (b) SeGAN outperforms state-of-the-art segmentation baselines for the invisible parts of objects; (c) trained on synthetic photo realistic images, SeGAN can reliably segment natural images; (d) by reasoning about occluder occludee relations, our method can infer depth layering.
Human visual scene understanding is so remarkable that we are able to recognize a revisited place when entering it from the opposite direction it was first visited, even in the presence of extreme variations in appearance. This capability is especially apparent during driving: a human driver can recognize where they are when travelling in the reverse direction along a route for the first time, without having to turn back and look. The difficulty of this problem exceeds any addressed in past appearance- and viewpoint-invariant visual place recognition (VPR) research, in part because large parts of the scene are not commonly observable from opposite directions. Consequently, as shown in this paper, the precision-recall performance of current state-of-the-art viewpoint- and appearance-invariant VPR techniques is orders of magnitude below what would be usable in a closed-loop system. Current engineered solutions predominantly rely on panoramic camera or LIDAR sensing setups; an eminently suitable engineering solution but one that is clearly very different to how humans navigate, which also has implications for how naturally humans could interact and communicate with the navigation system. In this paper we develop a suite of novel semantic- and appearance-based techniques to enable for the first time high performance place recognition in this challenging scenario. We first propose a novel Local Semantic Tensor (LoST) descriptor of images using the convolutional feature maps from a state-of-the-art dense semantic segmentation network. Then, to verify the spatial semantic arrangement of the top matching candidates, we develop a novel approach for mining semantically-salient keypoint correspondences.
Per-pixel ground-truth depth data is challenging to acquire at scale. To overcome this limitation, self-supervised learning has emerged as a promising alternative for training models to perform monocular depth estimation. In this paper, we propose a set of improvements, which together result in both quantitatively and qualitatively improved depth maps compared to competing self-supervised methods. Research on self-supervised monocular training usually explores increasingly complex architectures, loss functions, and image formation models, all of which have recently helped to close the gap with fully-supervised methods. We show that a surprisingly simple model, and associated design choices, lead to superior predictions. In particular, we propose (i) a minimum reprojection loss, designed to robustly handle occlusions, (ii) a full-resolution multi-scale sampling method that reduces visual artifacts, and (iii) an auto-masking loss to ignore training pixels that violate camera motion assumptions. We demonstrate the effectiveness of each component in isolation, and show high quality, state-of-the-art results on the KITTI benchmark.
이미지로부터 객체의 자세(pose)를 추정하는 것은 3D 장면 이해(3D scene understanding)의 핵심적인 작업이며, 최근의 접근법들은 매우 큰 벤치마크(benchmark)에서 유망한 결과를 보여주었다. 그러나 이러한 방법들은 미확인 객체(unseen object)를 다룰 때 성능이 크게 저하된다. 이 문제를 해결하기 위해, 우리는 미확인 객체를 모델링할 상당한 잠재력을 지닌 디퓨전 모델(diffusion model)(예: Stable Diffusion)의 특징(feature)에 대해 심도 있는 분석을 수행하였다. 이 분석을 바탕으로, 우리는 객체 자세 추정을 위해 이러한 디퓨전 특징을 혁신적으로 도입하였다. 객체 자세 추정에 대한 디퓨전 특징의 효능을 검증하기 위해, 디퓨전 특징을 포착하고 집계하는 세 가지의 서로 다른 아키텍처(vanilla, nonlinear, context-aware 가중치 집계)를 제안하여 비교 분석을 수행하였다. 효율적인 특징 집계를 달성하기 위해, 모든 특징을 사용하는 대신 판별력 있는 특징을 자동으로 선택하는 신뢰도 적응형 집계 네트워크(confidence adaptive aggregation network)를 제안하여, 더 나은 속도-정확도 트레이드오프(trade-off)를 달성하였다. 특히, 우리의 신뢰도 적응형 집계 네트워크는 미확인 객체에서 기존 최고 성능(previous best arts)보다 더 높은 정확도를 달성하였다: Unseen LM에서 97.7% 대 93.5%, Unseen O-LM에서 85.5% 대 76.3%로, 우리 방법의 강력한 일반화 능력(generalizability)을 보여준다. 대규모 BOP 벤치마크에서도 우리의 방법은 측정 가능한 향상을 제공하여, 평균 재현율(recall)이 기존의 57.9에서 58.3으로 증가하였다. 또한, CAA는 CWA 변형(variant)에 비해 비교 가능한 정확도를 유지하면서 계산 비용을 1.3~1.5배 절감한다. 더욱이, CAA는 68 FPS 이상을 달성하며 실시간(real-time) 성능에 도달하여, 정확도-효율성 트레이드오프를 상당히 개선하였다.
RS-Agent는 구조화된 작업 계획과 도구 오케스트레이션을 통해 사용자 의도를 전문적인 원격 탐사 워크플로우와 연결하는 도메인 적응형 지능 에이전트로, 중앙 컨트롤러, 동적 툴킷, Solution Space, Knowledge Space의 네 가지 구성요소로 이루어진다. Task-Aware Retrieval과 가중 이중 경로 검색-증강 생성 방법인 DualRAG를 도입하여 9개 데이터셋, 18개 원격 탐사 작업 실험에서 95% 이상의 작업 계획 정확도와 함께 장면 분류, 객체 계수, 시각적 질의응답에서 최신 MLLM 대비 현저히 높은 성능을 달성하였다.
We study multi-sensor fusion for 3D semantic segmentation that is important to scene understanding for many applications, such as autonomous driving and robotics. Existing fusion-based methods, however, may not achieve promising performance due to the vast difference between the two modalities. In this work, we investigate a collaborative fusion scheme called perception-aware multi-sensor fusion (PMF) to effectively exploit perceptual information from two modalities, namely, appearance information from RGB images and spatio-depth information from point clouds. To this end, we project point clouds to the camera coordinate using perspective projection, and process both inputs from LiDAR and cameras in 2D space while preventing the information loss of RGB images. Then, we propose a two-stream network to extract features from the two modalities, separately. The extracted features are fused by effective residual-based fusion modules. Moreover, we introduce additional perception-aware losses to measure the perceptual difference between the two modalities. Last, we propose an improved version of PMF, i.e., EPMF, which is more efficient and effective by optimizing data pre-processing and network architecture under perspective projection. Specifically, we propose cross-modal alignment and cropping to obtain tight inputs and reduce unnecessary computational costs. We then explore more efficient contextual modules under perspective projection and fuse the LiDAR features into the camera stream to boost the performance of the two-stream network. Extensive experiments on benchmark data sets show the superiority of our method. For example, on nuScenes test set, our EPMF outperforms the state-of-the-art method, i.e., RangeFormer, by 0.9% in mIoU. Our source code is available at https://github.com/ICEORY/PMF.