Conventional image motion based structure from motion methods first compute optical flow, then solve for the 3D motion parameters based on the epipolar constraint, and finally recover the 3D geometry of the scene. However, errors in optical flow due to regularization can lead to large errors in 3D motion and structure. This paper investigates whether performance and consistency can be improved by avoiding optical flow estimation in the early stages of the structure from motion pipeline, and it proposes a new direct method based on image gradients (normal flow) only. The main idea lies in a reformulation of the positive-depth constraint, which allows the use of well-known minimization techniques to solve for 3D motion. The 3D motion estimate is then refined and structure estimated adding a regularization based on depth. Experimental comparisons on standard synthetic datasets and the real-world driving benchmark dataset KITTI using three different optic flow algorithms show that the method achieves better accuracy in all but one case. Furthermore, it outperforms existing normal flow based 3D motion estimation techniques. Finally, the recovered 3D geometry is shown to be also very accurate.
Non-contact video-based physiological measurement has many applications in health care and human-computer interaction. Practical applications require measurements to be accurate even in the presence of large head rotations. We propose the first end-to-end system for video-based measurement of heart and breathing rate using a deep convolutional network. The system features a new motion representation based on a skin reflection model and a new attention mechanism using appearance information to guide motion estimation, both of which enable robust measurement under heterogeneous lighting and major motions. Our approach significantly outperforms all current state-of-the-art methods on both RGB and infrared video datasets. Furthermore, it allows spatial-temporal distributions of physiological signals to be visualized via the attention mechanism.
We address the unsupervised learning of several interconnected problems in low-level vision: single view depth prediction, camera motion estimation, optical flow, and segmentation of a video into the static scene and moving regions. Our key insight is that these four fundamental vision problems are coupled through geometric constraints. Consequently, learning to solve them together simplifies the problem because the solutions can reinforce each other. We go beyond previous work by exploiting geometry more explicitly and segmenting the scene into static and moving regions. To that end, we introduce Competitive Collaboration, a framework that facilitates the coordinated training of multiple specialized neural networks to solve complex problems. Competitive Collaboration works much like expectation-maximization, but with neural networks that act as both competitors to explain pixels that correspond to static or moving regions, and as collaborators through a moderator that assigns pixels to be either static or independently moving. Our novel method integrates all these problems in a common framework and simultaneously reasons about the segmentation of the scene into moving objects and the static background, the camera motion, depth of the static scene structure, and the optical flow of moving objects. Our model is trained without any supervision and achieves state-of-the-art performance among joint unsupervised methods on all sub-problems.
Per-pixel ground-truth depth data is challenging to acquire at scale. To overcome this limitation, self-supervised learning has emerged as a promising alternative for training models to perform monocular depth estimation. In this paper, we propose a set of improvements, which together result in both quantitatively and qualitatively improved depth maps compared to competing self-supervised methods. Research on self-supervised monocular training usually explores increasingly complex architectures, loss functions, and image formation models, all of which have recently helped to close the gap with fully-supervised methods. We show that a surprisingly simple model, and associated design choices, lead to superior predictions. In particular, we propose (i) a minimum reprojection loss, designed to robustly handle occlusions, (ii) a full-resolution multi-scale sampling method that reduces visual artifacts, and (iii) an auto-masking loss to ignore training pixels that violate camera motion assumptions. We demonstrate the effectiveness of each component in isolation, and show high quality, state-of-the-art results on the KITTI benchmark.
Despite significant progress in image-based 3D scene flow estimation, the performance of such approaches has not yet reached the fidelity required by many applications. Simultaneously, these applications are often not restricted to image-based estimation: laser scanners provide a popular alternative to traditional cameras, for example in the context of self-driving cars, as they directly yield a 3D point cloud. In this paper, we propose to estimate 3D motion from such unstructured point clouds using a deep neural network. In a single forward pass, our model jointly predicts 3D scene flow as well as the 3D bounding box and rigid body motion of objects in the scene. While the prospect of estimating 3D scene flow from unstructured point clouds is promising, it is also a challenging task. We show that the traditional global representation of rigid body motion prohibits inference by CNNs, and propose a translation equivariant representation to circumvent this problem. For training our deep network, a large dataset is required. Because of this, we augment real scans from KITTI with virtual objects, realistically modeling occlusions and simulating sensor noise. A thorough comparison with classic and learning-based techniques highlights the robustness of the proposed approach.
Cardiac motion estimation and segmentation play important roles in quantitatively assessing cardiac function and diagnosing cardiovascular diseases. In this paper, we propose a novel deep learning method for joint estimation of motion and segmentation from cardiac MR image sequences. The proposed network consists of two branches: a cardiac motion estimation branch which is built on a novel unsupervised Siamese style recurrent spatial transformer network, and a cardiac segmentation branch that is based on a fully convolutional network. In particular, a joint multi-scale feature encoder is learned by optimizing the segmentation branch and the motion estimation branch simultaneously. This enables the weakly-supervised segmentation by taking advantage of features that are unsupervisedly learned in the motion estimation branch from a large amount of unannotated data. Experimental results using cardiac MRI images from 220 subjects show that the joint learning of both tasks is complementary and the proposed models outperform the competing methods significantly in terms of accuracy and speed.
Recent terrorist attacks in major cities around the world have brought many casualties among innocent citizens. One potential threat is represented by abandoned luggage items (that could contain bombs or biological warfare) in public areas. In this paper, we describe an approach for real-time automatic detection of abandoned luggage in video captured by surveillance cameras. The approach is comprised of two stages: (i) static object detection based on background subtraction and motion estimation and (ii) abandoned luggage recognition based on a cascade of convolutional neural networks (CNN). To train our neural networks we provide two types of examples: images collected from the Internet and realistic examples generated by imposing various suitcases and bags over the scene's background. We present empirical results demonstrating that our approach yields better performance than a strong CNN baseline method.
도전적인 환경에서 이동 로봇의 원활한 운용을 위해서는 저지연 국소 운동 추정과 정확한 전역 위치 추정이 필요하다. 대부분의 센서 융합(sensor fusion) 접근법은 특정 시나리오를 위해 설계되지만, 본 연구는 일반성과 사용성을 특징으로 하는 작업 및 설정에 구애받지 않는 다중모달 센서 융합을 위한 유연한 오픈소스 솔루션을 제안한다. Holistic Fusion은 센서 융합을 i) 국소 및 전역 로봇 상태와 ii) (이론적으로 무제한적인 수의) 동적 변수(기준 좌표계의 자동 정렬 포함)의 결합 추정 문제로 공식화한다. 이 공식화는 개념적 수정 없이 수많은 실세계 응용에 적용 가능하며, 하드코딩된/작업 특화 접근법을 넘어서는 포괄적 솔루션을 제공한다. 제안된 팩터 그래프(factor graph) 공식화는 서로 다른 좌표계에 대해 표현된 임의 개수의 절대, 국소, 랜드마크(landmark) 측정값을 최적화(optimization)의 상태로 명시적으로 포함하고 그 진화를 랜덤 워크(random walk)로 모델링함으로써 직접 융합할 수 있게 한다. 또한 추정 점프를 방지하기 위해 국소적 평활성과 일관성에 특별한 주의를 기울인다. Holistic Fusion은 일반적인 로봇 하드웨어에서 저지연且 부드러운 온라인 상태 추정을 가능하게 하면서 동시에 IMU 측정 속도에서 저드리프트(low-drift) 전역 위치 추정을 제공한다. 공개된 이 프레임워크 [1]의 효과는 서로 다른 작업 요구사항을 갖는 세 가지 로봇 플랫폼에서 다섯 가지 실세계 시나리오를 통해 입증되며, 다양한 절대 측정 유형의 융합 이점 [2]을 강조한다. [1] 코드: https://github.com/leggedrobotics/holistic_fusion [2] 프로젝트: https://leggedrobotics.github.io/holistic_fusion
인간의 동작과 주변 환경을 재구성하는 것은 인간-장면 상호작용을 이해하고 장면 내 인간 움직임을 예측하는 데 필수적이다. 제약된 환경에서 인간-장면 상호작용을 포착하는 데 많은 진전이 이루어졌지만, 기존 방법들은 웹 비디오에서 자연스럽고 다양한 인간 동작과 장면 맥락을 재구성하기 어렵다. 본 연구에서는 단안 비디오로부터 야생 환경에서의 4D 인간-장면 재구성을 위한 새로운 최적화 기반 방법인 JOSH를 제안한다. JOSH는 밀집 장면 재구성(dense scene reconstruction)과 인간 메시 복원(human mesh recovery) 기술을 초기화로 활용하고, 이후 인간-장면 접촉 제약을 이용하여 장면, 카메라 포즈, 인간 동작을 공동으로 최적화한다. 실험 결과, JOSH는 장면 기하학과 인간 동작의 공동 최적화를 통해 전역 인간 동작 추정과 밀집 장면 재구성 모두에서 더 나은 결과를 달성함을 보인다. 우리는 나아가 더 효율적인 모델인 JOSH3R을 설계하고, 웹 비디오에서 얻은 의사 레이블(pseudo-label)로 직접 학습시킨다. JOSH3R은 JOSH에서 예측된 레이블만으로 학습함에도 불구하고 다른 최적화 불필요 방법들을 능가하며, 이는 그 정확성과 일반화 능력을 더욱 입증한다.
본 논문은 사전학습된 대규모 이미지 확산 모델의 기하학적·콘텐츠 사전(prior)을 활용하여 Stitched Image Rectangling(SIR) 및 Rolling Shutter Correction(RSC)과 같은 단일 이미지 보정(rectification) 작업에서 모션 추정을 수행하는 새로운 프레임워크 StableMotion을 제안한다. 구체적으로 StableMotion은 텍스트-이미지 Stable Diffusion(SD) 모델을 백본으로 사용하고 이를 이미지-모션 추정기로 재구성한다. 확산 모델이 생성하는 불일관한 출력을 완화하기 위해, 여러 출력을 응집력 있는 고품질 결과로 통합하는 Adaptive Ensemble Strategy(AES)를 제안한다. 또한 샘플링 스텝 수를 늘리면 오히려 결과가 나빠질 수 있는 역설적 현상인 Sampling Steps Disaster(SSD)를 제시하고, 이는 원스텝 추론 설계의 동기가 된다. StableMotion은 두 가지 이미지 보정 작업에서 평가되었으며 두 작업 모두에서 최신(state-of-the-art) 성능을 달성했고, 보지 못한 SIR-OOD 및 실제 촬영 RSC 벤치마크에 대한 정성적 예시와 무참조(no-reference) 평가를 통해 유망한 전이 가능성도 보여준다. SSD에 힘입어 StableMotion은 효율적인 원스텝 추론을 달성하며, 선택적 AES 후처리와 함께 사용하더라도 이전 확산 모델 기반 방법 대비 100× 이상의 속도 향상을 제공한다. 코드와 가중치는 https://github.com/ivowang/StableMotion 에서 제공된다.