Reliable anticipation of pedestrian trajectory is imperative for the operation of autonomous vehicles and can significantly enhance the functionality of advanced driver assistance systems. While significant progress has been made in the field of pedestrian detection, forecasting pedestrian trajectories remains a challenging problem due to the unpredictable nature of pedestrians and the huge space of potentially useful features. In this work, we present a deep learning approach for pedestrian trajectory forecasting using a single vehicle-mounted camera. Deep learning models that have revolutionized other areas in computer vision have seen limited application to trajectory forecasting, in part due to the lack of richly annotated training data. We address the lack of training data by introducing a scalable machine annotation scheme that enables our model to be trained using a large dataset without human annotation. In addition, we propose Dynamic Trajectory Predictor (DTP), a model for forecasting pedestrian trajectory up to one second into the future. DTP is trained using both human and machine-annotated data, and anticipates dynamic motion that is not captured by linear models. Experimental evaluation confirms the benefits of the proposed model.
This paper proposes a novel approach for predicting the motion of pedestrians interacting with others. It uses a Generative Adversarial Network (GAN) to sample plausible predictions for any agent in the scene. As GANs are very susceptible to mode collapsing and dropping, we show that the recently proposed Info-GAN allows dramatic improvements in multi-modal pedestrian trajectory prediction to avoid these issues. We also left out L2-loss in training the generator, unlike some previous works, because it causes serious mode collapsing though faster convergence. We show through experiments on real and synthetic data that the proposed method leads to generate more diverse samples and to preserve the modes of the predictive distribution. In particular, to prove this claim, we have designed a toy example dataset of trajectories that can be used to assess the performance of different methods in preserving the predictive distribution modes.
An active area of research is to increase the safety of self-driving vehicles. Although safety cannot be guarenteed completely, the capability of a vehicle to predict the future trajectories of its surrounding vehicles could help ensure this notion of safety to a greater deal. We cast the trajectory forecast problem in a multi-time step forecasting problem and develop a Convolutional Neural Network based approach to learn from trajectory sequences generated from completely raw dataset in real-time. Results show improvement over baselines.
Deciphering human behaviors to predict their future paths/trajectories and what they would do from videos is important in many applications. Motivated by this idea, this paper studies predicting a pedestrian's future path jointly with future activities. We propose an end-to-end, multi-task learning system utilizing rich visual features about human behavioral information and interaction with their surroundings. To facilitate the training, the network is learned with an auxiliary task of predicting future location in which the activity will happen. Experimental results demonstrate our state-of-the-art performance over two public benchmarks on future trajectory prediction. Moreover, our method is able to produce meaningful future activity prediction in addition to the path. The result provides the first empirical evidence that joint modeling of paths and activities benefits future path prediction.
In this work, we explore the correlation between people trajectories and their head orientations. We argue that people trajectory and head pose forecasting can be modelled as a joint problem. Recent approaches on trajectory forecasting leverage short-term trajectories (aka tracklets) of pedestrians to predict their future paths. In addition, sociological cues, such as expected destination or pedestrian interaction, are often combined with tracklets. In this paper, we propose MiXing-LSTM (MX-LSTM) to capture the interplay between positions and head orientations (vislets) thanks to a joint unconstrained optimization of full covariance matrices during the LSTM backpropagation. We additionally exploit the head orientations as a proxy for the visual attention, when modeling social interactions. MX-LSTM predicts future pedestrians location and head pose, increasing the standard capabilities of the current approaches on long-term trajectory forecasting. Compared to the state-of-the-art, our approach shows better performances on an extensive set of public benchmarks. MX-LSTM is particularly effective when people move slowly, i.e. the most challenging scenario for all other models. The proposed approach also allows for accurate predictions on a longer time horizon.
We introduce a spatio-temporal convolutional neural network model for trajectory forecasting from visual sources. Applied in an auto-regressive way it provides an explicit probability distribution over continuations of a given initial trajectory segment. We discuss it in relation to (more complicated) existing work and report on experiments on two standard datasets for trajectory forecasting: MNISTseq and Stanford Drones, achieving results on-par with or better than previous methods.
Developing safe human-robot interaction systems is a necessary step towards the widespread integration of autonomous agents in society. A key component of such systems is the ability to reason about the many potential futures (e.g. trajectories) of other agents in the scene. Towards this end, we present the Trajectron, a graph-structured model that predicts many potential future trajectories of multiple agents simultaneously in both highly dynamic and multimodal scenarios (i.e. where the number of agents in the scene is time-varying and there are many possible highly-distinct futures for each agent). It combines tools from recurrent sequence modeling and variational deep generative modeling to produce a distribution of future trajectories for each agent in a scene. We demonstrate the performance of our model on several datasets, obtaining state-of-the-art results on standard trajectory prediction metrics as well as introducing a new metric for comparing models that output distributions.
Spatiotemporal systems are common in the real-world. Forecasting the multi-step future of these spatiotemporal systems based on the past observations, or, Spatiotemporal Sequence Forecasting (STSF), is a significant and challenging problem. Although lots of real-world problems can be viewed as STSF and many research works have proposed machine learning based methods for them, no existing work has summarized and compared these methods from a unified perspective. This survey aims to provide a systematic review of machine learning for STSF. In this survey, we define the STSF problem and classify it into three subcategories: Trajectory Forecasting of Moving Point Cloud (TF-MPC), STSF on Regular Grid (STSF-RG) and STSF on Irregular Grid (STSF-IG). We then introduce the two major challenges of STSF: 1) how to learn a model for multi-step forecasting and 2) how to adequately model the spatial and temporal structures. After that, we review the existing works for solving these challenges, including the general learning strategies for multi-step forecasting, the classical machine learning based methods for STSF, and the deep learning based methods for STSF. We also compare these methods and point out some potential research directions.
When humans navigate a crowed space such as a university campus or the sidewalks of a busy street, they follow common sense rules based on social etiquette. In this paper, we argue that in order to enable the design of new algorithms that can take fully advantage of these rules to better solve tasks such as target tracking or trajectory forecasting, we need to have access to better data in the first place. To that end, we contribute the very first large scale dataset (to the best of our knowledge) that collects images and videos of various types of targets (not just pedestrians, but also bikers, skateboarders, cars, buses, golf carts) that navigate in a real-world outdoor environment such as a university campus. We present an extensive evaluation where different methods for trajectory forecasting are evaluated and compared. Moreover, we present a new algorithm for trajectory prediction that exploits the complexity of our new dataset and allows to: i) incorporate inter-class interactions into trajectory prediction models (e.g, pedestrian vs bike) as opposed to just intra-class interactions (e.g., pedestrian vs pedestrian); ii) model the degree to which the social forces are regulating an interaction. We call the latter "social sensitivity"and it captures the sensitivity to which a target is responding to a certain interaction. An extensive experimental evaluation demonstrates the effectiveness of our novel approach.
We address the problem of incrementally modeling and forecasting long-term goals of a first-person camera wearer: what the user will do, where they will go, and what goal they seek. In contrast to prior work in trajectory forecasting, our algorithm, DARKO, goes further to reason about semantic states (will I pick up an object?), and future goal states that are far in terms of both space and time. DARKO learns and forecasts from first-person visual observations of the user's daily behaviors via an Online Inverse Reinforcement Learning (IRL) approach. Classical IRL discovers only the rewards in a batch setting, whereas DARKO discovers the states, transitions, rewards, and goals of a user from streaming data. Among other results, we show DARKO forecasts goals better than competing methods in both noisy and ideal settings, and our approach is theoretically and empirically no-regret.
We present an interpretable framework for path prediction that leverages dependencies between agents' behaviors and their spatial navigation environment. We exploit two sources of information: the past motion trajectory of the agent of interest and a wide top-view image of the navigation scene. We propose a Clairvoyant Attentive Recurrent Network (CAR-Net) that learns where to look in a large image of the scene when solving the path prediction task. Our method can attend to any area, or combination of areas, within the raw image (e.g., road intersections) when predicting the trajectory of the agent. This allows us to visualize fine-grained semantic elements of navigation scenes that influence the prediction of trajectories. To study the impact of space on agents' trajectories, we build a new dataset made of top-view images of hundreds of scenes (Formula One racing tracks) where agents' behaviors are heavily influenced by known areas in the images (e.g., upcoming turns). CAR-Net successfully attends to these salient regions. Additionally, CAR-Net reaches state-of-the-art accuracy on the standard trajectory forecasting benchmark, Stanford Drone Dataset (SDD). Finally, we show CAR-Net's ability to generalize to unseen scenes.
Understanding human motion behavior is critical for autonomous moving platforms (like self-driving cars and social robots) if they are to navigate human-centric environments. This is challenging because human motion is inherently multimodal: given a history of human motion paths, there are many socially plausible ways that people could move in the future. We tackle this problem by combining tools from sequence prediction and generative adversarial networks: a recurrent sequence-to-sequence model observes motion histories and predicts future behavior, using a novel pooling mechanism to aggregate information across people. We predict socially plausible futures by training adversarially against a recurrent discriminator, and encourage diverse predictions with a novel variety loss. Through experiments on several datasets we demonstrate that our approach outperforms prior work in terms of accuracy, variety, collision avoidance, and computational complexity.
Recent approaches on trajectory forecasting use tracklets to predict the future positions of pedestrians exploiting Long Short Term Memory (LSTM) architectures. This paper shows that adding vislets, that is, short sequences of head pose estimations, allows to increase significantly the trajectory forecasting performance. We then propose to use vislets in a novel framework called MX-LSTM, capturing the interplay between tracklets and vislets thanks to a joint unconstrained optimization of full covariance matrices during the LSTM backpropagation. At the same time, MX-LSTM predicts the future head poses, increasing the standard capabilities of the long-term trajectory forecasting approaches. With standard head pose estimators and an attentional-based social pooling, MX-LSTM scores the new trajectory forecasting state-of-the-art in all the considered datasets (Zara01, Zara02, UCY, and TownCentre) with a dramatic margin when the pedestrians slow down, a case where most of the forecasting approaches struggle to provide an accurate solution.
This paper addresses the problem of path prediction for multiple interacting agents in a scene, which is a crucial step for many autonomous platforms such as self-driving cars and social robots. We present \textit{SoPhie}; an interpretable framework based on Generative Adversarial Network (GAN), which leverages two sources of information, the path history of all the agents in a scene, and the scene context information, using images of the scene. To predict a future path for an agent, both physical and social information must be leveraged. Previous work has not been successful to jointly model physical and social interactions. Our approach blends a social attention mechanism with a physical attention that helps the model to learn where to look in a large scene and extract the most salient parts of the image relevant to the path. Whereas, the social attention component aggregates information across the different agent interactions and extracts the most important trajectory information from the surrounding neighbors. SoPhie also takes advantage of GAN to generates more realistic samples and to capture the uncertain nature of the future paths by modeling its distribution. All these mechanisms enable our approach to predict socially and physically plausible paths for the agents and to achieve state-of-the-art performance on several different trajectory forecasting benchmarks.
상호작용 모델링은 고도로 비선형적이며 시간에 따라 변하는 동작을 보이는 상호작용 객체 시스템, 즉 복잡한 동적 시스템(dynamical systems)을 학습하는 데 매우 중요합니다. 이러한 시스템의 상당 부분은 기하학적 그래프($\textit{geometric graphs}$), 즉 교통 상황의 차량과 같이 $\textit{임의로}$ 선택된 글로벌 좌표계 하에서 유클리드 공간에 위치한 노드(node)를 갖는 그래프로 정형화될 수 있습니다. 임의의 글로벌 좌표계에도 불구하고, 해당 동적 시스템을 지배하는 동역학은 회전 및 변위(translation)에 대해 불변이며, 이는 갈릴레이 불변성($\textit{Galilean invariance}$)으로도 알려져 있습니다. 이러한 불변성을 무시하면 일반화 성능이 저하되므로, 본 연구에서는 상호작용 동적 시스템의 기하학적 그래프에 회전-변위 불변성을 유도하기 위해 노드 객체별 로컬 좌표계(local coordinate frames)를 제안합니다. 또한, 로컬 좌표계를 통해 그래프 신경망(GNN)에서 비등방성 필터링(anisotropic filtering)을 자연스럽게 정의할 수 있습니다. 교통 상황, 3D 모션 캡처, 충돌 입자에 대한 실험을 통해 제안된 접근 방식이 최근 SOTA(state-of-the-art)를 여유 있게 능가함을 입증합니다.
본 논문에서는 카메라 네트워크 전반에 걸쳐 이동하는 객체의 궤적을 예측하는 다중 카메라 궤적 예측(MCTF, Multi-Camera Trajectory Forecasting) 문제를 소개한다. 감시 및 교통 모니터링과 같은 응용 분야에서 다중 카메라 설정이 널리 사용되고 있음에도 불구하고, 기존 궤적 예측 기법들은 일반적으로 단일 카메라 궤적 예측(SCTF)에 집중되어 있어 이러한 응용 분야에서의 활용이 제한된다. 나아가 단일 카메라를 사용하는 것은 가용한 시야각(field-of-view)을 제한하여 장기적인 궤적 예측을 불가능하게 만든다. 저자들은 여러 시점으로부터 추정된 모든 상대적 객체 위치를 동시에 활용하고 가능한 모든 시점에서 객체의 미래 위치를 예측하는 MCTF 프레임워크를 개발함으로써 SCTF의 이러한 한점들을 해결한다. 제안하는 프레임워크는 객체가 어느 카메라에 나타나는지(Which), 그리고 카메라 뷰 내에서 언제(When) 어디에(Where) 나타나는지를 예측하는 Which-When-Where 접근법을 따른다. 이를 위해 다중 카메라 뷰에 걸친 궤적과 관련 불확실성을 인코딩하는 새로운 기법인 궤적 텐서(trajectory tensor) 개념을 제안한다. 궤적 텐서를 위한 여러 인코더-디코더 MCTF 모델을 개발하고, 특히 MCTF 작업을 위해 구축된 자체 데이터베이스(15개 카메라 뷰의 600시간 비디오 데이터로 구성)에서 광범위한 실험을 제시한다. 결과는 제안하는 궤적 텐서 모델이 좌표 궤적 기반 MCTF 모델 및 MCTF에 맞게 변형된 기존 SCTF 기법들보다 우수한 성능을 보임을 나타낸다. 코드는 https://github.com/olly-styles/Trajectory-Tensors 에서 확인할 수 있다.
Modeling the cognitive and experiential factors of human navigation is central to deepening our understanding of human-environment interaction and to enabling safe social navigation and effective assistive wayfinding. Most existing methods focus on forecasting motions in fully observed scenes and often neglect human factors that capture how people feel and respond to space. To address this gap, we propose EgoCogNav, a multimodal egocentric navigation framework that jointly forecasts perceived path uncertainty, trajectories and head motion from egocentric video, gaze, and motion history. To facilitate research in the field, we introduce the Cognition-aware Egocentric Navigation (CEN) dataset consisting of 6 hours real-world egocentric recordings capturing diverse navigation behaviors in real-world scenarios. Experiments show that EgoCogNav learns the perceived uncertainty that strongly correlates with human-like behaviors such as scanning, hesitation, and backtracking while improving trajectory and head-motion forecasting on held-out navigation recordings.
자율주행 궤적 예측 모델은 조건부 가우시안 혼합 모델(GMM)로 모델링되지만 winner-take-all(WTA) 손실의 강할당으로 학습되어, 모델링-학습 불일치가 모드 확률의 비정보성과 궤적 공간 과분할을 유발한다는 점을 K-means 관점에서 분석한다. 테스트 시점 사후분포 가중 병합과 1단계 기대-최대화(EM) 업데이트를 사후 처리로 적용하면 인접 모드 간 확률 질량을 공유하여, 재학습 없이 WTA로 학습된 여러 아키텍처에서 모드 사후분포 품질과 변위 지표 예측 성능을 개선한다.
기존의 시각 내비게이션 정책은 복잡한 환경에서 근시안적 의사결정과 모드 붕괴(mode collapse) 문제에 빈번히 어려움을 겪는다. 월드 모델(world model)은 유망한 대안을 제시하지만, 기존 패러다임은 일반적으로 인식, 생성, 제어를 분리하여 이들이 공유하는 시공간적 역학을 포착하는 데 실패한다. 본 논문에서는 잠재 월드 추론, 멀티모달 행동 예측, 제어 가능한 시각 생성을 원활하게 통합하는 통합 내비게이션 월드 모델인 NavWM을 제안한다. 핵심적으로, NavWM은 잠재 월드 토큰(latent world token)을 활용하여 기하학적·의미론적 사전 지식을 증류하고, 에이전트에게 강건한 구조적 이해 능력을 부여한다. 결정론적 정책의 한계를 극복하기 위해, 다양한 행동 공간을 생성하는 앵커 기반 멀티모달 궤적 예측 프레임워크를 도입한다. 이러한 내재적 다양성은 생성적 월드 모델이 강건한 폐루프(closed-loop) 플래너로 기능할 수 있도록 명시적으로 지원하며, 시각적 예지(visual foresight)를 활용하여 최적 경로를 평가하고 선택한다. 다양한 로보틱스 데이터셋에 걸친 광범위한 실험을 통해 NavWM이 최첨단 성능을 크게 향상시키며, 고충실도 미래 상태 생성과 제로샷(zero-shot) 내비게이션 성공률 모두에서 두드러진 개선을 달성함을 입증한다.