High-definition (HD) semantic maps are crucial in enabling autonomous vehicles to navigate urban environments. The traditional method of creating offline HD maps involves labor-intensive manual annotation processes, which are not only costly but also insufficient for timely updates. Recent studies have proposed an alternative approach that generates local maps using online sensor observations. However, this approach is limited by the sensor's perception range and its susceptibility to occlusions. In this study, we propose Neural Map Prior (NMP), a neural representation of global maps. This representation automatically updates itself and improves the performance of local map inference. Specifically, we utilize two approaches to achieve this. Firstly, to integrate a strong map prior into local map inference, we apply cross-attention, a mechanism that dynamically identifies correlations between current and prior features. Secondly, to update the global neural map prior, we utilize a learning-based fusion module that guides the network in fusing features from previous traversals. Our experimental results, based on the nuScenes dataset, demonstrate that our framework is highly compatible with various map segmentation and detection architectures. It significantly improves map prediction performance, even in challenging weather conditions and situations with a longer perception range. To the best of our knowledge, this is the first learning-based system for creating a global map prior.
The provision of natural language explanations for the predictions of deep-learning-based vehicle controllers is critical as it enhances transparency and easy audit. In this work, a state-of-the-art (SOTA) prediction and explanation model is thoroughly evaluated and validated (as a benchmark) on the new Sense--Assess--eXplain (SAX). Additionally, we developed a new explainer model that improved over the baseline architecture in two ways: (i) an integration of part of speech prediction and (ii) an introduction of special token penalties. On the BLEU metric, our explanation generation technique outperformed SOTA by a factor of 7.7 when applied on the BDD-X dataset. The description generation technique is also improved by a factor of 1.3. Hence, our work contributes to the realisation of future explainable autonomous vehicles.
The ability to predict future structure features of environments based on past perception information is extremely needed by autonomous vehicles, which helps to make the following decision-making and path planning more reasonable. Recently, point cloud prediction (PCP) is utilized to predict and describe future environmental structures by the point cloud form. In this letter, we propose a novel efficient Transformer-based network to predict the future LiDAR point clouds exploiting the past point cloud sequences. We also design a semantic auxiliary training strategy to make the predicted LiDAR point cloud sequence semantically similar to the ground truth and thus improves the significance of the deployment for more tasks in real-vehicle applications. Our approach is completely self-supervised, which means it does not require any manual labeling and has a solid generalization ability toward different environments. The experimental results show that our method outperforms the state-of-the-art PCP methods on the prediction results and semantic similarity, and has a good real-time performance. Our open-source code and pre-trained models are available at https://github.com/Blurryface0814/PCPNet.
Autonomous vehicles rely on a variety of sensors to gather information about their surrounding. The vehicle's behavior is planned based on the environment perception, making its reliability crucial for safety reasons. The active LiDAR sensor is able to create an accurate 3D representation of a scene, making it a valuable addition for environment perception for autonomous vehicles. Due to light scattering and occlusion, the LiDAR's performance change under adverse weather conditions like fog, snow or rain. This limitation recently fostered a large body of research on approaches to alleviate the decrease in perception performance. In this survey, we gathered, analyzed, and discussed different aspects on dealing with adverse weather conditions in LiDAR-based environment perception. We address topics such as the availability of appropriate data, raw point cloud processing and denoising, robust perception algorithms and sensor fusion to mitigate adverse weather induced shortcomings. We furthermore identify the most pressing gaps in the current literature and pinpoint promising research directions.
An ego vehicle following a virtual lead vehicle planned route is an essential component when autonomous and non-autonomous vehicles interact. Yet, there is a question about the driver's ability to follow the planned lead vehicle route. Thus, predicting the trajectory of the ego vehicle route given a lead vehicle route is of interest. We introduce a new dataset, the FollowMe dataset, which offers a motion and behavior prediction problem by answering the latter question of the driver's ability to follow a lead vehicle. We also introduce a deep spatio-temporal graph model FollowMe-STGCNN as a baseline for the dataset. In our experiments and analysis, we show the design benefits of FollowMe-STGCNN in capturing the interactions that lie within the dataset. We contrast the performance of FollowMe-STGCNN with prior motion prediction models showing the need to have a different design mechanism to address the lead vehicle following settings.
The performance of the current collision avoidance systems in Autonomous Vehicles (AV) and Advanced Driver Assistance Systems (ADAS) can be drastically affected by low light and adverse weather conditions. Collisions with large animals such as deer in low light cause significant cost and damage every year. In this paper, we propose the first AI-based method for future trajectory prediction of large animals and mitigating the risk of collision with them in low light. In order to minimize false collision warnings, in our multi-step framework, first, the large animal is accurately detected and a preliminary risk level is predicted for it and low-risk animals are discarded. In the next stage, a multi-stream CONV-LSTM-based encoder-decoder framework is designed to predict the future trajectory of the potentially high-risk animals. The proposed model uses camera motion prediction as well as the local and global context of the scene to generate accurate predictions. Furthermore, this paper introduces a new dataset of FIR videos for large animal detection and risk estimation in real nighttime driving scenarios. Our experiments show promising results of the proposed framework in adverse conditions. Our code is available online.
Set-valued state estimation when in the presence of uncertainties in the model have been addressed in the literature essentially following three main approaches: i) interval arithmetic of the uncertain dynamics with the estimates; ii) factorizing the uncertainty into matrices with unity rank; and, iii) performing the convex hull for the vertices of the uncertainty space. Approach i) and ii) introduce a lot of conservatism because both disregard the relationship of the parameters with the entries of the dynamics matrix. On the other hand, approach iii) has a large growth on the number of variables required to represent the set or is approximated losing its main advantage in comparison with i) and ii). In this paper, with the application of autonomous vehicles in GPS-denied areas that resort to beacon signals for localization, we develop an exact (meaning no added conservatism) and optimal (smallest growth in the number of variables) closed-form definition for the convex hull of Convex Constrained Generators (CCGs). This results in a more efficient method to represent the minimum volume convex set corresponding to the state estimation. Given that reductions methods are still lacking in the literature for CCGs, we employ an approximation using ray-shooting that is comparable in terms of accuracy with methods for Constrained Zonotopes as the ones implemented in CORA. Simulations illustrate the greater accuracy of CCGs with the proposed convex hull operation in comparison to Constrained Zonotopes.
The concerns to autonomous vehicles have been becoming more intriguing in coping with the more environmentally dynamics non-linear systems under some constraints and disturbances. These vehicles connect not only to the self-instruments yet to the neighborhoods components, making the diverse interconnected communications which should be handled locally to ease the computation and to fasten the decision. To deal with those interconnected networks, the distributed estimation to reach the untouched states, pursuing sensorless design, is approached, initiated by the construction of the modified pseudo measurement which, due to approximation, led to the weighted average consensus calculation within unscented filtering along with the bounded estimation errors. Moreover, the tested vehicles are also associated to certain robust control scenarios subject to noise and disturbance with some stability analysis to ensure the usage of the proposed estimation algorithm. The numerical instances are presented along with the performances of the control and estimation method. The results affirms the effectiveness of the method with limited error deviation compared to the other centralized and distributed filtering. Beyond these, the further research would be the directed sensorless design and fault-tolerant learning control subject to faults to negate the failures.
The ubiquitous multi-camera setup on modern autonomous vehicles provides an opportunity to construct surround-view depth. Existing methods, however, either perform independent monocular depth estimations on each camera or rely on computationally heavy self attention mechanisms. In this paper, we propose a novel guided attention architecture, EGA-Depth, which can improve both the efficiency and accuracy of self-supervised multi-camera depth estimation. More specifically, for each camera, we use its perspective view as the query to cross-reference its neighboring views to derive informative features for this camera view. This allows the model to perform attention only across views with considerable overlaps and avoid the costly computations of standard self-attention. Given its efficiency, EGA-Depth enables us to exploit higher-resolution visual features, leading to improved accuracy. Furthermore, EGA-Depth can incorporate more frames from previous time steps as it scales linearly w.r.t. the number of views and frames. Extensive experiments on two challenging autonomous driving benchmarks nuScenes and DDAD demonstrate the efficacy of our proposed EGA-Depth and show that it achieves the new state-of-the-art in self-supervised multi-camera depth estimation.
Connected Autonomous Vehicles (CAVs) are widely expected to improve traffic safety and efficiency by exploiting information from surrounding vehicles via V2V communication. A CAV typically adapts its speed based on information from the vehicle it follows. CAVs can also use information from vehicles further ahead within their communication range, and this results in improved traffic safety and efficiency. In mixed traffic scenarios, however, this may not always be possible due to the presence of human-driven vehicles that do not have communication capabilities. Furthermore, as wireless vehicular networks are unreliable, information from other vehicles can be delayed or lost, which brings more challenges for CAVs in utilizing information from multiple leading vehicles. A few studies have investigated the impact of CAVs where they use information from multiple leading vehicles on traffic safety and efficiency, but only in very limited scenarios (i.e., with a very small number of vehicles). In contrast, this paper investigates the impact of CAV car-following control based on multiple leading vehicles information on both mixed traffic safety and efficiency in realistic scenarios in terms of imperfect communication, vehicle modelling, and traffic scenario. Results show that exploiting information from multiple, rather than a single, leading vehicles in CAV controller design further improves both traffic safety and efficiency especially at high penetration rates. In addition to proper tuning of CAV controller parameters (control gains and time headways), the scale of the improvement depends on both market penetration rate (MPR) and communication reliability. A packet error rate (PER) of 70% leads to an increase in traffic efficiency by 4.18% (at 40% MPR) and 12.19% (at 70% MPR), compared to the simple single leading vehicle information based controller.
Cyber-physical systems (CPS) like autonomous vehicles, that utilize learning components, are often sensitive to noise and out-of-distribution (OOD) instances encountered during runtime. As such, safety critical tasks depend upon OOD detection subsystems in order to restore the CPS to a known state or interrupt execution to prevent safety from being compromised. However, it is difficult to guarantee the performance of OOD detectors as it is difficult to characterize the OOD aspect of an instance, especially in high-dimensional unstructured data. To distinguish between OOD data and data known to the learning component through the training process, an emerging technique is to incorporate variational autoencoders (VAE) within systems and apply classification or anomaly detection techniques on their latent spaces. The rationale for doing so is the reduction of the data domain size through the encoding process, which benefits real-time systems through decreased processing requirements, facilitates feature analysis for unstructured data and allows more explainable techniques to be implemented. This study places probably approximately correct (PAC) based guarantees on OOD detection using the encoding process within VAEs to quantify image features and apply conformal constraints over them. This is used to bound the detection error on unfamiliar instances with user-defined confidence. The approach used in this study is to empirically establish these bounds by sampling the latent probability distribution and evaluating the error with respect to the constraint violations that are encountered. The guarantee is then verified using data generated from CARLA, an open-source driving simulator.
Connected autonomous vehicles (CAV) constitute an important application of future-oriented traffic management .A vehicular system dominated by fully autonomous vehicles requires a robust and efficient vehicle-to-everything (V2X) infrastructure that will provide sturdy connection of vehicles in both short and long distances for a large number of devices, requiring high spectral efficiency (SE). Power domain non-orthogonal multiple access (PD-NOMA) technique has the potential to provide the required high SE levels. In this paper, a vehicular PD-NOMA testbed is implemented using software defined radio (SDR) nodes. The main concerns and their corresponding solutions arising from the implementation are highlighted. The bit error rates(BER) of vehicles with different channel conditions are measured for mobile and stationary cases. The extent of the estimation errors on the success rate beyond the idealized theoretical analysis view is investigated and the approaches to alleviate these errors are discussed. Finally, our perspective on possible PD-NOMA based CAV deployment scenarios is presented in terms of performance constraints and expectancy along with the overlooked open issues.
Self-adaptive systems are able to change their behaviour at run-time in response to changes. Self-adaptation is an important strategy for managing uncertainty that is present during the design of modern systems, such as autonomous vehicles. However, assuring the safety of self-adaptive systems remains a challenge, particularly when the adaptations have an impact on safety-critical functions. The field of safety engineering has established practices for analyzing the safety of systems. System Theoretic Process and Analysis (STPA) is a hazard analysis method that is well-suited for self-adaptive systems. This paper describes a design-time extension of STPA for self-adaptive systems. Then, it derives a reference model and analysis obligations to support the STPA activities. The method is applied to three self-adaptive systems described in the literature. The results demonstrate that STPA, when used in the manner described, is an applicable hazard analysis method for safety-critical self-adaptive systems.
Road accidents involving autonomous vehicles commonly occur in situations where a (pedestrian) obstacle presents itself in the path of the moving vehicle at very sudden time intervals, leaving the robot even lesser time to react to the change in scene. In order to tackle this issue, we propose a novel algorithmic implementation that classifies the intent of a single arbitrarily chosen pedestrian in a two dimensional frame into logic states in a procedural manner using quaternions generated from a MediaPipe pose estimation model. This bypasses the need to employ any relatively high latency deep-learning algorithms primarily due to the lack of necessity for depth perception as well as an implicit cap on the computational resources that most IoT edge devices present. The model was able to achieve an average testing accuracy of 83.56% with a reliable variance of 0.0042 while operating with an average latency of 48 milliseconds, demonstrating multiple notable advantages over the current standard of using spatio-temporal convolutional networks for these perceptive tasks.
Uncovering potential failure cases is a crucial step in the validation of safety critical systems such as autonomous vehicles. Failure search may be done through logging substantial vehicle miles in either simulation or real world testing. Due to the sparsity of failure events, naive random search approaches require significant amounts of vehicle operation hours to find potential system weaknesses. As a result, adaptive searching techniques have been proposed to efficiently explore and uncover failure trajectories of an autonomous policy in simulation. Adaptive Stress Testing (AST) is one such method that poses the problem of failure search as a Markov decision process and uses reinforcement learning techniques to find high probability failures. However, this formulation requires a probability model for the actions of all agents in the environment. In systems where the environment actions are discrete and dependencies among agents exist, it may be infeasible to fully characterize the distribution or find a suitable proxy. This work proposes the use of a data driven approach to learn a suitable classifier that tries to model how humans identify {critical states and use this to guide failure search in AST. We show that the incorporation of critical states into the AST framework generates failure scenarios with increased safety violations in an autonomous driving policy with a discrete action space.
This paper proposes a consensus controller for multi-agent systems that can guarantee the agents' safety. The controller, built with the idea of output prediction and the Newton-Raphson method, achieves consensus for a class of heterogeneous nonlinear systems. The Integral Control Barrier Function is applied in conjunction with the controller, such that the agents' states are confined within pre-defined safety sets. Due to the dynamically-defined control input, the resulting optimization problem from the barrier function is always a Quadratic Program, despite the nonlinearities that the system dynamics may have. We verify the proposed controller using a platoon of autonomous vehicles modeled by kinematic bicycles. A convergence analysis of the leader-follower consensus under the path graph topology is conducted. Simulation results show that the vehicles achieve consensus while keeping safe inter-agent distances, suggesting a potential in future applications.
To drive safely in complex traffic environments, autonomous vehicles need to make an accurate prediction of the future trajectories of nearby heterogeneous traffic agents (i.e., vehicles, pedestrians, bicyclists, etc). Due to the interactive nature, human drivers are accustomed to infer what the future situations will become if they are going to execute different maneuvers. To fully exploit the impacts of interactions, this paper proposes a ego-planning guided multi-graph convolutional network (EPG-MGCN) to predict the trajectories of heterogeneous agents using both historical trajectory information and ego vehicle's future planning information. The EPG-MGCN first models the social interactions by employing four graph topologies, i.e., distance graphs, visibility graphs, planning graphs and category graphs. Then, the planning information of the ego vehicle is encoded by both the planning graph and the subsequent planning-guided prediction module to reduce uncertainty in the trajectory prediction. Finally, a category-specific gated recurrent unit (CS-GRU) encoder-decoder is designed to generate future trajectories for each specific type of agents. Our network is evaluated on two real-world trajectory datasets: ApolloScape and NGSIM. The experimental results show that the proposed EPG-MGCN achieves state-of-the-art performance compared to existing methods.
In this paper we consider the problem of estimation of scalar field distribution (e.g. pollutant, moisture, temperature) from noisy measurements collected by unmanned autonomous vehicles such as UAVs. The field is modelled as a sum of Fourier components/modes, where the number of modes retained and estimated determines in a natural way the approximation quality. An algorithm for estimating the modes using an online optimization approach is presented, under the assumption that the noisy measurements are quantized. The algorithm can also estimate time-varying fields through the introduction of a forgetting factor. Simulation studies demonstrate the effectiveness of the proposed approach.
Predicting the behaviour (i.e., manoeuvre/trajectory) of other road users, including vehicles, is critical for the safe and efficient operation of autonomous vehicles (AVs), a.k.a., automated driving systems (ADSs). Due to the uncertain future behaviour of vehicles, multiple future behaviour modes are often plausible for a vehicle in a given driving scene. Therefore, multimodal prediction can provide richer information than single-mode prediction, enabling AVs to perform a better risk assessment. To this end, we propose a novel multimodal prediction framework that can predict multiple plausible behaviour modes and their likelihoods. The proposed framework includes a bespoke problem formulation for manoeuvre prediction, a novel transformer-based prediction model, and a tailored training method for multimodal manoeuvre and trajectory prediction. The performance of the framework is evaluated using three public highway driving datasets, namely NGSIM, highD, and exiD. The results show that our framework outperforms the state-of-the-art multimodal methods in terms of prediction error and is capable of predicting plausible manoeuvre and trajectory modes.
Object detection and multiple object tracking (MOT) are essential components of self-driving systems. Accurate detection and uncertainty quantification are both critical for onboard modules, such as perception, prediction, and planning, to improve the safety and robustness of autonomous vehicles. Collaborative object detection (COD) has been proposed to improve detection accuracy and reduce uncertainty by leveraging the viewpoints of multiple agents. However, little attention has been paid to how to leverage the uncertainty quantification from COD to enhance MOT performance. In this paper, as the first attempt to address this challenge, we design an uncertainty propagation framework called MOT-CUP. Our framework first quantifies the uncertainty of COD through direct modeling and conformal prediction, and propagates this uncertainty information into the motion prediction and association steps. MOT-CUP is designed to work with different collaborative object detectors and baseline MOT algorithms. We evaluate MOT-CUP on V2X-Sim, a comprehensive collaborative perception dataset, and demonstrate a 2% improvement in accuracy and a 2.67X reduction in uncertainty compared to the baselines, e.g. SORT and ByteTrack. In scenarios characterized by high occlusion levels, our MOT-CUP demonstrates a noteworthy $4.01\%$ improvement in accuracy. MOT-CUP demonstrates the importance of uncertainty quantification in both COD and MOT, and provides the first attempt to improve the accuracy and reduce the uncertainty in MOT based on COD through uncertainty propagation. Our code is public on https://coperception.github.io/MOT-CUP/.