Audio-visual (AV) lip biometrics is a promising authentication technique that leverages the benefits of both the audio and visual modalities in speech communication. Previous works have demonstrated the usefulness of AV lip biometrics. However, the lack of a sizeable AV database hinders the exploration of deep-learning-based audio-visual lip biometrics. To address this problem, we compile a moderate-size database using existing public databases. Meanwhile, we establish the DeepLip AV lip biometrics system realized with a convolutional neural network (CNN) based video module, a time-delay neural network (TDNN) based audio module, and a multimodal fusion module. Our experiments show that DeepLip outperforms traditional speaker recognition models in context modeling and achieves over 50% relative improvements compared with our best single modality baseline, with an equal error rate of 0.75% and 1.11% on the test datasets, respectively.
In this paper we introduce a novel way of estimating prediction uncertainty in deep networks through the use of uncertainty surrogates. These surrogates are features of the penultimate layer of a deep network that are forced to match predefined patterns. The patterns themselves can be, among other possibilities, a known visual symbol. We show how our approach can be used for estimating uncertainty in prediction and out-of-distribution detection. Additionally, the surrogates allow for interpretability of the ability of the deep network to learn and at the same time lend robustness against adversarial attacks. Despite its simplicity, our approach is superior to the state-of-the-art approaches on standard metrics as well as computational efficiency and ease of implementation. A wide range of experiments are performed on standard datasets to prove the efficacy of our approach.
Convolutional neural networks (CNNs) have found many applications in tasks involving two-dimensional (2D) data, such as image classification and image processing. Therefore, 2D convolution layers have been heavily optimized on CPUs and GPUs. However, in many applications - for example genomics and speech recognition, the data can be one-dimensional (1D). Such applications can benefit from optimized 1D convolution layers. In this work, we introduce our efficient implementation of a generic 1D convolution layer covering a wide range of parameters. It is optimized for x86 CPU architectures, in particular, for architectures containing Intel AVX-512 and AVX-512 BFloat16 instructions. We use the LIBXSMM library's batch-reduce General Matrix Multiplication (BRGEMM) kernel for FP32 and BFloat16 precision. We demonstrate that our implementation can achieve up to 80% efficiency on Intel Xeon Cascade Lake and Cooper Lake CPUs. Additionally, we show the generalization capability of our BRGEMM based approach by achieving high efficiency across a range of parameters. We consistently achieve higher efficiency than the 1D convolution layer with Intel oneDNN library backend for varying input tensor widths, filter widths, number of channels, filters, and dilation parameters. Finally, we demonstrate the performance of our optimized 1D convolution layer by utilizing it in the end-to-end neural network training with real genomics datasets and achieve up to 6.86x speedup over the oneDNN library-based implementation on Cascade Lake CPUs. We also demonstrate the scaling with 16 sockets of Cascade/Cooper Lake CPUs and achieve significant speedup over eight V100 GPUs using a similar power envelop. In the end-to-end training, we get a speedup of 1.41x on Cascade Lake with FP32, 1.57x on Cooper Lake with FP32, and 2.27x on Cooper Lake with BFloat16 over eight V100 GPUs with FP32.
We present PyTorch Geometric Temporal a deep learning framework combining state-of-the-art machine learning algorithms for neural spatiotemporal signal processing. The main goal of the library is to make temporal geometric deep learning available for researchers and machine learning practitioners in a unified easy-to-use framework. PyTorch Geometric Temporal was created with foundations on existing libraries in the PyTorch eco-system, streamlined neural network layer definitions, temporal snapshot generators for batching, and integrated benchmark datasets. These features are illustrated with a tutorial-like case study. Experiments demonstrate the predictive performance of the models implemented in the library on real world problems such as epidemiological forecasting, ridehail demand prediction and web-traffic management. Our sensitivity analysis of runtime shows that the framework can potentially operate on web-scale datasets with rich temporal features and spatial structure.
Indoor localization services are a crucial aspect for the realization of smart cyber-physical systems within cities of the future. Such services are poised to reinvent the process of navigation and tracking of people and assets in a variety of indoor and subterranean environments. The growing ownership of computationally capable smartphones has laid the foundations of portable fingerprinting-based indoor localization through deep learning. However, as the demand for accurate localization increases, the computational complexity of the associated deep learning models increases as well. We present an approach for reducing the computational requirements of a deep learning-based indoor localization framework while maintaining localization accuracy targets. Our proposed methodology is deployed and validated across multiple smartphones and is shown to deliver up to 42% reduction in prediction latency and 45% reduction in prediction energy as compared to the best-known baseline deep learning-based indoor localization model.
The Supervisory control and data acquisition (SCADA) systems have been continuously leveraging the evolution of network architecture, communication protocols, next-generation communication techniques (5G, 6G, Wi-Fi 6), and the internet of things (IoT). However, SCADA system has become the most profitable and alluring target for ransomware attackers. This paper proposes the deep learning-based novel ransomware detection framework in the SCADA controlled electric vehicle charging station (EVCS) with the performance analysis of three deep learning algorithms, namely deep neural network (DNN), 1D convolution neural network (CNN), and long short-term memory (LSTM) recurrent neural network. All three-deep learning-based simulated frameworks achieve around 97% average accuracy (ACC), more than 98% of the average area under the curve (AUC), and an average F1-score under 10-fold stratified cross-validation with an average false alarm rate (FAR) less than 1.88%. Ransomware driven distributed denial of service (DDoS) attack tends to shift the SOC profile by exceeding the SOC control thresholds. The severity has been found to increase as the attack progress and penetration increases. Also, ransomware driven false data injection (FDI) attack has the potential to damage the entire BES or physical system by manipulating the SOC control thresholds. It's a design choice and optimization issue that a deep learning algorithm can deploy based on the tradeoffs between performance metrics.
Alternative recommender systems are critical for ecommerce companies. They guide customers to explore a massive product catalog and assist customers to find the right products among an overwhelming number of options. However, it is a non-trivial task to recommend alternative products that fit customer needs. In this paper, we use both textual product information (e.g. product titles and descriptions) and customer behavior data to recommend alternative products. Our results show that the coverage of alternative products is significantly improved in offline evaluations as well as recall and precision. The final A/B test shows that our algorithm increases the conversion rate by 12 percent in a statistically significant way. In order to better capture the semantic meaning of product information, we build a Siamese Network with Bidirectional LSTM to learn product embeddings. In order to learn a similarity space that better matches the preference of real customers, we use co-compared data from historical customer behavior as labels to train the network. In addition, we use NMSLIB to accelerate the computationally expensive kNN computation for millions of products so that the alternative recommendation is able to scale across the entire catalog of a major ecommerce site.
Deep learning applications have been thriving over the last decade in many different domains, including computer vision and natural language understanding. The drivers for the vibrant development of deep learning have been the availability of abundant data, breakthroughs of algorithms and the advancements in hardware. Despite the fact that complex industrial assets have been extensively monitored and large amounts of condition monitoring signals have been collected, the application of deep learning approaches for detecting, diagnosing and predicting faults of complex industrial assets has been limited. The current paper provides a thorough evaluation of the current developments, drivers, challenges, potential solutions and future research needs in the field of deep learning applied to Prognostics and Health Management (PHM) applications.
Wireless signals contain transmitter specific features, which can be used to verify the identity of transmitters and assist in implementing an authentication and authorization system. Most recently, there has been wide interest in using deep learning for transmitter identification. However, the existing deep learning work has posed the problem as closed set classification, where a neural network classifies among a finite set of known transmitters. No matter how large this set is, it will not include all transmitters that exist. Malicious transmitters outside this closed set, once within communications range, can jeopardize the system security. In this paper, we propose a deep learning approach for transmitter authorization based on open set recognition. Our proposed approach identifies a set of authorized transmitters, while rejecting any other unseen transmitters by recognizing their signals as outliers. We propose three approaches for this problem and show their ability to reject signals from unauthorized transmitters on a dataset of WiFi captures. We consider the structure of training data needed, and we show that the accuracy improves by having signals from known unauthorized transmitters in the training set.
Time delay estimation is essential in Acoustic Source Localization (ASL) systems. One of the most used techniques for this purpose is the Generalized Cross Correlation (GCC) between a pair of signals and its use in Steered Response Power (SRP) techniques, which estimate the acoustic power at a specific location. Nowadays, Deep Learning strategies may outperform these methods. However, they are generally dependent on the geometric and sensor configuration conditions that are available during the training phases, thus having limited generalization capabilities when facing new environments if no re-training nor adaptation is applied. In this work, we propose a method based on an encoder-decoder CNN architecture capable of outperforming the well known SRP-PHAT algorithm, and also other Deep Learning strategies when working in mismatched training-testing conditions without requiring a model re-training. Our proposal aims to estimate a smoothed version of the correlation signals, that is then used to generate a refined acoustic power map, which leads to better performance on the ASL task. Our experimental evaluation uses three publicly available realistic datasets and provides a comparison with the SRP-PHAT algorithm and other recent proposals based on Deep Learning.
We tackle the problem of forecasting network-signal snapshots using past signal measurements acquired by a subset of network nodes. This task can be seen as a combination of multivariate time-series prediction and graph-signal interpolation. This is a fundamental problem for many applications wherein deploying a high granularity network is impractical. Our solution combines recurrent neural networks with frequency-analysis tools from graph signal processing, and assumes that data is sufficiently smooth with respect to the underlying graph. The proposed approach outperforms state-of-the-art deep learning techniques, especially when only a small fraction of the graph signals is accessible, considering two distinct real world datasets: temperatures in the US and speed flow in Seattle. The results also indicate that our method can handle noisy signals and missing data, making it suitable to many practical applications.
Automatic speech recognition in reverberant conditions is a challenging task as the long-term envelopes of the reverberant speech are temporally smeared. In this paper, we propose a neural model for enhancement of sub-band temporal envelopes for dereverberation of speech. The temporal envelopes are derived using the autoregressive modeling framework of frequency domain linear prediction (FDLP). The neural enhancement model proposed in this paper performs an envelop gain based enhancement of temporal envelopes and it consists of a series of convolutional and recurrent neural network layers. The enhanced sub-band envelopes are used to generate features for automatic speech recognition (ASR). The ASR experiments are performed on the REVERB challenge dataset as well as the CHiME-3 dataset. In these experiments, the proposed neural enhancement approach provides significant improvements over a baseline ASR system with beamformed audio (average relative improvements of 21% on the development set and about 11% on the evaluation set in word error rates for REVERB challenge dataset).
Deep learning, particularly convolutional neural networks (CNNs), have yielded rapid, significant improvements in computer vision and related domains. But conventional deep learning architectures perform poorly when data have an underlying graph structure, as in social, biological, and many other domains. This paper explores 1)how graph signal processing (GSP) can be used to extend CNN components to graphs in order to improve model performance; and 2)how to design the graph CNN architecture based on the topology or structure of the data graph.
Deep learning is able to functionally simulate the human brain and thus, it has attracted considerable interest. Optics-assisted deep learning is a promising approach to improve the forward-propagation speed and reduce the power consumption. However, present methods are based on a parallel processing approach that is inherently ineffective in dealing with serial data signals at the core of information and communication technologies. Here, we propose and demonstrate a serial optical deep learning concept that is specifically designed to directly process high-speed temporal data. By utilizing ultra-short coherent optical pulses as the information carriers, the neurons are distributed at different time slots in a serial pattern, and interconnected to each other through group delay dispersion. A 4-layer serial optical neural network (SONN) was constructed and trained for classification of both analog and digital signals with simulated accuracy rates of over 90% with proper individuality variance rates. Furthermore, we performed a proof-of-concept experiment of a pseudo-3-layer SONN to successfully recognize the ASCII (American Standard Code for Information Interchange) codes of English letters at a data rate of 12 Gbps. This concept represents a novel one-dimensional realization of artificial neural networks, enabling an efficient application of optical deep learning methods to the analysis and processing of serial data signals, while offering a new overall perspective for the temporal signal processing.
Deep learning methods can be found in many medical imaging applications. Recently, those methods were applied directly to the RF ultrasound multi-channel data to enhance the quality of the reconstructed images. In this paper, we apply a deep neural network to medical ultrasound imaging in the beamforming stage. Specifically, we train the network using simulated multi-channel data from two arrays with different sizes, using a variety of direction of arrival (DOA) angles, and test its generalization performance on real cardiac data. We demonstrate that our method can be used to improve image quality over standard methods, both in terms of resolution and contrast. Alternatively, it can be used to reduce the number of elements in the array, while maintaining the image quality. The utility of our method is demonstrated on both simulated and real data.
Machine learning (ML) and artificial neural networks (ANNs) have been successfully applied to simulating complex physics by learning physics models thanks to large data. Inspired by the successes of ANNs in physics modeling, we use deep neural networks (DNNs) to predict the radio signal strength field in an urban environment. Our algorithm relies on samples of signal strength collected across the prediction space and a 3D map of the environment, which enables it to predict the scattering of radio waves through the environment. While already extensive body of research exists in spatial signal strength prediction, our approach differs from most existing approaches in that it does not require the knowledge of the transmitter location, it does not require side channel information such as attenuation and shadowing parameters, and it is the first work, to the best of our knowledge, to use 3D maps to accomplish the task of signal strength prediction. This algorithm is developed for the purpose of placement optimization of a UAV or mobile robot to maximize the signal strength to or from a stationary transceiver but it also holds relevance to dynamic spectrum access networks, cellular coverage design, power control algorithms, etc.
Biological measurements are often contaminated with large amounts of non-stationary noise which require effective noise reduction techniques. We present a new real-time deep learning algorithm which produces adaptively a signal opposing the noise so that destructive interference occurs. As a proof of concept, we demonstrate the algorithm's performance by reducing electromyogram noise in electroencephalograms with the usage of a custom, flexible, 3D-printed, compound electrode. With this setup, an average of 4dB and a maximum of 10dB improvement of the signal-to-noise ratio of the EEG was achieved by removing wide band muscle noise. This concept has the potential to not only adaptively improve the signal-to-noise ratio of EEG but can be applied to a wide range of biological, industrial and consumer applications such as industrial sensing or noise cancelling headphones.
Recently, deep learned enabled end-to-end (E2E) communication systems have been developed to merge all physical layer blocks in the traditional communication systems, which make joint transceiver optimization possible. Powered by deep learning, natural language processing (NLP) has achieved great success in analyzing and understanding large amounts of language texts. Inspired by research results in both areas, we aim to providing a new view on communication systems from the semantic level. Particularly, we propose a deep learning based semantic communication system, named DeepSC, for text transmission. Based on the Transformer, the DeepSC aims at maximizing the system capacity and minimizing the semantic errors by recovering the meaning of sentences, rather than bit- or symbol-errors in traditional communications. Moreover, transfer learning is used to ensure the DeepSC applicable to different communication environments and to accelerate the model training process. To justify the performance of semantic communications accurately, we also initialize a new metric, named sentence similarity. Compared with the traditional communication system without considering semantic information exchange, the proposed DeepSC is more robust to channel variation and is able to achieve better performance, especially in the low signal-to-noise (SNR) regime, as demonstrated by the extensive simulation results.
In the paper we study a deep learning based method to solve the multicell power control problem for sum rate maximization subject to per-user rate constraints and per-base station (BS) power constraints. The core difficulty of this problem is how to ensure that the learned power control results by the deep neural network (DNN) satisfy the per-user rate constraints. To tackle the difficulty, we propose to cascade a projection block after a traditional DNN, which projects the infeasible power control results onto the constraint set. The projection block is designed based on a geometrical interpretation of the constraints, which is of low complexity, meeting the real-time requirement of online applications. Explicit-form expression of the backpropagated gradient is derived for the proposed projection block, with which the DNN can be trained to directly maximize the sum rate via unsupervised learning. We also develop a heuristic implementation of the projection block to reduce the size of DNN. Simulation results demonstrate the advantages of the proposed method over existing deep learning and numerical optimization~methods, and show the robustness of the proposed method with the model mismatch between training and testing~datasets.
We present a deep learning solution to the problem of localization of magnetoencephalography (MEG) brain signals. The proposed deep model architectures are tuned for single and multiple time point MEG data, and can estimate varying numbers of dipole sources. Results from simulated MEG data on the cortical surface of a real human subject demonstrated improvements against the popular RAP-MUSIC localization algorithm in specific scenarios with varying SNR levels, inter-source correlation values, and number of sources. Importantly, the deep learning models had robust performance to forward model errors and a significant reduction in computation time, to a fraction of 1 ms, paving the way to real-time MEG source localization.