Background:The electrocardiogram (ECG) is one of the most commonly used diagnostic tools in medicine and healthcare. Deep learning methods have achieved promising results on predictive healthcare tasks using ECG signals. Objective:This paper presents a systematic review of deep learning methods for ECG data from both modeling and application perspectives. Methods:We extracted papers that applied deep learning (deep neural network) models to ECG data that were published between Jan. 1st of 2010 and Feb. 29th of 2020 from Google Scholar, PubMed, and the DBLP. We then analyzed each article according to three factors: tasks, models, and data. Finally, we discuss open challenges and unsolved problems in this area. Results: The total number of papers extracted was 191. Among these papers, 108 were published after 2019. Different deep learning architectures have been used in various ECG analytics tasks, such as disease detection/classification, annotation/localization, sleep staging, biometric human identification, and denoising. Conclusion: The number of works on deep learning for ECG data has grown explosively in recent years. Such works have achieved accuracy comparable to that of traditional feature-based approaches and ensembles of multiple approaches can achieve even better results. Specifically, we found that a hybrid architecture of a convolutional neural network and recurrent neural network ensemble using expert features yields the best results. However, there are some new challenges and problems related to interpretability, scalability, and efficiency that must be addressed. Furthermore, it is also worth investigating new applications from the perspectives of datasets and methods. Significance: This paper summarizes existing deep learning research using ECG data from multiple perspectives and highlights existing challenges and problems to identify potential future research directions.
This study proposes a novel biologically-motivated learning method for deep convolutional neural networks (CNNs). The combination of CNNs and back propagation (BP) learning is the most powerful method in recent machine learning regimes. However, it requires large labeled data for training, and this requirement can occasionally become a barrier for real world applications. To address this problem and utilize unlabeled data, I propose to introduce unsupervised competitive learning which only requires forward propagating signals as a pre-training method for CNNs. The method was evaluated by image discrimination tasks using MNIST, CIFAR-10, and ImageNet datasets, and it achieved a state-of-the-art performance as a biologically-motivated method in the ImageNet experiment. The results suggested that the method enables higher-level learning representations solely from forward propagating signals without a backward error signal for the learning of convolutional layers. The proposed method could be useful for a variety of poorly labeled data, for example, time series or medical data.
This paper has proposed a new baseline deep learning model of more benefits for image classification. Different from the convolutional neural network(CNN) practice where filters are trained by back propagation to represent different patterns of an image, we are inspired by a method called "PCANet" in "PCANet: A Simple Deep Learning Baseline for Image Classification?" to choose filter vectors from basis vectors in frequency domain like Fourier coefficients or wavelets without back propagation. Researchers have demonstrated that those basis in frequency domain can usually provide physical insights, which adds to the interpretability of the model by analyzing the frequencies selected. Besides, the training process will also be more time efficient, mathematically clear and interpretable compared with the "black-box" training process of CNN.
Deep learning models have been criticized for their lack of easy interpretation, which undermines confidence in their use for important applications. Nevertheless, they are consistently utilized in many applications, consequential to humans' lives, mostly because of their better performance. Therefore, there is a great need for computational methods that can explain, audit, and debug such models. Here, we use flip points to accomplish these goals for deep learning models with continuous output scores (e.g., computed by softmax), used in social applications. A flip point is any point that lies on the boundary between two output classes: e.g. for a model with a binary yes/no output, a flip point is any input that generates equal scores for "yes" and "no". The flip point closest to a given input is of particular importance because it reveals the least changes in the input that would change a model's classification, and we show that it is the solution to a well-posed optimization problem. Flip points also enable us to systematically study the decision boundaries of a deep learning classifier. The resulting insight into the decision boundaries of a deep model can clearly explain the model's output on the individual-level, via an explanation report that is understandable by non-experts. We also develop a procedure to understand and audit model behavior towards groups of people. Flip points can also be used to alter the decision boundaries in order to improve undesirable behaviors. We demonstrate our methods by investigating several models trained on standard datasets used in social applications of machine learning. We also identify the features that are most responsible for particular classifications and misclassifications.
Nowadays, deep learning methods, especially the convolutional neural networks (CNNs), have shown impressive performance on extracting abstract and high-level features from the hyperspectral image. However, general training process of CNNs mainly considers the pixel-wise information or the samples' correlation to formulate the penalization while ignores the statistical properties especially the spectral variability of each class in the hyperspectral image. These samples-based penalizations would lead to the uncertainty of the training process due to the imbalanced and limited number of training samples. To overcome this problem, this work characterizes each class from the hyperspectral image as a statistical distribution and further develops a novel statistical loss with the distributions, not directly with samples for deep learning. Based on the Fisher discrimination criterion, the loss penalizes the sample variance of each class distribution to decrease the intra-class variance of the training samples. Moreover, an additional diversity-promoting condition is added to enlarge the inter-class variance between different class distributions and this could better discriminate samples from different classes in hyperspectral image. Finally, the statistical estimation form of the statistical loss is developed with the training samples through multi-variant statistical analysis. Experiments over the real-world hyperspectral images show the effectiveness of the developed statistical loss for deep learning.
Compared with cheap addition operation, multiplication operation is of much higher computation complexity. The widely-used convolutions in deep neural networks are exactly cross-correlation to measure the similarity between input feature and convolution filters, which involves massive multiplications between float values. In this paper, we present adder networks (AdderNets) to trade these massive multiplications in deep neural networks, especially convolutional neural networks (CNNs), for much cheaper additions to reduce computation costs. In AdderNets, we take the $\ell_1$-norm distance between filters and input feature as the output response. The influence of this new similarity measure on the optimization of neural network have been thoroughly analyzed. To achieve a better performance, we develop a special back-propagation approach for AdderNets by investigating the full-precision gradient. We then propose an adaptive learning rate strategy to enhance the training procedure of AdderNets according to the magnitude of each neuron's gradient. As a result, the proposed AdderNets can achieve 74.9% Top-1 accuracy 91.7% Top-5 accuracy using ResNet-50 on the ImageNet dataset without any multiplication in convolution layer. The codes are publicly available at: https://github.com/huaweinoah/AdderNet.
We describe AraNet, a collection of deep learning Arabic social media processing tools. Namely, we exploit an extensive host of publicly available and novel social media datasets to train bidirectional encoders from transformer models (BERT) to predict age, dialect, gender, emotion, irony, and sentiment. AraNet delivers state-of-the-art performance on a number of the cited tasks and competitively on others. In addition, AraNet has the advantage of being exclusively based on a deep learning framework and hence feature engineering free. To the best of our knowledge, AraNet is the first to performs predictions across such a wide range of tasks for Arabic NLP and thus meets a critical needs. We publicly release AraNet to accelerate research and facilitate comparisons across the different tasks.
Deep learning has achieved good success in cardiac magnetic resonance imaging (MRI) reconstruction, in which convolutional neural networks (CNNs) learn a mapping from the undersampled k-space to the fully sampled images. Although these deep learning methods can improve the reconstruction quality compared with iterative methods without requiring complex parameter selection or lengthy reconstruction time, the following issues still need to be addressed: 1) all these methods are based on big data and require a large amount of fully sampled MRI data, which is always difficult to obtain for cardiac MRI; 2) the effect of coil correlation on reconstruction in deep learning methods for dynamic MR imaging has never been studied. In this paper, we propose an unsupervised deep learning method for multi-coil cine MRI via a time-interleaved sampling strategy. Specifically, a time-interleaved acquisition scheme is utilized to build a set of fully encoded reference data by directly merging the k-space data of adjacent time frames. Then these fully encoded data can be used to train a parallel network for reconstructing images of each coil separately. Finally, the images from each coil are combined via a CNN to implicitly explore the correlations between coils. The comparisons with classic k-t FOCUSS, k-t SLR, L+S and KLR methods on in vivo datasets show that our method can achieve improved reconstruction results in an extremely short amount of time.
Automatic bill classification is an attractive task with many potential applications such as automated detection and counting in images or videos. To address this purpose we present a Deep Learning Model to classify Chilean Banknotes, because of its successful results in image processing applications. For optimal performance of the proposed model, data augmentation techniques are introduced due to the limited number of image samples. Positive results were achieved in this work, verifying that it could be a stating point to be extended to more complex applications.
Deep learning object detectors often return false positives with very high confidence. Although they optimize generic detection performance, such as mean average precision (mAP), they are not designed for reliability. For a reliable detection system, if a high confidence detection is made, we would want high certainty that the object has indeed been detected. To achieve this, we have developed a set of verification tests which a proposed detection must pass to be accepted. We develop a theoretical framework which proves that, under certain assumptions, our verification tests will not accept any false positives. Based on an approximation to this framework, we present a practical detection system that can verify, with high precision, whether each detection of a machine-learning based object detector is correct. We show that these tests can improve the overall accuracy of a base detector and that accepted examples are highly likely to be correct. This allows the detector to operate in a high precision regime and can thus be used for robotic perception systems as a reliable instance detection method. Code is available at https://github.com/siddancha/FlowVerify.
Point cloud learning has lately attracted increasing attention due to its wide applications in many areas, such as computer vision, autonomous driving, and robotics. As a dominating technique in AI, deep learning has been successfully used to solve various 2D vision problems. However, deep learning on point clouds is still in its infancy due to the unique challenges faced by the processing of point clouds with deep neural networks. Recently, deep learning on point clouds has become even thriving, with numerous methods being proposed to address different problems in this area. To stimulate future research, this paper presents a comprehensive review of recent progress in deep learning methods for point clouds. It covers three major tasks, including 3D shape classification, 3D object detection and tracking, and 3D point cloud segmentation. It also presents comparative results on several publicly available datasets, together with insightful observations and inspiring future research directions.
Attacks that aim to identify the training data of public neural networks represent a severe threat to the privacy of individuals participating in the training data set. A possible protection is offered by anonymization of the training data or training function with differential privacy. However, data scientists can choose between local and central differential privacy and need to select meaningful privacy parameters $\epsilon$ which is challenging for non-privacy experts. We empirically compare local and central differential privacy mechanisms under white- and black-box membership inference to evaluate their relative privacy-accuracy trade-offs. We experiment with several datasets and show that this trade-off is similar for both types of mechanisms. This suggests that local differential privacy is a sound alternative to central differential privacy for differentially private deep learning, since small $\epsilon$ in central differential privacy and large $\epsilon$ in local differential privacy result in similar membership inference attack risk.
Semi-Supervised Learning (SSL) algorithms have shown great potential in training regimes when access to labeled data is scarce but access to unlabeled data is plentiful. However, our experiments illustrate several shortcomings that prior SSL algorithms suffer from. In particular, poor performance when unlabeled and labeled data distributions differ. To address these observations, we develop RealMix, which achieves state-of-the-art results on standard benchmark datasets across different labeled and unlabeled set sizes while overcoming the aforementioned challenges. Notably, RealMix achieves an error rate of 9.79% on CIFAR10 with 250 labels and is the only SSL method tested able to surpass baseline performance when there is significant mismatch in the labeled and unlabeled data distributions. RealMix demonstrates how SSL can be used in real world situations with limited access to both data and compute and guides further research in SSL with practical applicability in mind.
We propose a novel quaternion product unit (QPU) to represent data on 3D rotation groups. The QPU leverages quaternion algebra and the law of 3D rotation group, representing 3D rotation data as quaternions and merging them via a weighted chain of Hamilton products. We prove that the representations derived by the proposed QPU can be disentangled into "rotation-invariant" features and "rotation-equivariant" features, respectively, which supports the rationality and the efficiency of the QPU in theory. We design quaternion neural networks based on our QPUs and make our models compatible with existing deep learning models. Experiments on both synthetic and real-world data show that the proposed QPU is beneficial for the learning tasks requiring rotation robustness.
Myocardial infarction is the leading cause of death worldwide. In this paper, we design domain-inspired neural network models to detect myocardial infarction. First, we study the contribution of various leads. This systematic analysis, first of its kind in the literature, indicates that out of 15 ECG leads, data from the v6, vz, and ii leads are critical to correctly identify myocardial infarction. Second, we use this finding and adapt the ConvNetQuake neural network model--originally designed to identify earthquakes--to attain state-of-the-art classification results for myocardial infarction, achieving $99.43\%$ classification accuracy on a record-wise split, and $97.83\%$ classification accuracy on a patient-wise split. These two results represent cardiologist-level performance level for myocardial infarction detection after feeding only 10 seconds of raw ECG data into our model. Third, we show that our multi-ECG-channel neural network achieves cardiologist-level performance without the need of any kind of manual feature extraction or data pre-processing.
Deep neural networks such as GoogLeNet, ResNet, and BERT have achieved impressive performance in tasks such as image and text classification. To understand how such performance is achieved, we probe a trained deep neural network by studying neuron activations, i.e., combinations of neuron firings, at various layers of the network in response to a particular input. With a large number of inputs, we aim to obtain a global view of what neurons detect by studying their activations. In particular, we develop visualizations that show the shape of the activation space, the organizational principle behind neuron activations, and the relationships of these activations within a layer. Applying tools from topological data analysis, we present TopoAct, a visual exploration system to study topological summaries of activation vectors. We present exploration scenarios using TopoAct that provide valuable insights into learned representations of neural networks. We expect TopoAct to give a topological perspective that enriches the current toolbox of neural network analysis, and to provide a basis for network architecture diagnosis and data anomaly detection.
We verified that the deep learning method named reading periodic table introduced by ref. Deep Learning Model for Finding New Superconductors, which utilizes deep learning to read the periodic table and the laws of the elements, is applicable not only for superconductors, for which the method was originally applied but also for other problems of materials by demonstrating band gap estimations. We then extended the method to learn the laws better by directly learning the cylindrical periodicity between the right- and left-most columns in the periodic table at the learning representation level, that is, by considering the left- and right-most columns to be adjacent to each other. Thus, while the original method handles the table as is, the extended method treats the periodic table as if its two edges are connected. This is achieved using novel layers named periodic convolution layers, which can handle inputs exhibiting periodicity and may be applied to other problems related to computer vision, time series, and so on for data that possess some periodicity. In the reading periodic table method, no material feature or descriptor is required as input. We demonstrated two types of deep learning estimation: methods to estimate the existence of a bandgap, and methods to estimate the value of the bandgap given when the existence of the bandgap in the materials is known. Finally, we discuss the limitations of the dataset and model evaluation method. We may be unable to distinguish good models based on the random train-test split scheme; thus, we must prepare an appropriate dataset where the training and test data are temporally separate. The code and data are open.
This report discusses three submissions based on the Duet architecture to the Deep Learning track at TREC 2019. For the document retrieval task, we adapt the Duet model to ingest a "multiple field" view of documents---we refer to the new architecture as Duet with Multiple Fields (DuetMF). A second submission combines the DuetMF model with other neural and traditional relevance estimators in a learning-to-rank framework and achieves improved performance over the DuetMF baseline. For the passage retrieval task, we submit a single run based on an ensemble of eight Duet models.
This letter introduces a deep learning (DL) framework for direction-of-arrival (DOA) estimation. Previous works in DL context mostly consider a single or two target scenario which is a strong limitation in practice. Hence, in this work, we propose a DL framework for multiple signal classification (DeepMUSIC). We design multiple deep convolutional neural networks (CNNs), each of which is dedicated to a subregion of the angular spectrum. In particular, each CNN is fed with the array covariance matrix and it learns the MUSIC spectra of the corresponding angular subregion. We have shown, through simulations, that the proposed DeepMUSIC framework has superior estimation accuracy and exhibits less computational complexity in comparison with both DL and non-DL based techniques.
There are many ways to represent a molecule as input to a machine learning model and each is associated with loss and retention of certain kinds of information. In the interest of preserving three-dimensional spatial information, including bond angles and torsions, we have developed libmolgrid, a general-purpose library for representing three-dimensional molecules using multidimensional arrays. This library also provides functionality for composing batches of data suited to machine learning workflows, including data augmentation, class balancing, and example stratification according to a regression variable or data subgroup, and it further supports temporal and spatial recurrences over that data to facilitate work with recurrent neural networks, dynamical data, and size extensive modeling. It was designed for seamless integration with popular deep learning frameworks, including Caffe, PyTorch, and Keras, providing good performance by leveraging graphical processing units (GPUs) for computationally-intensive tasks and efficient memory usage through the use of memory views over preallocated buffers. libmolgrid is a free and open source project that is actively supported, serving the growing need in the molecular modeling community for tools that streamline the process of data ingestion, representation construction, and principled machine learning model development.