Deep generative models are challenging the classical methods in the field of anomaly detection nowadays. Every new method provides evidence of outperforming its predecessors, often with contradictory results. The objective of this comparison is twofold: to compare anomaly detection methods of various paradigms with focus on deep generative models, and identification of sources of variability that can yield different results. The methods were compared on popular tabular and image datasets. We identified the main sources of variability to be experimental conditions: i) the type data set (tabular or image) and the nature of anomalies (statistical or semantic), and ii) strategy of selection of hyperparameters, especially the number of available anomalies in the validation set. Different methods perform the best in different contexts, i.e. combination of experimental conditions together with computational time. This explains the variability of the previous results and highlights the importance of careful specification of the context in the publication of a new method. All our code and results are available for download.
We consider bandit optimization of a smooth reward function, where the goal is cumulative regret minimization. This problem has been studied for $\alpha$-H\"older continuous (including Lipschitz) functions with $0<\alpha\leq 1$. Our main result is in generalization of the reward function to H\"older space with exponent $\alpha>1$ to bridge the gap between Lipschitz bandits and infinitely-differentiable models such as linear bandits. For H\"older continuous functions, approaches based on random sampling in bins of a discretized domain suffices as optimal. In contrast, we propose a class of two-layer algorithms that deploy misspecified linear/polynomial bandit algorithms in bins. We demonstrate that the proposed algorithm can exploit higher-order smoothness of the function by deriving a regret upper bound of $\tilde{O}(T^\frac{d+\alpha}{d+2\alpha})$ for when $\alpha>1$, which matches existing lower bound. We also study adaptation to unknown function smoothness over a continuous scale of H\"older spaces indexed by $\alpha$, with a bandit model selection approach applied with our proposed two-layer algorithms. We show that it achieves regret rate that matches the existing lower bound for adaptation within the $\alpha\leq 1$ subset.
The problem of multimodal clustering arises whenever the data are gathered with several physically different sensors. Observations from different modalities are not necessarily aligned in the sense there there is no obvious way to associate or to compare them in some common space. A solution may consist in considering multiple clustering tasks independently for each modality. The main difficulty with such an approach is to guarantee that the unimodal clusterings are mutually consistent. In this paper we show that multimodal clustering can be addressed within a novel framework, namely conjugate mixture models. These models exploit the explicit transformations that are often available between an unobserved parameter space (objects) and each one of the observation spaces (sensors). We formulate the problem as a likelihood maximization task and we derive the associated conjugate expectation-maximization algorithm. The convergence properties of the proposed algorithm are thoroughly investigated. Several local/global optimization techniques are proposed in order to increase its convergence speed. Two initialization strategies are proposed and compared. A consistent model-selection criterion is proposed. The algorithm and its variants are tested and evaluated within the task of 3D localization of several speakers using both auditory and visual data.
The computation required for a switching Kalman Filter (SKF) increases exponentially with the number of system operation modes. In this paper, a computationally tractable graph representation is proposed for a switching linear dynamic system (SLDS) along with the solution of a minimum-sum optimization problem for clustering to reduce the switching mode cardinality offline, before collecting measurements. It is shown that upon perfect mode detection, the induced error caused by mode clustering can be quantified exactly in terms of the dissimilarity measures in the proposed graph structure. Numerical results verify that clustering based on the proposed framework effectively reduces model complexity given uncertain mode detection and that the induced error can be well approximated if the underlying assumptions are satisfied.
For multi-class classification under class-conditional label noise, we prove that the accuracy metric itself can be robust. We concretize this finding's inspiration in two essential aspects: training and validation, with which we address critical issues in learning with noisy labels. For training, we show that maximizing training accuracy on sufficiently many noisy samples yields an approximately optimal classifier. For validation, we prove that a noisy validation set is reliable, addressing the critical demand of model selection in scenarios like hyperparameter-tuning and early stopping. Previously, model selection using noisy validation samples has not been theoretically justified. We verify our theoretical results and additional claims with extensive experiments. We show characterizations of models trained with noisy labels, motivated by our theoretical results, and verify the utility of a noisy validation set by showing the impressive performance of a framework termed noisy best teacher and student (NTS). Our code is released.
Modern neural network architectures typically have many millions of parameters and can be pruned significantly without substantial loss in effectiveness which demonstrates they are over-parameterized. The contribution of this work is two-fold. The first is a method for approximating a multivariate Bernoulli random variable by means of a deterministic and differentiable transformation of any real-valued multivariate random variable. The second is a method for model selection by element-wise multiplication of parameters with approximate binary gates that may be computed deterministically or stochastically and take on exact zero values. Sparsity is encouraged by the inclusion of a surrogate regularization to the $L_0$ loss. Since the method is differentiable it enables straightforward and efficient learning of model architectures by an empirical risk minimization procedure with stochastic gradient descent and theoretically enables conditional computation during training. The method also supports any arbitrary group sparsity over parameters or activations and therefore offers a framework for unstructured or flexible structured model pruning. To conclude experiments are performed to demonstrate the effectiveness of the proposed approach.
Motivated by a natural problem in online model selection with bandit information, we introduce and analyze a best arm identification problem in the rested bandit setting, wherein arm expected losses decrease with the number of times the arm has been played. The shape of the expected loss functions is similar across arms, and is assumed to be available up to unknown parameters that have to be learned on the fly. We define a novel notion of regret for this problem, where we compare to the policy that always plays the arm having the smallest expected loss at the end of the game. We analyze an arm elimination algorithm whose regret vanishes as the time horizon increases. The actual rate of convergence depends in a detailed way on the postulated functional form of the expected losses. Unlike known model selection efforts in the recent bandit literature, our algorithm exploits the specific structure of the problem to learn the unknown parameters of the expected loss function so as to identify the best arm as quickly as possible. We complement our analysis with a lower bound, indicating strengths and limitations of the proposed solution.
Estimation of the number of signals in the presence of noise is an important problem in several areas of statistical signal processing. There are a number of modern works on the design of an optimal solution to this problem in terms of some criteria. Each criterion generates a model order selection (MOS) algorithm. However, the minimum error probability criterion has not received significant attention, despite its importance. In this paper, we propose a new approach to the design of MOS algorithms partially based on the minimum error probability criterion. Also, we pay a lot of attention to the performance and consistency analysis of the MOS algorithms. In this study, an abridged error probability is used as a universal performance measure of the MOS algorithms. We propose a theoretical framework that makes it possible to provide a parametric consistency analysis and to obtain closed-form expressions for the abridged error probabilities of a wide range of MOS algorithms. As an example, we applied this framework to the consistency and performance analysis of several MOS algorithms. Using the obtained results, we provide a parametric optimization of the presented MOS algorithms. Finally, we examine a quasilikelihood approach to the design and analysis of the MOS algorithms. The proposed theoretical framework is used to obtain the abridged error probabilities as functions of the unknown signal parameter. These functions, in turn, allow us to find the scope of the quasilikelihood approach.
In this paper, we study the problem of direction of arrival estimation and model order selection for systems employing subarray sampling. Thereby, we focus on scenarios, where the number of active sources is not smaller than the number of simultaneously sampled antenna elements. For this purpose, we propose new schemes based on neural networks and estimators that combine neural networks with gradient steps on the likelihood function. These methods are able to outperform existing estimators in terms of mean squared error and model selection accuracy, especially in the low snapshot domain, at a drastically lower computational complexity.
In order to price contingent claims one needs to first understand the dynamics of these indices. Here we provide a first econometric analysis of the CRIX family within a time-series framework. The key steps of our analysis include model selection, estimation and testing. Linear dependence is removed by an ARIMA model, the diagnostic checking resulted in an ARIMA(2,0,2) model for the available sample period from Aug 1st, 2014 to April 6th, 2016. The model residuals showed the well known phenomenon of volatility clustering. Therefore a further refinement lead us to an ARIMA(2,0,2)-t-GARCH(1,1) process. This specification conveniently takes care of fat-tail properties that are typical for financial markets. The multivariate GARCH models are implemented on the CRIX index family to explore the interaction.
The cryptocurrency market is unique on many levels: Very volatile, frequently changing market structure, emerging and vanishing of cryptocurrencies on a daily level. Following its development became a difficult task with the success of cryptocurrencies (CCs) other than Bitcoin. For fiat currency markets, the IMF offers the index SDR and, prior to the EUR, the ECU existed, which was an index representing the development of European currencies. Index providers decide on a fixed number of index constituents which will represent the market segment. It is a challenge to fix a number and develop rules for the constituents in view of the market changes. In the frequently changing CC market, this challenge is even more severe. A method relying on the AIC is proposed to quickly react to market changes and therefore enable us to create an index, referred to as CRIX, for the cryptocurrency market. CRIX is chosen by model selection such that it represents the market well to enable each interested party studying economic questions in this market and to invest into the market. The diversified nature of the CC market makes the inclusion of altcoins in the index product critical to improve tracking performance. We have shown that assigning optimal weights to altcoins helps to reduce the tracking errors of a CC portfolio, despite the fact that their market cap is much smaller relative to Bitcoin. The codes used here are available via www.quantlet.de.
Many economic panel and dynamic models, such as rational behavior and Euler equations, imply that the parameters of interest are identified by conditional moment restrictions. We introduce a novel inference method without any prior information about which conditioning instruments are weak or irrelevant. Building on Bierens (1990), we propose penalized maximum statistics and combine bootstrap inference with model selection. Our method optimizes asymptotic power by solving a data-dependent max-min problem for tuning parameter selection. Extensive Monte Carlo experiments, based on an empirical example, demonstrate the extent to which our inference procedure is superior to those available in the literature.
A Hidden Markov Model for intraday momentum trading is presented which specifies a latent momentum state responsible for generating the observed securities' noisy returns. Existing momentum trading models suffer from time-lagging caused by the delayed frequency response of digital filters. Time-lagging results in a momentum signal of the wrong sign, when the market changes trend direction. A key feature of this state space formulation, is no such lagging occurs, allowing for accurate shifts in signal sign at market change points. The number of latent states in the model is estimated using three techniques, cross validation, penalized likelihood criteria and simulation-based model selection for the marginal likelihood. All three techniques suggest either 2 or 3 hidden states. Model parameters are then found using Baum-Welch and Markov Chain Monte Carlo, whilst assuming a single (discretized) univariate Gaussian distribution for the emission matrix. Often a momentum trader will want to condition their trading signals on additional information. To reflect this, learning is also carried out in the presence of side information. Two sets of side information are considered, namely a ratio of realized volatilities and intraday seasonality. It is shown that splines can be used to capture statistically significant relationships from this information, allowing returns to be predicted. An Input Output Hidden Markov Model is used to incorporate these univariate predictive signals into the transition matrix, presenting a possible solution for dealing with the signal combination problem. Bayesian inference is then carried out to predict the securities $t+1$ return using the forward algorithm. Simple modifications to the current framework allow for a fully non-parametric model with asynchronous prediction.
In this study, we develop a methodology for model reduction and selection informed by global sensitivity analysis (GSA) methods. We apply these techniques to a control model that takes systolic blood pressure and thoracic tissue pressure data as inputs and predicts heart rate in response to the Valsalva maneuver (VM). The study compares four GSA methods based on Sobol' indices (SIs) quantifying the parameter influence on the difference between the model output and the heart rate data. The GSA methods include standard scalar SIs determining the average parameter influence over the time interval studied and three time-varying methods analyzing how parameter influence changes over time. The time-varying methods include a new technique, termed limited-memory SIs, predicting parameter influence using a moving window approach. Using the limited-memory SIs, we perform model reduction and selection to analyze the necessity of modeling both the aortic and carotid baroreceptor regions in response to the VM. We compare the original model to three systematically reduced models including (i) the aortic and carotid regions, (ii) the aortic region only, and (iii) the carotid region only. Model selection is done quantitatively using the Akaike and Bayesian Information Criteria and qualitatively by comparing the neurological predictions. Results show that it is necessary to incorporate both the aortic and carotid regions to model the VM.
Equation learning methods present a promising tool to aid scientists in the modeling process for biological data. Previous equation learning studies have demonstrated that these methods can infer models from rich datasets, however, the performance of these methods in the presence of common challenges from biological data has not been thoroughly explored. We present an equation learning methodology comprised of data denoising, equation learning, model selection and post-processing steps that infers a dynamical systems model from noisy spatiotemporal data. The performance of this methodology is thoroughly investigated in the face of several common challenges presented by biological data, namely, sparse data sampling, large noise levels, and heterogeneity between datasets. We find that this methodology can accurately infer the correct underlying equation and predict unobserved system dynamics from a small number of time samples when the data is sampled over a time interval exhibiting both linear and nonlinear dynamics. Our findings suggest that equation learning methods can be used for model discovery and selection in many areas of biology when an informative dataset is used. We focus on glioblastoma multiforme modeling as a case study in this work to highlight how these results are informative for data-driven modeling-based tumor invasion predictions.
The deluge of single-cell data obtained by sequencing, imaging and epigenetic markers has led to an increasingly detailed description of cell state. However, it remains challenging to identify how cells transition between different states, in part because data are typically limited to snapshots in time. A prerequisite for inferring cell state transitions from such snapshots is to distinguish whether transitions are coupled to cell divisions. To address this, we present two minimal branching process models of cell division and differentiation in a well-mixed population. These models describe dynamics where differentiation and division are coupled or uncoupled. For each model, we derive analytic expressions for each subpopulation's mean and variance and for the likelihood, allowing exact Bayesian parameter inference and model selection in the idealised case of fully observed trajectories of differentiation and division events. In the case of snapshots, we present a sample path algorithm and use this to predict optimal temporal spacing of measurements for experimental design. We then apply this methodology to an \textit{in vitro} dataset assaying the clonal growth of epiblast stem cells in culture conditions promoting self-renewal or differentiation. Here, the larger number of cell states necessitates approximate Bayesian computation. For both culture conditions, our inference supports the model where cell state transitions are coupled to division. For culture conditions promoting differentiation, our analysis indicates a possible shift in dynamics, with these processes becoming more coupled over time.
Power system functionality is determined on the basis of the power system state estimation (PSSE). Thus, corruption of the PSSE may lead to severe consequences, such as financial losses, maintenance damage, and disruptions in electricity distribution. Classical bad data detection (BDD) methods, developed to ensure PSSE reliability, are unable to detect well-designed attacks, named unobservable false data injection (FDI) attacks. In this paper, we develop novel structural-constrained methods for the detection of unobservable FDI attacks, the identification of the attacked buses' locations, and PSSE under the presence of such attacks. The proposed methods are based on formulating structural, sparse constraints on both the attack and the system loads. First, we exploit these constraints in order to compose an appropriate model selection problem. Then, we develop the associated generalized information criterion (GIC) for this problem. However, for large networks, the GIC method's computational complexity grows exponentially with the network size. Thus, based on the proposed structural and sparse constraints, we develop two novel low-complexity methods for unobservable FDI attack identification: 1) a modification of the state-of-the-art orthogonal matching pursuit (OMP); and 2) a method that utilizes the graph Markovian property in power systems, i.e. the second-neighbor relationship between the power data at the system's buses. The methods' performance is evaluated on a IEEE-30 bus test case system.
This paper extends the analysis of Muni Toke and Yoshida (2020) to the case of marked point processes. We consider multiple marked point processes with intensities defined by three multiplicative components, namely a common baseline intensity, a state-dependent component specific to each process, and a state-dependent component specific to each mark within each process. We show that for specific mark distributions, this model is a combination of the ratio models defined in Muni Toke and Yoshida (2020). We prove convergence results for the quasi-maximum and quasi-Bayesian likelihood estimators of this model and provide numerical illustrations of the asymptotic variances. We use these ratio processes in order to model transactions occuring in a limit order book. Model flexibility allows us to investigate both state-dependency (emphasizing the role of imbalance and spread as significant signals) and clustering. Calibration, model selection and prediction results are reported for high-frequency trading data on multiple stocks traded on Euronext Paris. We show that the marked ratio model outperforms other intensity-based methods (such as "pure" Hawkes-based methods) in predicting the sign and aggressiveness of market orders on financial markets.
Biomedical data, particularly in the field of genomics, has characteristics which make it challenging for machine learning applications - it can be sparse, high dimensional and noisy. Biomedical applications also present challenges to model selection - whilst powerful, accurate predictions are necessary, they alone are not sufficient for a model to be deemed useful. Due to the nature of the predictions, a model must also be trustworthy and transparent, empowering a practitioner with confidence that its use is appropriate and reliable. In this paper, we propose that this can be achieved through the use of judiciously built feature sets coupled with Bayesian models, specifically Gaussian processes. We apply Gaussian processes to drug discovery, using inexpensive transcriptomic profiles from human cell lines to predict animal kidney and liver toxicity after treatment with specific chemical compounds. This approach has the potential to reduce invasive and expensive animal testing during clinical trials if in vitro human cell line analysis can accurately predict model animal phenotypes. We compare results across a range of feature sets and models, to highlight model importance for medical applications.
Quantile Factor Models (QFM) represent a new class of factor models for high-dimensional panel data. Unlike Approximate Factor Models (AFM), where only location-shifting factors can be extracted, QFM also allow to recover unobserved factors shifting other relevant parts of the distributions of observed variables. A quantile regression approach, labeled Quantile Factor Analysis (QFA), is proposed to consistently estimate all the quantile-dependent factors and loadings. Their asymptotic distribution is then derived using a kernel-smoothed version of the QFA estimators. Two consistent model selection criteria, based on information criteria and rank minimization, are developed to determine the number of factors at each quantile. Moreover, in contrast to the conditions required for the use of Principal Components Analysis in AFM, QFA estimation remains valid even when the idiosyncratic errors have heavy-tailed distributions. Three empirical applications (regarding macroeconomic, climate and finance panel data) provide evidence that extra factors shifting the quantiles other than the means could be relevant in practice.