In the speaker extraction problem, it is found that additional information from the target speaker contributes to the tracking and extraction of the target speaker, which includes voiceprint, lip movement, facial expression, and spatial information. However, no one cares for the cue of sound onset, which has been emphasized in the auditory scene analysis and psychology. Inspired by it, we explicitly modeled the onset cue and verified the effectiveness in the speaker extraction task. We further extended to the onset/offset cues and got performance improvement. From the perspective of tasks, our onset/offset-based model completes the composite task, a complementary combination of speaker extraction and speaker-dependent voice activity detection. We also combined voiceprint with onset/offset cues. Voiceprint models voice characteristics of the target while onset/offset models the start/end information of the speech. From the perspective of auditory scene analysis, the combination of two perception cues can promote the integrity of the auditory object. The experiment results are also close to state-of-the-art performance, using nearly half of the parameters. We hope that this work will inspire communities of speech processing and psychology, and contribute to communication between them. Our code will be available in https://github.com/aispeech-lab/wase/.
This paper briefly describes the systems presented by the Software Technologies Working Group (http://gtts.ehu.es, GTTS) of the University of the Basque Country (UPV/EHU) to the Query-by-Example Search on Speech Task (QUESST) at MediaEval 2014. The GTTS-EHU systems consist of four modules: (1) feature extraction; (2) speech activity detection; (3) DTW-based query matching; and (4) score calibration and fusion. The submitted systems follow the same approach used in our SWS 2013 submissions, with two minor changes (needed to address the new task): the search stops at the most likely query detection (no further detections are looked for) and a score is produced for each (query, document) pair. The two approximate matching types introduced in QUESST have not received special treatment. This year, we have just explored the use of reduced feature sets, obtaining worse results but at lower computational costs.
Internet of Things (IoT) has triggered a rapid increase in the number of connected devices and new use cases of wireless communications. To meet the new demands, the fifth generation (5G) of wireless communication systems features native machine type communication (MTC) services in addition to traditional human type communication (HTC) services. Some of the main challenges are the heterogeneous requirements and the sporadic traffic of massive MTC (mMTC), which makes the orthogonal allocation of resources infeasible. To overcome this problem, grant free non-orthogonal multiple access schemes have been proposed alongside with sparse signal recovery algorithms. While most of the related works have considered only homogeneous networks, we focus on a scenario where an enhanced mobile broadband (eMBB) device and multiple MTC devices share the same radio resources. We exploit the approximate message passing (AMP) algorithm for joint device activity detection and channel estimation of MTC devices in the presence of interference from eMBB, and evaluate the system performance in terms of receiver operating characteristics (ROC) and channel estimation errors. Moreover, we also propose two new pilot sequence generation strategies which improve the detection capabilities of the MTC receiver without affecting the eMBB service.
The task of voice activity detection (VAD) is an often required module in various speech processing, analysis and classification tasks. While state-of-the-art neural network based VADs can achieve great results, they often exceed computational budgets and real-time operating requirements. In this work, we propose a computationally efficient real-time VAD network that achieves state-of-the-art results on several public real recording datasets. We investigate different training targets for the VAD and show that using the segmental voice-to-noise ratio (VNR) is a better and more noise-robust training target than the clean speech level based VAD. We also show that multi-target training improves the performance further.
We propose a computationally efficient algorithm for the device activity detection problem in the multi-cell massive multi-input multi-output (MIMO) system, where the active devices transmit their signature sequences to multiple BSs in multiple cells and all the BSs cooperate to detect the active devices. The device activity detection problem has been formulated as a maximum likelihood maximization (MLE) problem in the literature. The state-of-the-art algorithm for solving the problem is the (random) coordinate descent (CD) algorithm. However, the CD algorithm fails to exploit the special sparsity structure of the solution of the device activity detection problem, i.e., most of devices are not active in each time slot. In this paper, we propose a novel active set selection strategy to accelerate the CD algorithm and propose an efficient active set CD algorithm for solving the considered problem. Specifically, at each iteration, the proposed active set CD algorithm first selects a small subset of all devices, namely the active set, which contains a few devices that contribute the most to the deviation from the first-order optimality condition of the MLE problem thus potentially can provide the most improvement to the objective function, then applies the CD algorithm to perform the detection for the devices in the active set. Simulation results show that the proposed active set CD algorithm significantly outperforms the state-of-the-art CD algorithm in terms of the computational efficiency.
In this paper, we propose a turbo receiver for joint activity detection and data decoding in grant-free massive random access, which iterates between a detector and a belief propagation (BP)-based channel decoder. Specifically, responsible for user activity detection, channel estimation, and soft data symbol detection, the detector is developed based on a bilinear inference problem that exploits the common sparsity pattern in the received pilot and data signals. The bilinear generalized approximate message passing (BiG-AMP) algorithm is adopted to solve the problem using probabilities of the transmitted data symbols estimated by the channel decoder as prior knowledge. In addition, extrinsic information is derived from the detector to improve the channel decoding accuracy of the decoder. Simulation results show significant improvements achieved by the proposed turbo receiver compared with conventional designs.
The audio segmentation mismatch between training data and those seen at run-time is a major problem in direct speech translation. Indeed, while systems are usually trained on manually segmented corpora, in real use cases they are often presented with continuous audio requiring automatic (and sub-optimal) segmentation. After comparing existing techniques (VAD-based, fixed-length and hybrid segmentation methods), in this paper we propose enhanced hybrid solutions to produce better results without sacrificing latency. Through experiments on different domains and language pairs, we show that our methods outperform all the other techniques, reducing by at least 30% the gap between the traditional VAD-based approach and optimal manual segmentation.
Grant-free access is considered as a key enabler to massive machine-type communications (mMTC) as it promotes energy-efficiency and small signalling overhead. Due to the sporadic user activity in mMTC, joint user identification and channel estimation (JUICE) is a main challenge. This paper addresses the JUICE in single-cell mMTC with single-antenna users and a multi-antenna base station (BS) under spatially correlated fading channels. In particular, by leveraging the sporadic user activity, we solve the JUICE in a multi measurement vector compressed sensing (CS) framework under two different cases, with and without the knowledge of prior channel distribution information (CDI) at the BS. First, for the case without prior information, we formulate the JUICE as an iterative reweighted $\ell_{2,1}$-norm minimization problem. Second, when the CDI is known to the BS, we exploit the available information and formulate the JUICE from a Bayesian estimation perspective as a maximum \emph{a posteriori} probability (MAP) estimation problem. For both JUICE formulations, we derive efficient iterative solutions based on the alternating direction method of multipliers (ADMM). The numerical experiments show that the proposed solutions achieve higher channel estimation quality and activity detection accuracy with shorter pilot sequences compared to existing algorithms.
Long-term complex activity recognition and localisation can be crucial for decision making in autonomous systems such as smart cars and surgical robots. Here we address the problem via a novel deformable, spatiotemporal scene graph approach, consisting of three main building blocks: (i) action tube detection, (ii) the modelling of the deformable geometry of parts, and (iii) a graph convolutional network. Firstly, action tubes are detected in a series of snippets. Next, a new 3D deformable RoI pooling layer is designed for learning the flexible, deformable geometry of the constituent action tubes. Finally, a scene graph is constructed by considering all parts as nodes and connecting them based on different semantics such as order of appearance, sharing the same action label and feature similarity. We also contribute fresh temporal complex activity annotation for the recently released ROAD autonomous driving and SARAS-ESAD surgical action datasets and show the adaptability of our framework to different domains. Our method is shown to significantly outperform graph-based competitors on both augmented datasets.
Secure wireless access in ultra-reliable low-latency communications (URLLC), which is a critical aspect of 5G security, has become increasingly important due to its potential support of grant-free configuration. In grant-free URLLC, precise allocation of different pilot resources to different users that share the same time-frequency resource is essential for the next generation NodeB (gNB) to exactly identify those users under access collision and to maintain precise channel estimation required for reliable data transmission. However, this process easily suffers from attacks on pilots. We in this paper propose a quantum learning based nonrandom superimposed coding method to encode and decode pilots on multidimensional resources, such that the uncertainty of attacks can be learned quickly and eliminated precisely. Particularly, multiuser pilots for uplink access are encoded as distinguishable subcarrier activation patterns (SAPs) and gNB decodes pilots of interest from observed SAPs, a superposition of SAPs from access users, by joint design of attack mode detection and user activity detection though a quantum learning network (QLN). We found that the uncertainty lies in the identification process of codeword digits from the attacker, which can be always modelled as a black-box model, resolved by a quantum learning algorithm and quantum circuit. Novel analytical closed-form expressions of failure probability are derived to characterize the reliability of this URLLC system with short packet transmission. Simulations how that our method can bring ultra-high reliability and low latency despite attacks on pilots.
Grant-free non-orthogonal multiple access (NOMA) is considered as one of the supporting technology for massive connectivity for future networks. In the grant-free NOMA systems with a massive number of users, user activity detection is of great importance. Existing multi-user detection (MUD) techniques rely on complicated update steps which may cause latency in signal detection. In this paper, we propose a generative neural network-based MUD (GenMUD) framework to utilize low-complexity neural networks, which are trained to reconstruct signals in a small fixed number of steps. By exploiting the uncorrelated user behaviours, we design a network architecture to achieve higher recovery accuracy with a low computational cost. Experimental results show significant performance gains in detection accuracy compared to conventional solutions under different channel conditions and user sparsity levels. We also provide a sparsity estimator through extensive experiments. Simulation results of the sparsity estimator showed high estimation accuracy, strong robustness to channel variations and neglectable impact on support detection accuracy.
Massive machine-type communication (mMTC) has been regarded as one of the most important use scenarios in the fifth generation (5G) and beyond wireless networks, which demands scalable access for a large number of devices. While grant-free random access has emerged as a promising mechanism for massive access, its potential has not been fully unleashed. Particularly, the two key tasks in massive access systems, namely, user activity detection and data detection, were handled separately in most existing studies, which ignored the common sparsity pattern in the received pilot and data signal. Moreover, error detection and correction in the payload data provide additional mechanisms for performance improvement. In this paper, we propose a data-assisted activity detection framework, which aims at supporting more active users by reducing the activity detection error, consisting of false alarm and missed detection errors. Specifically, after an initial activity detection step based on the pilot symbols, the false alarm users are filtered by applying energy detection for the data symbols; once data symbols of some active users have been successfully decoded, their effect in activity detection will be resolved via successive pilot interference cancellation, which reduces the missed detection error. Simulation results show that the proposed algorithm effectively increases the activity detection accuracy, and it is able to support $\sim 20\%$ more active users compared to a conventional method in some sample scenarios.
This paper proposes a computationally efficient algorithm to solve the joint data and activity detection problem for massive random access with massive multiple-input multiple-output (MIMO). The BS acquires the active devices and their data by detecting the transmitted preassigned nonorthogonal signature sequences. This paper employs a covariance based approach that formulates the detection problem as a maximum likelihood estimation (MLE) problem. To efficiently solve the problem, this paper designs a novel iterative algorithm with low complexity in the regime where the device activity pattern is sparse $\unicode{x2013}$ a key feature that existing algorithmic designs have not previously exploited for reducing complexity. Specifically, at each iteration, the proposed algorithm focuses on only a small subset of all potential sequences, namely the active set, which contains a few most likely active sequences (i.e., transmitted sequences by all active devices), and performs the detection for the sequences in the active set. The active set is carefully selected at each iteration based on the current detection result and the first-order optimality condition of the MLE problem. Simulation results show that the proposed active set algorithm enjoys significantly better computational efficiency (in terms of the CPU time) than the state-of-the-art algorithms.
Voice activity detection (VAD) remains a challenge in noisy environments. With access to multiple microphones, prior studies have attempted to improve the noise robustness of VAD by creating multi-channel VAD (MVAD) methods. However, MVAD is relatively new compared to single-channel VAD (SVAD), which has been thoroughly developed in the past. It might therefore be advantageous to improve SVAD methods with pre-processing to obtain superior VAD, which is under-explored. This paper improves SVAD through two pre-processing methods, a beamformer and a spatial target speaker detector. The spatial detector sets signal frames to zero when no potential speaker is present within a target direction. The detector may be implemented as a filter, meaning the input signal for the SVAD is filtered according to the detector's output; or it may be implemented as a spatial VAD to be combined with the SVAD output. The evaluation is made on a noisy reverberant speech database, with clean speech from the Aurora 2 database and with white and babble noise. The results show that SVAD algorithms are significantly improved by the presented pre-processing methods, especially the spatial detector, across all signal-to-noise ratios. The SVAD algorithms with pre-processing significantly outperform a baseline MVAD in challenging noise conditions.
Speaker segmentation consists in partitioning a conversation between one or more speakers into speaker turns. Usually addressed as the late combination of three sub-tasks (voice activity detection, speaker change detection, and overlapped speech detection), we propose to train an end-to-end segmentation model that does it directly. Inspired by the original end-to-end neural speaker diarization approach (EEND), the task is modeled as a multi-label classification problem using permutation-invariant training. The main difference is that our model operates on short audio chunks (5 seconds) but at a much higher temporal resolution (every 16ms). Experiments on multiple speaker diarization datasets conclude that our model can be used with great success on both voice activity detection and overlapped speech detection. Our proposed model can also be used as a post-processing step, to detect and correctly assign overlapped speech regions. Relative diarization error rate improvement over the best considered baseline (VBx) reaches 17% on AMI, 13% on DIHARD 3, and 13% on VoxConverse.
Current IoT networks are characterized by an ultra-high density of devices with different energy budget constraints, typically having sparse and sporadic activity patterns. Access points require an efficient strategy to identify the active devices for a timely allocation of resources to enable massive machine-type communication. Recently, group testing based approaches have been studied to handle sparse activity detection in massive random access problems. In this paper, a non-adaptive group testing strategy is proposed which can take into account the energy constraints on different sensor clusters. A theoretical extension of the existing randomized group testing strategies to the case of multiple clusters is presented and the necessary constraints that the optimal sampling parameters should satisfy in order to improve the efficiency of group tests is established. The cases of fixed activity pattern where there is a fixed set of active sensors and random activity pattern where each sensor can be independently active with certain probability are examined. The theoretical results are verified and validated by Monte-Carlo simulations. In massive wireless sensor networks comprising of devices with different energy efficiencies, our proposed low-power-use mode of access can potentially extend the lifetime of battery powered sensors with finite energy budget.
In recent years malware has become increasingly sophisticated and difficult to detect prior to exploitation. While there are plenty of approaches to malware detection, they all have shortcomings when it comes to identifying malware correctly prior to exploitation. The trade-off is usually between false positives, causing overhead, preventing normal usage and the risk of letting the malware execute and cause damage to the target. We present a novel end-to-end solution for in-memory malicious activity detection done prior to exploitation by leveraging machine learning capabilities based on data from unique run-time logs, which are carefully curated in order to detect malicious activity in the memory of protected processes. This solution achieves reduced overhead and false positives as well as deployment simplicity. We implemented our solution for Windows-based systems, employing multi disciplinary knowledge from malware research, machine learning, and operating system internals. Our experimental evaluation yielded promising results. As we expect future sophisticated malware may try to bypass it, we also discuss how our solution can be extended to thwart such bypassing attempts.
Joint user identification and channel estimation (JUICE) is a main challenge in grant-free massive machine-type communications (mMTC). The sparse pattern in users' activity allows to solve the JUICE as a compressed sensing problem in a multiple measurement vector (MMV) setup. This paper addresses the JUICE under the practical spatially correlated fading channel. We formulate the JUICE as an iterative reweighted $\ell_{2,1}$-norm optimization. We develop a computationally efficient alternating direction method of multipliers (ADMM) approach to solve it. In particular, by leveraging the second-order statistics of the channels, we reformulate the JUICE problem to exploit the covariance information and we derive its ADMM-based solution. The simulation results highlight the significant improvements brought by the proposed approach in terms of channel estimation and activity detection performances.
This paper presents a new hybrid architecture for voice activity detection (VAD) incorporating both convolutional neural network (CNN) and bidirectional long short-term memory (BiLSTM) layers trained in an end-to-end manner. In addition, we focus specifically on optimising the computational efficiency of our architecture in order to deliver robust performance in difficult in-the-wild noise conditions in a severely under-resourced setting. Nested k-fold cross-validation was used to explore the hyperparameter space, and the trade-off between optimal parameters and model size is discussed. The performance effect of a BiLSTM layer compared to a unidirectional LSTM layer was also considered. We compare our systems with three established baselines on the AVA-Speech dataset. We find that significantly smaller models with near optimal parameters perform on par with larger models trained with optimal parameters. BiLSTM layers were shown to improve accuracy over unidirectional layers by $\approx$2% absolute on average. With an area under the curve (AUC) of 0.951, our system outperforms all baselines, including a much larger ResNet system, particularly in difficult noise conditions.
Recently, the end-to-end approach has been successfully applied to multi-speaker speech separation and recognition in both single-channel and multichannel conditions. However, severe performance degradation is still observed in the reverberant and noisy scenarios, and there is still a large performance gap between anechoic and reverberant conditions. In this work, we focus on the multichannel multi-speaker reverberant condition, and propose to extend our previous framework for end-to-end dereverberation, beamforming, and speech recognition with improved numerical stability and advanced frontend subnetworks including voice activity detection like masks. The techniques significantly stabilize the end-to-end training process. The experiments on the spatialized wsj1-2mix corpus show that the proposed system achieves about 35% WER relative reduction compared to our conventional multi-channel E2E ASR system, and also obtains decent speech dereverberation and separation performance (SDR=12.5 dB) in the reverberant multi-speaker condition while trained only with the ASR criterion.