Large Language Models (LLMs) have shown remarkable capabilities in text and multimodal processing, yet they fundamentally lack physical awareness--understanding of real-world physical phenomena. In this work, we present ACORN, a framework that teaches LLMs physical awareness through sound, focusing on fundamental physical phenomena like the Doppler effect, multipath effect, and spatial relationships. To overcome data scarcity, ACORN introduce a physics-based simulator combining real-world sound sources with controlled physical channels to generate diverse training data. Using this simulator, we build AQA-PHY, a comprehensive Audio Question-Answer dataset, and propose an audio encoder that processes both magnitude and phase information. By connecting our audio encoder to state-of-the-art LLMs, we demonstrate reasonable results in both simulated and real-world tasks, such as line-of-sight detection, Doppler effect estimation, and Direction-of-Arrival estimation, paving the way for enabling LLMs to understand physical world.
Over the past decades, there has been a surge of interest in studying low-dimensional structures within high-dimensional data. Statistical factor models $-$ i.e., low-rank plus diagonal covariance structures $-$ offer a powerful framework for modeling such structures. However, traditional methods for fitting statistical factor models, such as principal component analysis (PCA) or maximum likelihood estimation assuming the data is Gaussian, are highly sensitive to heavy tails and outliers in the observed data. In this paper, we propose a novel expectation-maximization (EM) algorithm for robustly fitting statistical factor models. Our approach is based on Tyler's M-estimator of the scatter matrix for an elliptical distribution, and consists of solving Tyler's maximum likelihood estimation problem while imposing a structural constraint that enforces the low-rank plus diagonal covariance structure. We present numerical experiments on both synthetic and real examples, demonstrating the robustness of our method for direction-of-arrival estimation in nonuniform noise and subspace recovery.
Sound event localization and detection (SELD) involves predicting active sound event classes over time while estimating their positions. The localization subtask in SELD is usually treated as a direction of arrival estimation problem, ignoring source distance. Only recently, SELD was extended to 3D by incorporating distance estimation, enabling the prediction of sound event positions in 3D space (3D SELD). However, existing methods lack input features designed for distance estimation. We argue that reverberation encodes valuable information for this task. This paper introduces two novel feature formats for 3D SELD based on reverberation: one using direct-to-reverberant ratio (DRR) and another leveraging signal autocorrelation to provide the model with insights into early reflections. Pre-training on synthetic data improves relative distance error (RDE) and overall SELD score, with autocorrelation-based features reducing RDE by over 3 percentage points on the STARSS23 dataset. The code to extract the features is available at github.com/dberghi/SELD-distance-features.
In future sixth-generation (6G) mobile networks, radar sensing is expected to be offered as an additional service to its original purpose of communication. Merging these two functions results in integrated sensing and communication (ISAC) systems. In this context, bistatic ISAC appears as a possibility to exploit the distributed nature of cellular networks while avoiding highly demanding hardware requirements such as full-duplex operation. Recent studies have introduced strategies to perform required synchronization and data exchange between nodes for bistatic ISAC operation, based on orthogonal frequency-division multiplexing (OFDM), however, only for single-input single-output architectures. In this article, a system concept for a bistatic multiple-input multiple-output (MIMO)-OFDM-based ISAC system with beamforming at both transmitter and receiver is proposed, and a distribution synchronization concept to ensure coherence among the different receive channels for direction-of-arrival estimation is presented. After a discussion on the ISAC processing chain, including relevant aspects for practical deployments such as transmitter digital pre-distortion and receiver calibration, a 4x8 MIMO measurement setup at 27.5 GHz and results are presented to validate the proposed system and distribution synchronization concepts.
One-bit quantization has garnered significant attention in recent years for various signal processing and communication applications. Estimating model parameters from one bit quantized data can be challenging, particularly when the quantization process is explicitly accounted for in the estimator. In many cases, the estimator disregards quantization effects, leading to model misspecification. Consequently, estimation errors arise from both quantization and misspecification. Traditional performance bounds, such as the Cramer-Rao bound (CRB), fail to capture the impact of misspecification on estimation performance. To address this limitation, we derive the misspecified CRB (MCRB) for parameter estimation in a quantized data model consisting of a signal component in additive Gaussian noise. We apply this bound to direction-of-arrival estimation using quantized measurements from a sensor array and to frequency estimation with oversampled quantized data. The simulations show that the MCRB is asymptotically achieved by the mean-squared-error of the misspecified maximum-likelihood estimator. Our results demonstrate that, unlike in finely quantized scenarios, oversampling can significantly enhance the estimation performance in the presence of misspecified one-bit quantized measurements.
Covariance matrix reconstruction has been the most widely used guiding objective in gridless direction-of-arrival (DoA) estimation for sparse linear arrays. Many semidefinite programming (SDP)-based methods fall under this category. Although deep learning-based approaches enable the construction of more sophisticated objective functions, most methods still rely on covariance matrix reconstruction. In this paper, we propose new loss functions that are invariant to the scaling of the matrices and provide a comparative study of losses with varying degrees of invariance. The proposed loss functions are formulated based on the scale-invariant signal-to-distortion ratio between the target matrix and the Gram matrix of the prediction. Numerical results show that a scale-invariant loss outperforms its non-invariant counterpart but is inferior to the recently proposed subspace loss that is invariant to the change of basis. These results provide evidence that designing loss functions with greater degrees of invariance is advantageous in deep learning-based gridless DoA estimation.
This paper introduces CADMM-Net and CHADMM-Net, two deep neural networks for direction of arrival estimation within the least-absolute shrinkage and selection operator (LASSO) framework. These two networks are based on a structured deep unfolding of the alternating direction method of multipliers (ADMM) algorithm through the use of circulant as well as Hermitian-circulant matrices. Along with a computational complexity of $\mathcal{O}(N\log(N))$ per layer for the inference, where $N$ is the length of the dictionary $\mathbf{A}$, they additionally exhibit a memory footprint of $N$ and approximately half of $N$ for CADMMNet and CHADMM-Net, respectively, compared with $N^{2}$ for ADMM-Net. Furthermore, these structured networks exhibit a competitive performance against ADMM-Net, LISTA, TLISTA, and THLISTA with respect to the detection rate, the angular root-mean square error, and the normalized mean squared error.
This paper presents Toeplitz-Hermitian ADMM-Net (THADMM-Net), a deep neural network obtained by deep unfolding the alternating direction method of multipliers (ADMM) algorithm for solving the least absolute shrinkage thresholding operator problem in the context of direction of arrival estimation. By imposing both a Toeplitz-Hermitian as well as positve semi-definite constraint on the learnable matrices, the total parameter count required per layer is reduced from $N^2$ to approximately $N$ where $N$ is the length of the dictionary used in the sparse recovery problem. Numerical simulations show that with a lower parameter count and depth, THADMM-Net outperforms Toeplitz-Lista with respect to the normalized mean-squared error, the detection rate, as well as the root mean-squared error over a signal-to-noise ratio between 0 dB and 35 dB.
We consider a phase-shift mixing model for linear sensor arrays in the context of blind source extraction. We derive a blind Capon beamformer that seeks the direction where the output is independent of the other signals in the mixture. The algorithm is based on Independent Component Extraction and imposes an orthogonal constraint, thanks to which it optimizes only one real-valued parameter related to the angle of arrival. The Cram\'er-Rao lower bound for the mean interference-to-signal ratio is derived. The algorithm and the bound are compared with conventional blind and direction-of-arrival estimation+beamforming methods, showing improvements in terms of extraction accuracy. An application is demonstrated in frequency-domain speaker extraction in a low-reverberation room.
In this paper, we propose an extension to the Propagator algorithm for source bearing estimation by performing root decomposition which eliminates the need for spectral search over angles. Further the propagator spatial spectrum is reused after performing the polynomial rooting to alleviate the observed drawback of an increase in RMSE at lower SNRs for estimating the directions-of-arrival of signals impinging on a linear array of sensors. Our proposed algorithm achieves a 98% reduction in computational complexity compared to Propagator alongside an improved angular resolution suitable for real time DOA estimation for wireless communications.
To estimate the direction of arrival (DOA) of multiple speakers, subspace-based prototype transfer function matching methods such as multiple signal classification (MUSIC) or relative transfer function (RTF) vector matching are commonly employed. In general, these methods require calibrated microphone arrays, which are characterized by a known array geometry or a set of known prototype transfer functions for several directions. In this paper, we consider a partially calibrated microphone array, composed of a calibrated binaural hearing aid and a (non-calibrated) external microphone at an unknown location with no available set of prototype transfer functions. We propose a procedure for completing sets of prototype transfer functions by exploiting the orthogonality of subspaces, allowing to apply matching-based DOA estimation methods with partially calibrated microphone arrays. For the MUSIC and RTF vector matching methods, experimental results for two speakers in noisy and reverberant environments clearly demonstrate that for all locations of the external microphone DOAs can be estimated more accurately with completed sets of prototype transfer functions than with incomplete sets. \c{opyright}20XX IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.
We propose a fully Bayesian approach to wideband, or broadband, direction-of-arrival (DoA) estimation and signal detection. Unlike previous works in wideband DoA estimation and detection, where the signals were modeled in the time-frequency domain, we directly model the time-domain representation and treat the non-causal part of the source signal as latent variables. Furthermore, our Bayesian model allows for closed-form marginalization of the latent source signals by leveraging conjugacy. To further speed up computation, we exploit the sparse ``stripe matrix structure'' of the considered system, which stems from the circulant matrix representation of linear time-invariant (LTI) systems. This drastically reduces the time complexity of computing the likelihood from $\mathcal{O}(N^3 k^3)$ to $\mathcal{O}(N k^3)$, where $N$ is the number of samples received by the array and $k$ is the number of sources. These computational improvements allow for efficient posterior inference through reversible jump Markov chain Monte Carlo (RJMCMC). We use the non-reversible extension of RJMCMC (NRJMCMC), which often achieves lower autocorrelation and faster convergence than the conventional reversible variant. Detection, estimation, and reconstruction of the latent source signals can then all be performed in a fully Bayesian manner through the samples drawn using NRJMCMC. We evaluate the detection performance of the procedure by comparing against generalized likelihood ratio testing (GLRT) and information criteria.
Sound event localization and detection (SELD) has seen substantial advancements through learning-based methods. These systems, typically trained from scratch on specific datasets, have shown considerable generalization capabilities. Recently, deep neural networks trained on large-scale datasets have achieved remarkable success in the sound event classification (SEC) field, prompting an open question of whether these advancements can be extended to develop general-purpose SELD models. In this paper, leveraging the power of pre-trained SEC models, we propose pre-trained SELD networks (PSELDNets) on large-scale synthetic datasets. These synthetic datasets, generated by convolving sound events with simulated spatial room impulse responses (SRIRs), contain 1,167 hours of audio clips with an ontology of 170 sound classes. These PSELDNets are transferred to downstream SELD tasks. When we adapt PSELDNets to specific scenarios, particularly in low-resource data cases, we introduce a data-efficient fine-tuning method, AdapterBit. PSELDNets are evaluated on a synthetic-test-set using collected SRIRs from TAU Spatial Room Impulse Response Database (TAU-SRIR DB) and achieve satisfactory performance. We also conduct our experiments to validate the transferability of PSELDNets to three publicly available datasets and our own collected audio recordings. Results demonstrate that PSELDNets surpass state-of-the-art systems across all publicly available datasets. Given the need for direction-of-arrival estimation, SELD generally relies on sufficient multi-channel audio clips. However, incorporating the AdapterBit, PSELDNets show more efficient adaptability to various tasks using minimal multi-channel or even just monophonic audio clips, outperforming the traditional fine-tuning approaches.
Direction-of-arrival estimation of multiple speakers in a room is an important task for a wide range of applications. In particular, challenging environments with moving speakers, reverberation and noise, lead to significant performance degradation for current methods. With the aim of better understanding factors affecting performance and improving current methods, in this paper multi-speaker direction-of-arrival (DOA) estimation is investigated using a modified version of the local space domain distance (LSDD) algorithm in a noisy, dynamic and reverberant environment employing a wearable microphone array. This study utilizes the recently published EasyCom speech dataset, recorded using a wearable microphone array mounted on eyeglasses. While the original LSDD algorithm demonstrates strong performance in static environments, its efficacy significantly diminishes in the dynamic settings of the EasyCom dataset. Several enhancements to the LSDD algorithm are developed following a comprehensive performance and system analysis, which enable improved DOA estimation under these challenging conditions. These improvements include incorporating a weighted reliability approach and introducing a new quality measure that reliably identifies the more accurate DOA estimates, thereby enhancing both the robustness and accuracy of the algorithm in challenging environments.
Accurately estimating the direction-of-arrival (DOA) of a speech source using a compact microphone array (CMA) is often complicated by background noise and reverberation. A commonly used DOA estimation method is the steered response power with phase transform (SRP-PHAT) function, which has been shown to work reliably in moderate levels of noise and reverberation. Since for closely spaced microphones the spatial coherence of noise and reverberation may be high over an extended frequency range, this may negatively affect the SRP-PHAT spectra, resulting in DOA estimation errors. Assuming the availability of an auxiliary microphone at an unknown position which is spatially separated from the CMA, in this paper we propose to compute the SRP-PHAT spectra between the microphones of the CMA based on the SRP-PHAT spectra between the auxiliary microphone and the microphones of the CMA. For different levels of noise and reverberation, we show how far the auxiliary microphone needs to be spatially separated from the CMA for the auxiliary microphone-based SRP-PHAT spectra to be more reliable than the SRP-PHAT spectra without the auxiliary microphone. These findings are validated based on simulated microphone signals for several auxiliary microphone positions and two different noise and reverberation conditions.
This paper proposes design techniques for partially-calibrated sparse linear subarrays and algorithms to perform direction-of-arrival (DOA) estimation. First, we introduce array architectures that incorporate two distinct array categories, namely type-I and type-II arrays. The former breaks down a known sparse linear geometry into as many pieces as we need, and the latter employs each subarray such as it fits a preplanned sparse linear geometry. Moreover, we devise two Direction of Arrival (DOA) estimation algorithms that are suitable for partially-calibrated array scenarios within the coarray domain. The algorithms are capable of estimating a greater number of sources than the number of available physical sensors, while maintaining the hardware and computational complexity within practical limits for real-time implementation. To this end, we exploit the intersection of projections onto affine spaces by devising the Generalized Coarray Multiple Signal Classification (GCA-MUSIC) in conjunction with the estimation of a refined projection matrix related to the noise subspace, as proposed in the GCA root-MUSIC algorithm. An analysis is performed for the devised subarray configurations in terms of degrees of freedom, as well as the computation of the Cram\`er-Rao Lower Bound for the utilized data model, in order to demonstrate the good performance of the proposed methods. Simulations assess the performance of the proposed design methods and algorithms against existing approaches.
Direction-of-Arrival (DoA) estimation assisted with an Intelligent Reflecting Surface (IRS) is crucial for various wireless applications, especially in challenging Non-Line-of-Sight (NLoS) environments. This paper presents a novel neural network-based architecture to address this challenge. The key innovation is the introduction of a dedicated, learnable IRS layer integrated integrated within a carefully designed network structure established upon the physical and geometrical basis of the problem. Unlike conventional neural network layers, this specific layer incorporates block diagonal sinusoidal weight constraints, where the phase arguments of these sinusoids are learned during training to directly emulate the phase shifts of the IRS elements. This allows the network to optimize the IRS configuration for enhanced DoA estimation, eliminating the need for separate IRS optimization algorithms. Moreover, various trainable DoA regressors, including a proposed structure, are presented and examined. Numerical simulations, conducted under various conditions and noise levels, where controlled coherent multi-path components are introduced due to the presence of the IRS, demonstrate the superior performance of the proposed approach compared to others and highlight its potential to significantly improve the accuracy of DoA estimation in complex IRS-assisted wireless systems.
One key aspect differentiating data-driven single- and multi-channel speech enhancement and dereverberation methods is that both the problem formulation and complexity of the solutions are considerably more challenging in the latter case. Additionally, with limited computational resources, it is cumbersome to train models that require the management of larger datasets or those with more complex designs. In this scenario, an unverified hypothesis that single-channel methods can be adapted to multi-channel scenarios simply by processing each channel independently holds significant implications, boosting compatibility between sound scene capture and system input-output formats, while also allowing modern research to focus on other challenging aspects, such as full-bandwidth audio enhancement, competitive noise suppression, and unsupervised learning. This study verifies this hypothesis by comparing the enhancement promoted by a basic single-channel speech enhancement and dereverberation model with two other multi-channel models tailored to separate clean speech from noisy 3D mixes. A direction of arrival estimation model was used to objectively evaluate its capacity to preserve spatial information by comparing the output signals with ground-truth coordinate values. Consequently, a trade-off arises between preserving spatial information with a more straightforward single-channel solution at the cost of obtaining lower gains in intelligibility scores.
Despite many advantages of direction-of-arrivals (DOAs) in sparse representation domain, they have high computational complexity. This paper presents a new method for real-valued 2-D DOAs estimation of sources in a uniform circular array configuration. This method uses a transformation based on phase mode excitation in uniform circular arrays which called real beamspace L1-SVD (RB-L1SVD). This unitary transformation converts complex manifold matrix to real one, so that the computational complexity is decreased with respect to complex valued computations,its computation, at least, is one,fourth of the complex-valued case; moreover, some benefits from using this transformation are robustness to array imperfections, a better noise suppression because of exploiting an additional real structure, and etc. Numerical results demonstrate the better performance of the proposed approach over previous techniques such as C-L1SVD, RB-ESPRIT, and RB-MUSIC, especially in low signal-to-noise ratios.
Covariance fitting (CF) is a comprehensive approach for direction of arrival (DoA) estimation, consolidating many common solutions. Standard practice is to use Euclidean criteria for CF, disregarding the intrinsic Hermitian positive-definite (HPD) geometry of the spatial covariance matrices. We assert that this oversight leads to inherent limitations. In this paper, as a remedy, we present a comprehensive study of the use of various Riemannian metrics of HPD matrices in CF. We focus on the advantages of the Affine-Invariant (AI) and the Log-Euclidean (LE) Riemannian metrics. Consequently, we propose a new practical beamformer based on the LE metric and derive analytically its spatial characteristics, such as the beamwidth and sidelobe attenuation, under noisy conditions. Comparing these features to classical beamformers shows significant advantage. In addition, we demonstrate, both theoretically and experimentally, the LE beamformer's robustness in scenarios with small sample sizes and in the presence of noise, interference, and multipath channels.