Single image super-resolution (SISR) is an image processing task which obtains high-resolution (HR) image from a low-resolution (LR) image. Recently, due to the capability in feature extraction, a series of deep learning methods have brought important crucial improvement for SISR. However, we observe that no matter how deeper the networks are designed, they usually do not have good generalization ability, which leads to the fact that almost all of existing SR methods have poor performances on restoration of the weak texture details. To solve these problems, we propose a weak texture information map guided image super-resolution with deep residual networks. It contains three sub-networks, one main network which extracts the main features and fuses weak texture details, another two auxiliary networks extract the weak texture details fallen in the main network. Two part of networks work cooperatively, the auxiliary networks predict and integrates week texture information into the main network, which is conducive to the main network learning more inconspicuous details. Experiments results demonstrate that our method's performs achieve the state-of-the-art quantitatively. Specifically, the image super-resolution results of our method own more weak texture details.
Integrating deep learning techniques into the video coding framework gains significant improvement compared to the standard compression techniques, especially, applying super-resolution (up-sampling) in down-sampling based video coding as post-processing. However, different from the conventional super-resolution works, the output can be distorted by the various compression artifacts in the decoded frames. The straightforward solution is to integrate the artifacts removing techniques before super-resolution. But some helpful features may be removed together with the artifacts, which will degrade the performance of super-resolution. To address this problem, we proposed a restoration-reconstruction deep neural network (RR-DnCNN) using the degradation-aware techniques. Moreover, to prevent the loss of essential features in the very deep network from restoration to super-resolution, we leverage up-sampling skip connections to compensate for the lost information from restoration layers. It is called restoration-reconstruction u-shaped deep neural network (RR-DnCNN v2.0). As a result, our RR-DnCNN v2.0 can achieve 17.02% BD-rate reduction on UHD resolution compared to the standard H.265/HEVC. The source code is available at https://minhmanho.github.io/rrdncnn/.
The quality of the reconstructed photoacoustic image largely depends on the amount of photoacoustic (PA) boundary data available, which in turn is proportional to the number of detectors employed. In case of limited data (owing to less number of detectors due to cost/instrumentation constraints), the reconstructed PA images suffer from artifacts and are often noisy. In this work, for the first time, a deep learning based model was developed to super resolve and denoise the photoacoustic sinogram data. The proposed method was compared with existing nearest neighbor interpolation and wavelet based denoising techniques and was shown to outperform them both in numerical and in-vivo cases. The improvement obtained in Root Mean Square Error (RMSE) and Peak Signal to Noise Ratio (PSNR) for the reconstructed PA image using the sinogram data that was super-resolved and denoised using proposed neural network based method was as high as 41.70 % and 6.93 dB respectively compared to utilizing limited sinogram data.
Given an image generated by the convolution of point sources with a band-limited function, the deconvolution problem is to reconstruct the source number, positions, and amplitudes. Rayleigh investigated the limit of this problem and formulated the Rayleigh limit, for the case of two sources with identical amplitudes. On the other hand, many numerical experiments demonstrate that a stable recovery of the sources is possible even if the sources are separated below the Rayleigh limit. This is so-called "super-resolution". In this paper, a new mathematical theory for super-resolution will be developed. The theory will address the issue when one can recover the source number exactly. The key is a new concept "computational resolution limit" which is defined to be the minimum separation distance between the sources such that exact recovery of the source number is possible. This new resolution limit is determined by the signal-to-noise ratio and the sparsity of sources, in addition to the cutoff frequency of the image. Sharp upper bound for this limit is derived, which reveals the importance of the sparsity as well as the signal-to-noise ratio to the recovery problem. The stability for recovering the source positions is further derived when the separation distance is beyond the upper bound. Moreover, a MUSIC-type algorithm is proposed to recover the source number and to verify our theoretical results on the computational resolution limit. Its performance in the super-resolution regime when the sources are separated below the Rayleigh limit is analyzed both theoretically and numerically. The results are based on a multipole expansion method and a novel non-linear approximation theory in Vandermonde space.
Open spina bifida (SB) is one of the most common congenital defects and can lead to impaired brain development. Emerging fetal surgery methods have shown considerable success in the treatment of patients with this severe anomaly. Afterwards, alterations in the brain development of these fetuses have been observed. Currently no longitudinal studies exist to show the effect of fetal surgery on brain development. In this work, we present a fetal MRI neuro-imaging analysis pipeline for fetuses with SB, including automated fetal ventricle segmentation and deformation-based morphometry, and demonstrate its applicability with an analysis of ventricle enlargement in fetuses with SB. Using a robust super-resolution algorithm, we reconstructed fetal brains at both pre-operative and post-operative time points and trained a U-Net CNN in order to automatically segment the ventricles. We investigated the change of ventricle shape post-operatively, and the impacts of lesion size, type, and GA at operation on the change in ventricle shape. No impact was found, except for moderately larger ventriculomegaly progression in myeloschisis patients. Prenatal ventricle volume growth was also investigated. Our method allows for the quantification of longitudinal morphological changes to fully quantify the impact of prenatal SB repair and could be applied to predict postnatal outcomes.
Ghost imaging is demonstrated using a poly-energetic reactor source of thermal neutrons. The method presented enables position resolution to be incorporated, into a variety of neutron instruments that are not position resolving. In an imaging context, ghost imaging can be beneficial for dose reduction and resolution enhancement. We also demonstrate a super-resolution variant of the method, namely a parallel form of neutron ghost imaging, with the ability to significantly increase the spatial resolution of a pixelated detector such as a CCD or CMOS camera. Extensions of our neutron ghost-imaging protocol are discussed in detail and include neutron ghost tomography, neutron ghost microscopy, dark-field neutron ghost imaging, and isotope-resolved color neutron ghost imaging via prompt gamma-ray bucket detection.
This paper focuses on the problem of localising a transmitting mobile station (MS) using multiple cooperative base stations (BSs). There are two technical difficulties: one is the data association between intermediate parameters and scatters; the other is to identify Line-of-Sight (LoS) and Non-Line-of-Sight (NLoS) paths. Our main contribution is a unified approach bypassing both data association and LoS/NLoS path identification. This is achieved by introducing virtual scatters and then a direct localisation formulation. Modern super-resolution technique is then adapted to cast the localisation problem as convex programming and solve it. Our approach addresses long-standing issues not completely solved in the literature, guarantees a global convergence, achieves good localisation accuracy and demonstrates its robustness to noise in our numerical tests.
Hyperspectral super-resolution (HSR) aims at fusing a hyperspectral image (HSI) and a multispectral image (MSI) to produce a super-resolution image (SRI). Recently, a coupled tensor factorization approach was proposed to handle this challenging problem, which admits a series of advantages over the classic matrix factorization-based methods. In particular, modeling the HSI and MSI as low-rank tensors following the {\it canonical polyadic decomposition} (CPD) model, the approach is able to provably identify the SRI, under some mild conditions. However, the latent factors in the CPD model have no physical meaning, which makes utilizing prior information of spectral images as constraints or regularizations difficult---but using such information is often important in practice, especially when the data is noisy. In this work, we propose an alternative coupled tensor decomposition approach, where the HSI and MSI are assumed to follow the {\it block-term decomposition (BTD)} model. Notably, the new method also entails identifiability of the SRI under realistic conditions. More importantly, when modeling a spectral image as a BTD tensor, the latent factors have clear physical meaning, and thus prior knowledge about spectral images can be naturally incorporated. Simulations using real hyperspectral images are employed to showcase the effectiveness of the proposed approach with nonnegativity constraints.
Incoherent Fourier ptychography (IFP) is a newly developed super-resolution method, where accurate knowledge of translation positions is essential for image reconstruction.To release this limitation, we propose a preprocessing algorithm capable of extracting translation positions of the structure light directly from raw images of IFP, termed translation position extracting (TPE). TPE mainly involves two steps. First, the speckle parts mixed in the acquired intensities, in which the illumination motion is encoded, are isolated by intensity averaging and division. Then the cross-correlations of the speckle dataset are computed to determine the shift positions. TPE-IFP improves the previous IFP by removal of the requirement for prior knowledge of translation positions. Its effectiveness is demonstrated by obtaining high-quality super-resolution images in absence of location information in both simulations and experiments. By further relaxing the practical conditions, the proposed TPE may accelerate the applications of IFP. What is more, as a preprocessing approach, TPE might also contribute to the estimation of pattern positions for the similar speckle-based imaging.
It has previously been demonstrated that model-based reconstruction methods relying on a priori knowledge of the imaging point spread function (PSF) coupled to sparsity priors on the object to image can provide super-resolution in photoacoustic (PA) or in ultrasound (US) imaging. Here, we experimentally show that such reconstruction also leads to super-resolution in both PA and US imaging with arrays having much less elements than used conventionally (sparse arrays). As a proof of concept, we obtained super-resolution PA and US cross-sectional images of microfluidic channels with only 8 elements of a 128-elements linear array using a reconstruction approach based on a linear propagation forward model and assuming sparsity of the imaged structure. Although the microchannels appear indistinguishable in the conventional delay-and-sum images obtained with all the 128 transducer elements, the applied sparsity-constrained model-based reconstruction provides super-resolution with down to only 8 elements. We also report simulation results showing that the minimal number of transducer elements required to obtain a correct reconstruction is fundamentally limited by the signal-to-noise ratio. The proposed method can be straigthforwardly applied to any transducer geometry, including 2D sparse arrays for 3D super-resolution PA and US imaging.
This paper develops a first-order optimization method for coupled structured matrix factorization (CoSMF) problems that arise in the context of hyperspectral super-resolution (HSR) in remote sensing. To best leverage the problem structures for computational efficiency, we introduce a hybrid inexact block coordinate descent (HiBCD) scheme wherein one coordinate is updated via the fast proximal gradient (FPG) method, while another via the Frank-Wolfe (FW) method. The FPG-type methods are known to take less number of iterations to converge, by numerical experience, while the FW-type methods can offer lower per-iteration complexity in certain cases; and we wish to take the best of both. We show that the limit points of this HiBCD scheme are stationary. Our proof treats HiBCD as an optimization framework for a class of multi-block structured optimization problems, and our stationarity claim is applicable not only to CoSMF but also to many other problems. Previous optimization research showed the same stationarity result for inexact block coordinate descent with either FPG or FW updates only. Numerical results indicate that the proposed HiBCD scheme is computationally much more efficient than the state-of-the-art CoSMF schemes in HSR.
In this paper, we introduce a novel algorithm that can dramatically reduce the number of antenna elements needed to accurately predict the direction of arrival (DOA) for multiple input multiple output (MIMO) radar. The new proposed algorithm predicts the received signal of a large antenna setup using reduced number of antenna by using coupled dictionary learning. Hence, this enables the MIMO radar to resolve more paths, which could not be resolved by the fewer antennas. Specifically, we overcome the problem of inaccurate DOA estimation due to a small virtual array setup. For example, we can use dictionary learning to predict 100 virtual array elements using only 25. To evaluate our algorithm, we used multiple signal classification (MUSIC) as a DOA estimation technique to estimate the DOA for non coherent multiple targets. The results show that using the predicted received signal, the proposed algorithm could resolve all the targets in the scene, which could not been resolved using only the received signal from the reduced antenna setup.
Coupled structured matrix factorization (CoSMF) for hyperspectral super-resolution (HSR) has recently drawn significant interest in hyperspectral imaging for remote sensing. Presently there is very few work that studies the theoretical recovery guarantees of CoSMF. This paper makes one such endeavor by considering the CoSMF formulation by Wei et al., which, simply speaking, is similar to coupled non-negative matrix factorization. Assuming no noise, we show sufficient conditions under which the globably optimal solution to the CoSMF problem is guaranteed to deliver certain recovery accuracies. Our analysis suggests that sparsity and the pure-pixel (or separability) condition play a hidden role in enabling CoSMF to achieve some good recovery characteristics.
This paper develops efficient channel estimation techniques for millimeter-wave (mmWave) massive multiple-input multiple-output (MIMO) systems under practical hardware limitations, including an arbitrary array geometry and a hybrid hardware structure. Taking on an angle-based approach, this work adopts a generalized array manifold separation approach via Jacobi-Anger approximation, which transforms a non-ideal, non-uniform array manifold into a virtual array domain with a desired uniform geometric structure to facilitate super-resolution angle estimation and channel acquisition. Accordingly, structure-based optimization techniques are developed to effectively estimate both the channel covariance and the instantaneous channel state information (CSI) within a short sensing time. In particular, the difference in time-variation of channel path angles and path gains is capitalized to design a two-step CSI estimation scheme that can quickly sense fading channels. Theoretical results are provided on the fundamental limits of the proposed technique in terms of sample efficiency. For computational efficiency, a fast iterative algorithm is developed via the alternating direction method of multipliers. Other related issues such as spurious-peak cancellation in nonuniform linear arrays and extensions to higher-dimensional cases are also discussed. Simulations testify the effectiveness of the proposed approaches in hybrid mmWave massive MIMO systems with arbitrary arrays.
Convex optimization recently emerges as a compelling framework for performing super resolution, garnering significant attention from multiple communities spanning signal processing, applied mathematics, and optimization. This article offers a friendly exposition to atomic norm minimization as a canonical convex approach to solve super resolution problems. The mathematical foundations and performances guarantees of this approach are presented, and its application in super resolution image reconstruction for single-molecule fluorescence microscopy are highlighted.
High-content biological microscopy targets high-resolution imaging across large fields-of-view (FOVs). Recent works have demonstrated that computational imaging can provide efficient solutions for high-content microscopy. Here, we use speckle structured illumination microscopy (SIM) as a robust and cost-effective solution for high-content fluorescence microscopy with simultaneous high-content quantitative phase (QP). This multi-modal compatibility is essential for studies requiring cross-correlative biological analysis. Our method uses laterally-translated Scotch tape to generate high-resolution speckle illumination patterns across a large FOV. Custom optimization algorithms then jointly reconstruct the sample's super-resolution fluorescent (incoherent) and QP (coherent) distributions, while digitally correcting for system imperfections such as unknown speckle illumination patterns, system aberrations and pattern translations. Beyond previous linear SIM works, we achieve resolution gains of 4x the objective's diffraction-limited native resolution, resulting in 700 nm fluorescence and 1.2 um QP resolution, across a FOV of 2x2.7 mm^2, giving a space-bandwidth product (SBP) of 60 megapixels.
We address the problem of search-free direction of arrival (DOA) estimation for sensor arrays of arbitrary geometry under the challenging conditions of a single snapshot and coherent sources. We extend a method of searchfree super-resolution beamforming, originally applicable only for uniform linear arrays, to arrays of arbitrary geometry. The infinite dimensional primal atomic norm minimization problem in continuous angle domain is converted to a dual problem. By exploiting periodicity, the dual function is then represented with a trigonometric polynomial using a truncated Fourier series. A linear rule of thumb is derived for selecting the minimum number of Fourier coefficients required for accurate polynomial representation, based on the distance of the farthest sensor from a reference point. The dual problem is then expressed as a semidefinite program and solved efficiently. Finally, the searchfree DOA estimates are obtained through polynomial rooting, and source amplitudes are recovered through least squares. Simulations using circular and random planar arrays show perfect DOA estimation in noise-free cases.
This paper presents an efficient optimization technique for gridless {2-D} line spectrum estimation, named decoupled atomic norm minimization (D-ANM). The framework of atomic norm minimization (ANM) is considered, which has been successfully applied in 1-D problems to allow super-resolution frequency estimation for correlated sources even when the number of snapshots is highly limited. The state-of-the-art 2-D ANM approach vectorizes the 2-D measurements to their 1-D equivalence, which incurs huge computational cost and may become too costly for practical applications. We develop a novel decoupled approach of 2-D ANM via semi-definite programming (SDP), which introduces a new matrix-form atom set to naturally decouple the joint observations in both dimensions without loss of optimality. Accordingly, the original large-scale 2-D problem is equivalently reformulated via two decoupled one-level Toeplitz matrices, which can be solved by simple 1-D frequency estimation with pairing. Compared with the conventional vectorized approach, the proposed D-ANM technique reduces the computational complexity by several orders of magnitude with respect to the problem size. It also retains the benefits of ANM in terms of precise signal recovery, small number of required measurements, and robustness to source correlation. The complexity benefits are particularly attractive for large-scale antenna systems such as massive MIMO, radar signal processing and radio astronomy.
In example-based super-resolution, the function relating low-resolution images to their high-resolution counterparts is learned from a given dataset. This data-driven approach to solving the inverse problem of increasing image resolution has been implemented with deep learning algorithms. In this work, we explore modifying the imaging hardware in order to collect more informative low-resolution images for better ultimate high-resolution image reconstruction. We show that this "physical preprocessing" allows for improved image reconstruction with deep learning in Fourier ptychographic microscopy. Fourier ptychographic microscopy is a technique allowing for both high resolution and high field-of-view at the cost of temporal resolution. In Fourier ptychographic microscopy, variable illumination patterns are used to collect multiple low-resolution images. These low-resolution images are then computationally combined to create an image with resolution exceeding that of any single image from the microscope. We use deep learning to jointly optimize the illumination pattern with the post-processing reconstruction algorithm for a given sample type, allowing for single-shot imaging with both high resolution and high field-of-view. We demonstrate, with simulated data, that the joint optimization yields improved image reconstruction as compared with sole optimization of the post-processing reconstruction algorithm.
We propose a mixed-resolution point-cloud representation and an example-based super-resolution framework, from which several processing tools can be derived, such as compression, denoising and error concealment. By inferring the high-frequency content of low-resolution frames based on the similarities between adjacent full-resolution frames, the proposed framework achieves an average 1.18 dB gain over low-pass versions of the point-cloud, for a projection-based distortion metric[1-2].