The effectiveness of spectral-spatial feature learning is crucial for the hyperspectral image (HSI) classification task. Diffusion models, as a new class of groundbreaking generative models, have the ability to learn both contextual semantics and textual details from the distinct timestep dimension, enabling the modeling of complex spectral-spatial relations in HSIs. However, existing diffusion-based HSI classification methods only utilize manually selected single-timestep single-stage features, limiting the full exploration and exploitation of rich contextual semantics and textual information hidden in the diffusion model. To address this issue, we propose a novel diffusion-based feature learning framework that explores Multi-Timestep Multi-Stage Diffusion features for HSI classification for the first time, called MTMSD. Specifically, the diffusion model is first pretrained with unlabeled HSI patches to mine the connotation of unlabeled data, and then is used to extract the multi-timestep multi-stage diffusion features. To effectively and efficiently leverage multi-timestep multi-stage features,two strategies are further developed. One strategy is class & timestep-oriented multi-stage feature purification module with the inter-class and inter-timestep prior for reducing the redundancy of multi-stage features and alleviating memory constraints. The other one is selective timestep feature fusion module with the guidance of global features to adaptively select different timestep features for integrating texture and semantics. Both strategies facilitate the generality and adaptability of the MTMSD framework for diverse patterns of different HSI data. Extensive experiments are conducted on four public HSI datasets, and the results demonstrate that our method outperforms state-of-the-art methods for HSI classification, especially on the challenging Houston 2018 dataset.
Deep learning has achieved impressive results on hyperspectral image (HSI) classification. Among them, both convolutional neural networks (CNNs) and graph neural networks (GNNs) have great potential for HSI classification. Supervised CNNs can efficiently extract hierarchical spatial–spectral features of HSIs, but these methods face the problem of high time complexity as the number of network layers increases. Semisupervised GNNs can rapidly capture the structural information of HSIs, while they cannot be well extended to HSI applications because of the process of adjacency matrix consuming large amount of memory resources. In this article, we propose a fast dynamic graph convolutional network and CNN (FDGC) parallel network for HSI classification. We first obtain two classification features by flattening and pooling operations on the results of the convolution layers, which fully exploits the spatial–spectral information contained in the hyperspectral data cube. Then, a dynamic graph convolution module is applied to extract the intrinsic structural information of each patch. Finally, we can obtain the HSI classification results based on these spatial, spectral, and structural features. By using three branches, FDGC can parallelly process multiple features of HSI in a supervised learning manner. In addition, regularization techniques, such as DropBlock and label smoothing, are applied to further improve the generalization capability of the model. Experimental results on three datasets show that our proposed algorithm is comparable with the state-of-the-art supervised learning models in terms of accuracy while also significantly outperforming in terms of training and inference time.
Insufficient prior knowledge of a captured hyperspectral image (HSI) scene may lead the experts or the automatic labeling systems to offer incorrect labels or ambiguous labels (i.e., assigning each training sample to a group of candidate labels, among which only one of them is valid; this is also known as partial label learning) during the labeling process. Accordingly, how to learn from such data with ambiguous labels is a problem of great practical importance. In this letter, we propose a novel superpixelwise low-rank approximation (LRA)-based partial label learning method, namely SLAP, which is the first to take into account partial label learning in HSI classification. SLAP is mainly composed of two phases: disambiguating the training labels and acquiring the predictive model. Specifically, in the first phase, we propose a superpixelwise LRA-based model, preparing the affinity graph for the subsequent label propagation process while extracting the discriminative representation to enhance the following classification task of the second phase. Then to disambiguate the training labels, label propagation propagates the labeling information via the affinity graph of training pixels. In the second phase, we take advantage of the resulting disambiguated training labels and the discriminative representations to enhance the classification performance. The extensive experiments validate the advantage of the proposed SLAP method over state-of-the-art methods.
Hyperspectral Image Classification (HSIC) is a difficult task due to high inter and intra-class similarity and variability, nested regions, and overlapping. 2D Convolutional Neural Networks (CNN) emerged as a viable network whereas, 3D CNNs are a better alternative due to accurate classification. However, 3D CNNs are highly computationally complex due to their volume and spectral dimensions. Moreover, down-sampling and hierarchical filtering (high frequency) i.e., texture features need to be smoothed during the forward pass which is crucial for accurate HSIC. Furthermore, CNN requires tons of tuning parameters which increases the training time. Therefore, to overcome the aforesaid issues, Sharpened Cosine Similarity (SCS) concept as an alternative to convolutions in a Neural Network for HSIC is introduced. SCS is exceptionally parameter efficient due to skipping the non-linear activation layers, normalization, and dropout after the SCS layer. Use of MaxAbsPool instead of MaxPool which selects the element with the highest magnitude of activity, even if it's negative. Experimental results on publicly available HSI datasets proved the performance of SCS as compared to the convolutions in Neural Networks.
Most domain adaptation (DA) methods in cross-scene hyperspectral image classification focus on cases where source data (SD) and target data (TD) with the same classes are obtained by the same sensor. However, the classification performance is significantly reduced when there are new classes in TD. In addition, domain alignment, as one of the main approaches in DA, is carried out based on local spatial information, rarely taking into account nonlocal spatial information (nonlocal relationships) with strong correspondence. A graph information aggregation cross-domain few-shot learning (Gia-CFSL) framework is proposed, intending to make up for the above-mentioned shortcomings by combining FSL with domain alignment based on graph information aggregation. SD with all label samples and TD with a few label samples are implemented for FSL episodic training. Meanwhile, intradomain distribution extraction block (IDE-block) and cross-domain similarity aware block (CSA-block) are designed. The IDE-block is used to characterize and aggregate the intradomain nonlocal relationships and the interdomain feature and distribution similarities are captured in the CSA-block. Furthermore, feature-level and distribution-level cross-domain graph alignments are used to mitigate the impact of domain shift on FSL. Experimental results on three public HSI datasets demonstrate the superiority of the proposed method. The codes will be available from the website: https://github.com/YuxiangZhang-BIT/IEEE_TNNLS_Gia-CFSL.
Learning from a limited number of labeled samples (pixels) remains a key challenge in the hyperspectral image (HSI) classification. To address this issue, we propose a deep metric learning-based feature embedding model, which can meet the tasks both for same- and cross-scene HSI classifications. In the first task, when only a few labeled samples are available, we employ ideas from metric learning based on deep embedding features and make a similarity learning between pairs of samples. In this case, the proposed model can learn well to compare whether two samples belong to the same class. In another task, when an HSI image (target scene) that needs to be classified is not labeled at all, the embedding model can learn from another similar HSI image (source scene) with sufficient labeled samples and then transfer to the target model by using an unsupervised domain adaptation technique, which not only employs the adversarial approach to make the embedding features from the source and target samples indistinguishable but also encourages the target scene's embeddings to form similar clusters with the source scene one. After the domain adaptation between the HSIs of the two scenes is finished, any traditional HSI classifier can be used. In a simple manner, the nearest neighbor (NN) algorithm is selected as the classifier for the classification tasks throughout this article. The experimental results from a series of popular HSIs demonstrate the advantages of the proposed model both in the same- and cross-scene classification tasks.
Deep learning methods have been widely used in hyperspectral image classification and have achieved state-of-the-art performance. Nonetheless, the existing deep learning methods are restricted by a limited receptive field, inflexibility, and difficult generalization problems in hyperspectral image classification. To solve these problems, we propose HSI-BERT, where BERT stands for bidirectional encoder representations from transformers and HSI stands for hyperspectral imagery. The proposed HSI-BERT has a global receptive field that captures the global dependence among pixels regardless of their spatial distance. HSI-BERT is very flexible and enables the flexible and dynamic input regions. Furthermore, HSI-BERT has good generalization ability because the jointly trained HSI-BERT can be generalized from regions with different shapes without retraining. HSI-BERT is primarily built on a multihead self-attention (MHSA) mechanism in an MHSA layer. Moreover, several attentions are learned by different heads, and each head of the MHSA layer encodes the semantic context-aware representation to obtain discriminative features. Because all head-encoded features are merged, the resulting features exhibit spatial-spectral information that is essential for accurate pixel-level classification. Quantitative and qualitative results demonstrate that HSI-BERT outperforms any other CNN-based model in terms of both classification accuracy and computational time and achieves state-of-the-art performance on three widely used hyperspectral image data sets.
Spectral observations along the spectrum in many narrow spectral bands through hyperspectral imaging provides valuable information towards material and object recognition, which can be consider as a classification task. Most of the existing studies and research efforts are following the conventional pattern recognition paradigm, which is based on the construction of complex handcrafted features. However, it is rarely known which features are important for the problem at hand. In contrast to these approaches, we propose a deep learning based classification method that hierarchically constructs high-level features in an automated way. Our method exploits a Convolutional Neural Network to encode pixels' spectral and spatial information and a Multi-Layer Perceptron to conduct the classification task. Experimental results and quantitative validation on widely used datasets showcasing the potential of the developed approach for accurate hyperspectral data classification.
Feature extraction is known to be an effective way in both reducing computational complexity and increasing accuracy of hyperspectral image classification. In this paper, a simple yet quite powerful feature extraction method based on image fusion and recursive filtering (IFRF) is proposed. First, the hyperspectral image is partitioned into multiple subsets of adjacent hyperspectral bands. Then, the bands in each subset are fused together by averaging, which is one of the simplest image fusion methods. Finally, the fused bands are processed with transform domain recursive filtering to get the resulting features for classification. Experiments are performed on different hyperspectral images, with the support vector machines (SVMs) serving as the classifier. By using the proposed method, the accuracy of the SVM classifier can be improved significantly. Furthermore, compared with other hyperspectral classification methods, the proposed IFRF method shows outstanding performance in terms of classification accuracy and computational efficiency.
Hyperspectral images consist of large number of bands which require sophisticated analysis to extract. One approach to reduce computational cost, information representation, and accelerate knowledge discovery is to eliminate bands that do not add value to the classification and analysis method which is being applied. In particular, algorithms that perform band elimination should be designed to take advantage of the structure of the classification method used. This letter introduces an embedded-feature-selection (EFS) algorithm that is tailored to operate with support vector machines (SVMs) to perform band selection and classification simultaneously. We have successfully applied this algorithm to determine a reasonable subset of bands without any user-defined stopping criteria on some sample AVIRIS images; a problem occurs in benchmarking recursive-feature-elimination methods for the SVMs.
Hyperspectral image (HSI) classification is challenging due to spatial variability caused by complex imaging conditions. Prior methods suffer from limited representation ability, as they train specially designed networks from scratch on limited annotated data. We propose a tri-spectral image generation pipeline that transforms HSI into high-quality tri-spectral images, enabling the use of off-the-shelf ImageNet pretrained backbone networks for feature extraction. Motivated by the observation that there are many homogeneous areas with distinguished semantic and geometric properties in HSIs, which can be used to extract useful contexts, we propose an end-to-end segmentation network named DCN-T. It adopts transformers to effectively encode regional adaptation and global aggregation spatial contexts within and between the homogeneous areas discovered by similarity-based clustering. To fully exploit the rich spectrums of the HSI, we adopt an ensemble approach where all segmentation results of the tri-spectral images are integrated into the final prediction through a voting scheme. Extensive experiments on three public benchmarks show that our proposed method outperforms state-of-the-art methods for HSI classification.
Hyperspectral Image (HSI) classification is an important issue in remote sensing field with extensive applications in earth science. In recent years, a large number of deep learning-based HSI classification methods have been proposed. However, existing methods have limited ability to handle high-dimensional, highly redundant, and complex data, making it challenging to capture the spectral-spatial distributions of data and relationships between samples. To address this issue, we propose a generative framework for HSI classification with diffusion models (SpectralDiff) that effectively mines the distribution information of high-dimensional and highly redundant data by iteratively denoising and explicitly constructing the data generation process, thus better reflecting the relationships between samples. The framework consists of a spectral-spatial diffusion module, and an attention-based classification module. The spectral-spatial diffusion module adopts forward and reverse spectral-spatial diffusion processes to achieve adaptive construction of sample relationships without requiring prior knowledge of graphical structure or neighborhood information. It captures spectral-spatial distribution and contextual information of objects in HSI and mines unsupervised spectral-spatial diffusion features within the reverse diffusion process. Finally, these features are fed into the attention-based classification module for per-pixel classification. The diffusion features can facilitate cross-sample perception via reconstruction distribution, leading to improved classification performance. Experiments on three public HSI datasets demonstrate that the proposed method can achieve better performance than state-of-the-art methods. For the sake of reproducibility, the source code of SpectralDiff will be publicly available at https://github.com/chenning0115/SpectralDiff.
For the abundant spectral and spatial information recorded in hyperspectral images (HSIs), fully exploring spectral-spatial relationships has attracted widespread attention in hyperspectral image classification (HSIC) community. However, there are still some intractable obstructs. For one thing, in the patch-based processing pattern, some spatial neighbor pixels are often inconsistent with the central pixel in land-cover class. For another thing, linear and nonlinear correlations between different spectral bands are vital yet tough for representing and excavating. To overcome these mentioned issues, an adaptive mask sampling and manifold to Euclidean subspace learning (AMS-M2ESL) framework is proposed for HSIC. Specifically, an adaptive mask based intra-patch sampling (AMIPS) module is firstly formulated for intra-patch sampling in an adaptive mask manner based on central spectral vector oriented spatial relationships. Subsequently, based on distance covariance descriptor, a dual channel distance covariance representation (DC-DCR) module is proposed for modeling unified spectral-spatial feature representations and exploring spectral-spatial relationships, especially linear and nonlinear interdependence in spectral domain. Furthermore, considering that distance covariance matrix lies on the symmetric positive definite (SPD) manifold, we implement a manifold to Euclidean subspace learning (M2ESL) module respecting Riemannian geometry of SPD manifold for high-level spectral-spatial feature learning. Additionally, we introduce an approximate matrix square-root (ASQRT) layer for efficient Euclidean subspace projection. Extensive experimental results on three popular HSI data sets with limited training samples demonstrate the superior performance of the proposed method compared with other state-of-the-art methods. The source code is available at https://github.com/lms-07/AMS-M2ESL.
Existing multiple modality fusion methods, such as concatenation, summation, and encoder-decoder-based fusion, have recently been employed to combine modality characteristics of Hyperspectral Image (HSI) and Light Detection And Ranging (LiDAR). However, these methods consider the relationship of HSI-LiDAR signals from limited perspectives. More specifically, they overlook the contextual information across modalities of HSI and LiDAR and the intra-modality characteristics of LiDAR. In this paper, we provide a new insight into feature fusion to explore the relationships across HSI and LiDAR modalities comprehensively. An Interconnected Fusion (IF) framework is proposed. Firstly, the center patch of the HSI input is extracted and replicated to the size of the HSI input. Then, nine different perspectives in the fusion matrix are generated by calculating self-attention and cross-attention among the replicated center patch, HSI input, and corresponding LiDAR input. In this way, the intra- and inter-modality characteristics can be fully exploited, and contextual information is considered in both intra-modality and inter-modality manner. These nine interrelated elements in the fusion matrix can complement each other and eliminate biases, which can generate a multi-modality representation for classification accurately. Extensive experiments have been conducted on three widely used datasets: Trento, MUUFL, and Houston. The IF framework achieves state-of-the-art results on these datasets compared to existing approaches.
Existing deep learning-based hyperspectral image (HSI) classification works still suffer from the limitation of the fixed-sized receptive field, leading to difficulties in distinctive spectral-spatial features for ground objects with various sizes and arbitrary shapes. Meanwhile, plenty of previous works ignore asymmetric spectral-spatial dimensions in HSI. To address the above issues, we propose a multi-stage search architecture in order to overcome asymmetric spectral-spatial dimensions and capture significant features. First, the asymmetric pooling on the spectral-spatial dimension maximally retains the essential features of HSI. Then, the 3D convolution with a selectable range of receptive fields overcomes the constraints of fixed-sized convolution kernels. Finally, we extend these two searchable operations to different layers of each stage to build the final architecture. Extensive experiments are conducted on two challenging HSI benchmarks including Indian Pines and Houston University, and results demonstrate the effectiveness of the proposed method with superior performance compared with the related works.
Deep learning methods have been successfully applied to hyperspectral image (HSI) classification with remarkable performance. Because of limited labelled HSI data, earlier studies primarily adopted a patch-based classification framework, which divides images into overlapping patches for training and testing. However, this approach results in redundant computations and possible information leakage. In this study, we propose an objective evaluation-based high-efficiency learning framework for tiny HSI classification. This framework comprises two main parts: (i) a leakage-free balanced sampling strategy, and (ii) a modified end-to-end fully convolutional network (FCN) architecture that optimizes the trade-off between accuracy and efficiency. The leakage-free balanced sampling strategy generates balanced and non-overlapping training and testing data by partitioning an HSI and the ground truth image into small windows, each of which corresponds to one training or testing sample. The proposed high-efficiency FCN exhibits a pixel-to-pixel architecture with modifications aimed at faster inference speed and improved parameter efficiency. Experiments conducted on four representative datasets demonstrated that the proposed sampling strategy can provide objective performance evaluation and that the proposed network outperformed many state-of-the-art approaches with respect to the speed/accuracy tradeoff. Our source code is available at https://github.com/xmzhang2018.
Hyperspectral image (HSI) classification is used in multiple domains like precision agriculture, mineral exploration, remote sensing, and others. Conventionally, couvolutional neural networks (CNNs) were used in HSI classification, however they have limitations in exploiting spectral-spatial relationships, which is a key factor in understanding HSI. Even though deeper CNN architectures and use of 3-D-CNNs mitigate the above problem to a certain extent, they have increased computational complexity, which inhibits their use in resource-limited devices like IoT and edge computing devices. In this paper, we propose a novel method based on the concept of matrix capsules with Expectation-Maximization (EM) routing algorithm which is specifically designed to accommodate the nuances in the HSI data to efficiently tackle the aforementioned problems. The capsule units enable effective identification of spectral siguatures and part-whole relationships in the data while EM routing ensures viewpoint-invariance. Three representative HSI data sets are used to verify the effectiveness of the proposed method. The empirical results demonstrate that the proposed method is better than the current state-of-the-art methods in terms of accuracy while having 25 times fewer model parameters and requiring over 65 times less storage space. The source code can be found at https://github.com/DeepMatrixCapsules/DeepMatrixCapsules.
Deep learning (DL) has shown great potential for hyperspectral image (HSI) classification due to its powerful ability of nonlinear modeling and end-to-end optimization. However, DL models easily get trapped into overfitting due to limited training labels since the labeling process is time-consuming and laborious in a real classification scenario. To overcome this issue, we propose a novel spectral–spatial Siamese network (S3Net) for few-shot HSI classification. First, a lightweight spectral–spatial network (SSN) composed of 1-D and 2-D convolution is proposed to extract spectral–spatial features. Second, S3Net is constructed by two SSNs in dual branches, which can augment the training set by feeding sample pairs into each branch and thus enhancing the model separability. To provide more features for the model, differentiated patches are fed into each branch, where negative samples are randomly selected to avoid redundancy. Finally, a weighted contrastive loss is designed to promote the model to fit in the right direction by focusing on sample pairs that are hardly to be identified. Moreover, another adaptive cross-entropy loss is conceived to learn the fusion ratio of the two branches. Experiments based on three commonly used HSI datasets demonstrate that S3Net outperforms traditional and state-of-the-art DL-based HSI classification methods under a few-shot training scenario. In addition, the weighted contrastive loss and the adaptive cross-entropy loss jointly improve the discrimination power of the model. Published in: IEEE Tr
To mine the spectral-spatial information of target pixel in hyperspectral image classification (HSIC), convolutional neural network (CNN)-based models widely adopt patch-based input pattern, where a patch represents its central pixel and the neighbor pixels play auxiliary roles in the classification process. However, compared to the central pixel, its neighbor pixels often have different contributions for classification. Although many existing patch-based CNNs could adaptively emphasize the spatial neighbor information, most of them ignore the latent relationship between the center pixel and its neighbor pixels. Moreover, efficient spectral-spatial feature extraction has been a difficult yet vital topic for HSIC. To address the mentioned problems, a central vector oriented self-similarity network (CVSSN) is proposed for HSIC. Specifically, based on two similarity measures, we firstly design an adaptive weight addition based spectral vector self-similarity module (AWA-SVSS) in input space and a Euclidean distance based feature vector self-similarity module (ED-FVSS) in feature space to fully mine the central vector oriented spatial relationships. Besides, a spectral-spatial information fusion module (SSIF) is formulated as a new pattern to fuse the central 1D spectral vector and the corresponding 3D patch for efficient spectral-spatial feature learning of the subsequent modules. Moreover, we implement a channel spatial separation convolution module (CSS-Conv) and a scale information complementary convolution module (SIC-Conv) for efficient spectral-spatial feature learning. Extensive experimental results on four popular HSI data sets demonstrate the effectiveness and efficiency of the proposed method compared with other state-of-the-art methods. The source code is available at https://github.com/lms-07/CVSSN.
This paper proposes a probabilistic deep metric learning (PDML) framework for hyperspectral image classification, which aims to predict the category of each pixel for an image captured by hyperspectral sensors. The core problem for hyperspectral image classification is the spectral variability between intraclass materials and the spectral similarity between interclass materials, motivating the further incorporation of spatial information to differentiate a pixel based on its surrounding patch. However, different pixels and even the same pixel in one patch might not encode the same material due to the low spatial resolution of most hyperspectral sensors, leading to an inconsistent judgment of a specific pixel. To address this issue, we propose a probabilistic deep metric learning framework to model the categorical uncertainty of the spectral distribution of an observed pixel. We propose to learn a global probabilistic distribution for each pixel in the patch and a probabilistic metric to model the distance between distributions. We treat each pixel in a patch as a training sample, enabling us to exploit more information from the patch compared with conventional methods. Our framework can be readily applied to existing hyperspectral image classification methods with various network architectures and loss functions. Extensive experiments on four widely used datasets including IN, UP, KSC, and Houston 2013 datasets demonstrate that our framework improves the performance of existing methods and further achieves the state of the art. Code is available at: https://github.com/wzzheng/PDML.