Neural ordinary differential equations (neural ODEs) have emerged as a novel network architecture that bridges dynamical systems and deep learning. However, the gradient obtained with the continuous adjoint method in the vanilla neural ODE is not reverse-accurate. Other approaches suffer either from an excessive memory requirement due to deep computational graphs or from limited choices for the time integration scheme, hampering their application to large-scale complex dynamical systems. To achieve accurate gradients without compromising memory efficiency and flexibility, we present a new neural ODE framework, PNODE, based on high-level discrete adjoint algorithmic differentiation. By leveraging discrete adjoint time integrators and advanced checkpointing strategies tailored for these integrators, PNODE can provide a balance between memory and computational costs, while computing the gradients consistently and accurately. We provide an open-source implementation based on PyTorch and PETSc, one of the most commonly used portable, scalable scientific computing libraries. We demonstrate the performance through extensive numerical experiments on image classification and continuous normalizing flow problems. We show that PNODE achieves the highest memory efficiency when compared with other reverse-accurate methods. On the image classification problems, PNODE is up to two times faster than the vanilla neural ODE and up to 2.3 times faster than the best existing reverse-accurate method. We also show that PNODE enables the use of the implicit time integration methods that are needed for stiff dynamical systems.
It is helpful in preventing colorectal cancer to detect and treat polyps in the gastrointestinal tract early. However, there have been few studies to date on designing polyp image classification networks that balance efficiency and accuracy. This challenge is mainly attributed to the fact that polyps are similar to other pathologies and have complex features influenced by texture, color, and morphology. In this paper, we propose a novel network DFE-IANet based on both spectral transformation and feature interaction. Firstly, to extract detailed features and multi-scale features, the features are transformed by the multi-scale frequency domain feature extraction (MSFD) block to extract texture details at the fine-grained level in the frequency domain. Secondly, the multi-scale interaction attention (MSIA) block is designed to enhance the network's capability of extracting critical features. This block introduces multi-scale features into self-attention, aiming to adaptively guide the network to concentrate on vital regions. Finally, with a compact parameter of only 4M, DFE-IANet outperforms the latest and classical networks in terms of efficiency. Furthermore, DFE-IANet achieves state-of-the-art (SOTA) results on the challenging Kvasir dataset, demonstrating a remarkable Top-1 accuracy of 93.94%. This outstanding accuracy surpasses ViT by 8.94%, ResNet50 by 1.69%, and VMamba by 1.88%. Our code is publicly available at https://github.com/PURSUETHESUN/DFE-IANet.
초분광 이미지(HSI) 작물 분석은 정밀 농업에 필수적이며, 정확한 작물 모니터링과 평가를 위해 풍부한 스펙트럼 및 공간 정보를 포착합니다. 그러나 HSI 분류는 높은 스펙트럼 차원, 공간적 복잡성, 클래스 불균형, 제한된 레이블 샘플로 인해 여전히 어려운 과제입니다. 이러한 문제를 해결하기 위해 본 논문은 다중 스케일 합성곱 특징 추출, 스펙트럼 어텐션, 양방향 상태 공간 모델링, 양자 영감 학습을 결합한 BiSpectral Mamba 기반 프레임워크를 제안합니다. 다중 스케일 CNN 백본은 먼저 여러 해상도에 걸친 특징 융합을 통해 계층적 공간-스펙트럼 표현을 추출합니다. 그런 다음 스펙트럼 어텐션 메커니즘은 중복되고 잡음이 있는 채널을 억제하면서 정보성 밴드를 강조합니다. 정제된 특징은 초분광 특징 맵을 순차적 토큰으로 모델링하여 순방향 및 역방향 모두에서 장거리 의존성을 포착하는 BiSpectral Mamba 모듈에서 처리됩니다. 또한, 학습 안정성을 개선하고 클래스 불균형을 완화하기 위해 클래스 가중치 최적화 및 특징 융합 전략이 통합됩니다. UAVHSI-Crop 데이터셋에 대한 실험적 평가는 제안된 프레임워크의 효과를 입증하며, 84.83%의 전체 정확도를 달성했습니다. 결과는 합성곱, 어텐션 기반, 상태 공간 모델링 구성 요소를 통합하면 작물 분류를 위한 강건한 공간-스펙트럼 특징 학습이 가능함을 보여줍니다. 제안된 프레임워크는 작물 질병 탐지, 수확량 예측, 토양 수분 추정을 포함한 광범위한 농업 및 원격 탐사 응용 분야에서 잠재력을 보여주며, 초분광 이미지 분석을 위한 구조화된 상태 공간 및 양자 영감 아키텍처의 효과를 강조합니다.
Despite the tremendous progress in zero-shot learning(ZSL), the majority of existing methods still rely on human-annotated attributes, which are difficult to annotate and scale. An unsupervised alternative is to represent each class using the word embedding associated with its semantic class name. However, word embeddings extracted from pre-trained language models do not necessarily capture visual similarities, resulting in poor zero-shot performance. In this work, we argue that online textual documents, e.g., Wikipedia, contain rich visual descriptions about object classes, therefore can be used as powerful unsupervised side information for ZSL. To this end, we propose I2DFormer, a novel transformer-based ZSL framework that jointly learns to encode images and documents by aligning both modalities in a shared embedding space. In order to distill discriminative visual words from noisy documents, we introduce a new cross-modal attention module that learns fine-grained interactions between image patches and document words. Consequently, our I2DFormer not only learns highly discriminative document embeddings that capture visual similarities but also gains the ability to localize visually relevant words in image regions. Quantitatively, we demonstrate that our I2DFormer significantly outperforms previous unsupervised semantic embeddings under both zero-shot and generalized zero-shot learning settings on three public datasets. Qualitatively, we show that our method leads to highly interpretable results where document words can be grounded in the image regions.
As growing usage of social media websites in the recent decades, the amount of news articles spreading online rapidly, resulting in an unprecedented scale of potentially fraudulent information. Although a plenty of studies have applied the supervised machine learning approaches to detect such content, the lack of gold standard training data has hindered the development. Analysing the single data format, either fake text description or fake image, is the mainstream direction for the current research. However, the misinformation in real-world scenario is commonly formed as a text-image pair where the news article/news title is described as text content, and usually followed by the related image. Given the strong ability of learning features without labelled data, contrastive learning, as a self-learning approach, has emerged and achieved success on the computer vision. In this paper, our goal is to explore the constrastive learning in the domain of misinformation identification. We developed a self-learning model and carried out the comprehensive experiments on a public data set named COSMOS. Comparing to the baseline classifier, our model shows the superior performance of non-matched image-text pair detection (approximately 10%) when the training data is insufficient. In addition, we observed the stability for contrsative learning and suggested the use of it offers large reductions in the number of training data, whilst maintaining comparable classification results.
This work aims to leverage pre-trained foundation models, such as contrastive language-image pre-training (CLIP) and segment anything model (SAM), to address weakly supervised semantic segmentation (WSSS) using image-level labels. To this end, we propose a coarse-to-fine framework based on CLIP and SAM for generating high-quality segmentation seeds. Specifically, we construct an image classification task and a seed segmentation task, which are jointly performed by CLIP with frozen weights and two sets of learnable task-specific prompts. A SAM-based seeding (SAMS) module is designed and applied to each task to produce either coarse or fine seed maps. Moreover, we design a multi-label contrastive loss supervised by image-level labels and a CAM activation loss supervised by the generated coarse seed map. These losses are used to learn the prompts, which are the only parts need to be learned in our framework. Once the prompts are learned, we input each image along with the learned segmentation-specific prompts into CLIP and the SAMS module to produce high-quality segmentation seeds. These seeds serve as pseudo labels to train an off-the-shelf segmentation network like other two-stage WSSS methods. Experiments show that our method achieves the state-of-the-art performance on PASCAL VOC 2012 and competitive results on MS COCO 2014. Code is available at https://github.com/HAL-42/FMA-WSSS.git.
단일 세포 광학 현미경 이미지는 세포 표현형(phenotype)을 특성화하는 데 중요한 데이터 원천이 되었지만, 그 복잡성과 이질성(heterogeneity)은 고속 자동 분석에 어려움을 제기한다. 기존 표현 학습(representation learning) 방법은 대부분 과제 지향(task-oriented) 모델링에 의존하며, 이는 특정 데이터셋과 사전 정의된 과제에 제한되어 서로 다른 세포 유형, 현미경 방식(modality), 실험 조건에 걸쳐 일반화하기 어렵다. 최근 범용(general-purpose) 방법이 이미지 표현의 일반화 능력을 향상시켰지만, 실험 배경과 생물학적 문맥 정보의 제한적 활용은 여전히 복잡한 표현형 분석에서 과제를 남긴다. 본 연구에서는 단일 세포 광학 현미경 이미지 표현을 위한 비전-언어 파운데이션 모델인 scMIR을 제안한다. 자기지도(self-supervised) 이미지 재구성과 텍스트 유도 교차 모달 정렬(cross-modal alignment)을 시너지적으로 결합함으로써, scMIR은 형태학적 정보와 생물학적 의미 정보를 통합 표현 공간에서 동시에 인코딩할 수 있다. scMIR은 다양한 세포 유형, 현미경 방식, 교란(perturbation) 조건을 아우르는 207,957개의 이미지-텍스트 쌍으로 사전학습(pre-training)되었다. scMIR은 16개 벤치마크 데이터셋을 사용한 다양한 복잡 과제에 대한 체계적 평가에서 세포 분류, 클러스터링, 표현형 추론, 배치 효과 보정 과제를 포함하여 기존 범용 모델과 과제 지향 방법을 능가한다. 또한 scMIR은 과제별 미세조정 없이도 다양한 과제에 걸쳐 강한 일반화 능력을 보인다. 고유한 장점을 바탕으로, scMIR은 다양한 하류(downstream) 분석 과제를 지원함으로써 고속 표현형 분석(phenotyping) 워크플로의 표준화와 자동화를 촉진할 수 있을 것으로 기대한다.
Aims. The treatment of astronomical image time series has won increasing attention in recent years. Indeed, numerous surveys following up on transient objects are in progress or under construction, such as the Vera Rubin Observatory Legacy Survey for Space and Time (LSST), which is poised to produce huge amounts of these time series. The associated scientific topics are extensive, ranging from the study of objects in our galaxy to the observation of the most distant supernovae for measuring the expansion of the universe. With such a large amount of data available, the need for robust automatic tools to detect and classify celestial objects is growing steadily. Methods. This study is based on the assumption that astronomical images contain more information than light curves. In this paper, we propose a novel approach based on deep learning for classifying different types of space objects directly using images. We named our approach ConvEntion, which stands for CONVolutional attENTION. It is based on convolutions and transformers, which are new approaches for the treatment of astronomical image time series. Our solution integrates spatio-temporal features and can be applied to various types of image datasets with any number of bands. Results. In this work, we solved various problems the datasets tend to suffer from and we present new results for classifications using astronomical image time series with an increase in accuracy of 13%, compared to state-of-the-art approaches that use image time series, and a 12% increase, compared to approaches that use light curves.
Over the past decades, state-of-the-art medical image segmentation has heavily rested on signal processing paradigms, most notably registration-based label propagation and pair-wise patch comparison, which are generally slow despite a high segmentation accuracy. In recent years, deep learning has revolutionalized computer vision with many practices outperforming prior art, in particular the convolutional neural network (CNN) studies on image classification. Deep CNN has also started being applied to medical image segmentation lately, but generally involves long training and demanding memory requirements, achieving limited success. We propose a patch-based deep learning framework based on a revisit to the classic neural network model with substantial modernization, including the use of Rectified Linear Unit (ReLU) activation, dropout layers, 2.5D tri-planar patch multi-pathway settings. In a test application to hippocampus segmentation using 100 brain MR images from the ADNI database, our approach significantly outperformed prior art in terms of both segmentation accuracy and speed: scoring a median Dice score up to 90.98% on a near real-time performance (<1s).
Explainable artificial intelligence is the attempt to elucidate the workings of systems too complex to be directly accessible to human cognition through suitable side-information referred to as "explanations". We present a trainable explanation module for convolutional image classifiers we call bounded logit attention (BLA). The BLA module learns to select a subset of the convolutional feature map for each input instance, which then serves as an explanation for the classifier's prediction. BLA overcomes several limitations of the instancewise feature selection method "learning to explain" (L2X) introduced by Chen et al. (2018): 1) BLA scales to real-world sized image classification problems, and 2) BLA offers a canonical way to learn explanations of variable size. Due to its modularity BLA lends itself to transfer learning setups and can also be employed as a post-hoc add-on to trained classifiers. Beyond explainability, BLA may serve as a general purpose method for differentiable approximation of subset selection. In a user study we find that BLA explanations are preferred over explanations generated by the popular (Grad-)CAM method.
Vision Transformers (ViT) have made many breakthroughs in computer vision tasks. However, considerable redundancy arises in the spatial dimension of an input image, leading to massive computational costs. Therefore, We propose a coarse-to-fine vision transformer (CF-ViT) to relieve computational burden while retaining performance in this paper. Our proposed CF-ViT is motivated by two important observations in modern ViT models: (1) The coarse-grained patch splitting can locate informative regions of an input image. (2) Most images can be well recognized by a ViT model in a small-length token sequence. Therefore, our CF-ViT implements network inference in a two-stage manner. At coarse inference stage, an input image is split into a small-length patch sequence for a computationally economical classification. If not well recognized, the informative patches are identified and further re-split in a fine-grained granularity. Extensive experiments demonstrate the efficacy of our CF-ViT. For example, without any compromise on performance, CF-ViT reduces 53% FLOPs of LV-ViT, and also achieves 2.01x throughput.
Medical visual question answering (VQA) is a challenging multimodal task, where Vision-Language Pre-training (VLP) models can effectively improve the generalization performance. However, most methods in the medical field treat VQA as an answer classification task which is difficult to transfer to practical application scenarios. Additionally, due to the privacy of medical images and the expensive annotation process, large-scale medical image-text pairs datasets for pretraining are severely lacking. In this paper, we propose a large-scale MultI-task Self-Supervised learning based framework (MISS) for medical VQA tasks. Unlike existing methods, we treat medical VQA as a generative task. We unify the text encoder and multimodal encoder and align image-text features through multi-task learning. Furthermore, we propose a Transfer-and-Caption method that extends the feature space of single-modal image datasets using Large Language Models (LLMs), enabling those traditional medical vision field task data to be applied to VLP. Experiments show that our method achieves excellent results with fewer multimodal datasets and demonstrates the advantages of generative VQA models.
Gradient inversion attacks aim to reconstruct local training data from intermediate gradients exposed in the federated learning framework. Despite successful attacks, all previous methods, starting from reconstructing a single data point and then relaxing the single-image limit to batch level, are only tested under hard label constraints. Even for single-image reconstruction, we still lack an analysis-based algorithm to recover augmented soft labels. In this work, we change the focus from enlarging batchsize to investigating the hard label constraints, considering a more realistic circumstance where label smoothing and mixup techniques are used in the training process. In particular, we are the first to initiate a novel algorithm to simultaneously recover the ground-truth augmented label and the input feature of the last fully-connected layer from single-input gradients, and provide a necessary condition for any analytical-based label recovery methods. Extensive experiments testify to the label recovery accuracy, as well as the benefits to the following image reconstruction. We believe soft labels in classification tasks are worth further attention in gradient inversion attacks.
Neural networks struggle with image classification when biases are learned and misleads correlations, affecting their generalization and performance. Previous methods require attribute labels (e.g. background, color) or utilizes Generative Adversarial Networks (GANs) to mitigate biases. We introduce DiffuBias, a novel pipeline for text-to-image generation that enhances classifier robustness by generating bias-conflict samples, without requiring training during the generation phase. Utilizing pretrained diffusion and image captioning models, DiffuBias generates images that challenge the biases of classifiers, using the top-$K$ losses from a biased classifier ($f_B$) to create more representative data samples. This method not only debiases effectively but also boosts classifier generalization capabilities. To the best of our knowledge, DiffuBias is the first approach leveraging a stable diffusion model to generate bias-conflict samples in debiasing tasks. Our comprehensive experimental evaluations demonstrate that DiffuBias achieves state-of-the-art performance on benchmark datasets. We also conduct a comparative analysis of various generative models in terms of carbon emissions and energy consumption to highlight the significance of computational efficiency.
Sparse representation based classification (SRC) is popularly used in many applications such as face recognition, and implemented in two steps: representation coding and classification. For a given set of testing images, SRC codes every image over the base images as a sparse representation then classifies it to the class with the least representation error. This scheme utilizes an individual representation rather than the collective one to classify such a set of images, doing so obviously ignores the correlation among the given images. In this paper, a joint representation classification (JRC) for collective face recognition is proposed. JRC takes the correlation of multiple images as well as a single representation into account. Under the assumption that the given face images are generally related to each other, JRC codes all the testing images over the base images simultaneously to facilitate recognition. To this end, the testing inputs are aligned into a matrix and the joint representation coding is formulated to a generalized $l_{2,q}-l_{2,p}$-minimization problem. To uniformly solve the induced optimization problems for any $q\in[1,2]$ and $p\in (0,2]$, an iterative quadratic method (IQM) is developed. IQM is proved to be a strict descent algorithm with convergence to the optimal solution. Moreover, a more practical IQM is proposed for large-scale case. Experimental results on three public databases show that the JRC with practical IQM no only saves much computational cost but also achieves better performance in collective face recognition than the state-of-the-arts.
Breast cancer is the most common invasive cancer in women, affecting more than 10% of women worldwide. Microscopic analysis of a biopsy remains one of the most important methods to diagnose the type of breast cancer. This requires specialized analysis by pathologists, in a task that i) is highly time- and cost-consuming and ii) often leads to nonconsensual results. The relevance and potential of automatic classification algorithms using hematoxylin-eosin stained histopathological images has already been demonstrated, but the reported results are still sub-optimal for clinical use. With the goal of advancing the state-of-the-art in automatic classification, the Grand Challenge on BreAst Cancer Histology images (BACH) was organized in conjunction with the 15th International Conference on Image Analysis and Recognition (ICIAR 2018). A large annotated dataset, composed of both microscopy and whole-slide images, was specifically compiled and made publicly available for the BACH challenge. Following a positive response from the scientific community, a total of 64 submissions, out of 677 registrations, effectively entered the competition. From the submitted algorithms it was possible to push forward the state-of-the-art in terms of accuracy (87%) in automatic classification of breast cancer with histopathological images. Convolutional neuronal networks were the most successful methodology in the BACH challenge. Detailed analysis of the collective results allowed the identification of remaining challenges in the field and recommendations for future developments. The BACH dataset remains publically available as to promote further improvements to the field of automatic classification in digital pathology.
Despite the recent success of convolutional neural networks in texture recognition, model-based descriptors are still competitive, especially when we do not have access to large amounts of annotated data for training and the interpretation of the model is an important issue. Among the model-based approaches, fractal geometry has been one of the most popular, especially in biological applications. Nevertheless, fractals are part of a much broader family of models, which are the non-linear operators, studied in chaos theory. In this context, we propose here a chaos-based local descriptor for texture recognition. More specifically, we map the image into the three-dimensional Euclidean space, iterate a chaotic map over this three-dimensional structure and convert it back to the original image. From such chaos-transformed image at each iteration we collect local descriptors (here we use local binary patters) and those descriptors compose the feature representation of the texture. The performance of our method was verified on the classification of benchmark databases and in the identification of Brazilian plant species based on the texture of the leaf surface. The achieved results confirmed our expectation of a competitive performance, even when compared with some learning-based modern approaches in the literature.
CLIP에서 이미지와 텍스트 표현 간의 모달리티 갭(modality gap)을 줄이는 것은 일반적으로 교차 모달 정렬(cross-modal alignment) 및 다운스트림 성능을 향상시킬 것으로 기대된다. 그러나 평균 이미지-텍스트 갭이 작아진다고 해서 반드시 일관된 정확도 향상으로 이어지는 것은 아니다. 본 논문에서는 입력 이미지에 대해 가장 유사한 클래스-텍스트 프로토타입을 선택하는 제로샷(zero-shot) 분류(classification)의 결정 구조 관점에서 이러한 불일치를 분석한다. 제로샷 정확도는 평균 이미지-텍스트 정렬뿐만 아니라 클래스별 결정 마진(decision margin)에도 의존한다. 해석적으로 다루기 쉬운 사례로 선형 보정(Linear correction)을 활용하여, 모달리티 갭 보정이 클래스 간의 상대적 결정 구조를 변경하고 예측이 소수의 클래스 하위 집합에 집중되도록 유도할 수 있음을 보인다. 본 논문에서는 이러한 출력 공간(output-space) 실패 모드를 예측 수준 허브니스(prediction-level hubness)라고 칭한다. 나아가 여러 데이터셋(dataset)에 걸친 실험을 통해 갭 보정 시 발생하는 정확도 저하가 선형 보정과 학습 기반 보정 방법 모두에서 예측 집중도 증가와 일관되게 연관되어 있음을 보인다. 이는 다운스트림 결정 구조 관점에서 모달리티 갭 감소가 왜 CLIP의 제로샷 정확도를 일관되게 향상시키지 못하는지에 대한 체계적인 설명을 제공한다. 본 연구 결과는 갭 보정이 평균 정렬뿐만 아니라 다운스트림 예측 구조에 미치는 영향으로도 평가되어야 함을 시사한다.
Semi-supervised learning plays an important role in large-scale machine learning. Properly using additional unlabeled data (largely available nowadays) often can improve the machine learning accuracy. However, if the machine learning model is misspecified for the underlying true data distribution, the model performance could be seriously jeopardized. This issue is known as model misspecification. To address this issue, we focus on generative models and propose a criterion to detect the onset of model misspecification by measuring the performance difference between models obtained using supervised and semi-supervised learning. Then, we propose to automatically modify the generative models during model training to achieve an unbiased generative model. Rigorous experiments were carried out to evaluate the proposed method using two image classification data sets PASCAL VOC'07 and MIR Flickr. Our proposed method has been demonstrated to outperform a number of state-of-the-art semi-supervised learning approaches for the classification task.
We present a new end-to-end network architecture for facial expression recognition with an attention model. It focuses attention in the human face and uses a Gaussian space representation for expression recognition. We devise this architecture based on two fundamental complementary components: (1) facial image correction and attention and (2) facial expression representation and classification. The first component uses an encoder-decoder style network and a convolutional feature extractor that are pixel-wise multiplied to obtain a feature attention map. The second component is responsible for obtaining an embedded representation and classification of the facial expression. We propose a loss function that creates a Gaussian structure on the representation space. To demonstrate the proposed method, we create two larger and more comprehensive synthetic datasets using the traditional BU3DFE and CK+ facial datasets. We compared results with the PreActResNet18 baseline. Our experiments on these datasets have shown the superiority of our approach in recognizing facial expressions.