PyTorchKR Papers

“Image Classification”

총 18,385편
태그·기간 등 상세 필터

카테고리 탐색

arXiv:2405.13285

Enhancing Active Learning for Sentinel 2 Imagery through Contrastive Learning and Uncertainty Estimation

In this paper, we introduce a novel method designed to enhance label efficiency in satellite imagery analysis by integrating semi-supervised learning (SSL) with active learning strategies. Our approach utilizes contrastive learning together with uncertainty estimations via Monte Carlo Dropout (MC Dropout), with a particular focus on Sentinel-2 imagery analyzed using the Eurosat dataset. We explore the effectiveness of our method in scenarios featuring both balanced and unbalanced class distributions. Our results show that the proposed method performs better than several other popular methods in this field, enabling significant savings in labeling effort while maintaining high classification accuracy. These findings highlight the potential of our approach to facilitate scalable and cost-effective satellite image analysis, particularly advantageous for extensive environmental monitoring and land use classification tasks.

arXiv:2406.01235

Boosting Spatial-Spectral Masked Auto-Encoder Through Mining Redundant Spectra for HSI-SAR/LiDAR Classification

Although recent masked image modeling (MIM)-based HSI-LiDAR/SAR classification methods have gradually recognized the importance of the spectral information, they have not adequately addressed the redundancy among different spectra, resulting in information leakage during the pretraining stage. This issue directly impairs the representation ability of the model. To tackle the problem, we propose a new strategy, named Mining Redundant Spectra (MRS). Unlike randomly masking spectral bands, MRS selectively masks them by similarity to increase the reconstruction difficulty. Specifically, a random spectral band is chosen during pretraining, and the selected and highly similar bands are masked. Experimental results demonstrate that employing the MRS strategy during the pretraining stage effectively improves the accuracy of existing MIM-based methods on the Berlin and Houston 2018 datasets.

arXiv:2501.04873

Back Home: A Machine Learning Approach to Seashell Classification and Ecosystem Restoration

In Costa Rica, an average of 5 tons of seashells are extracted from ecosystems annually. Confiscated seashells, cannot be returned to their ecosystems due to the lack of origin recognition. To address this issue, we developed a convolutional neural network (CNN) specifically for seashell identification. We built a dataset from scratch, consisting of approximately 19000 images from the Pacific and Caribbean coasts. Using this dataset, the model achieved a classification accuracy exceeding 85%. The model has been integrated into a user-friendly application, which has classified over 36,000 seashells to date, delivering real-time results within 3 seconds per image. To further enhance the system's accuracy, an anomaly detection mechanism was incorporated to filter out irrelevant or anomalous inputs, ensuring only valid seashell images are processed.

arXiv:2505.16923

TULiP: Test-time Uncertainty Estimation via Linearization and Weight Perturbation

A reliable uncertainty estimation method is the foundation of many modern out-of-distribution (OOD) detectors, which are critical for safe deployments of deep learning models in the open world. In this work, we propose TULiP, a theoretically-driven post-hoc uncertainty estimator for OOD detection. Our approach considers a hypothetical perturbation applied to the network before convergence. Based on linearized training dynamics, we bound the effect of such perturbation, resulting in an uncertainty score computable by perturbing model parameters. Ultimately, our approach computes uncertainty from a set of sampled predictions. We visualize our bound on synthetic regression and classification datasets. Furthermore, we demonstrate the effectiveness of TULiP using large-scale OOD detection benchmarks for image classification. Our method exhibits state-of-the-art performance, particularly for near-distribution samples.

arXiv:2312.15663

IQAGPT: Image Quality Assessment with Vision-language and ChatGPT Models

Large language models (LLMs), such as ChatGPT, have demonstrated impressive capabilities in various tasks and attracted an increasing interest as a natural language interface across many domains. Recently, large vision-language models (VLMs) like BLIP-2 and GPT-4 have been intensively investigated, which learn rich vision-language correlation from image-text pairs. However, despite these developments, the application of LLMs and VLMs in image quality assessment (IQA), particularly in medical imaging, remains to be explored, which is valuable for objective performance evaluation and potential supplement or even replacement of radiologists' opinions. To this end, this paper introduces IQAGPT, an innovative image quality assessment system integrating an image quality captioning VLM with ChatGPT for generating quality scores and textual reports. First, we build a CT-IQA dataset for training and evaluation, comprising 1,000 CT slices with diverse quality levels professionally annotated. To better leverage the capabilities of LLMs, we convert annotated quality scores into semantically rich text descriptions using a prompt template. Second, we fine-tune the image quality captioning VLM on the CT-IQA dataset to generate quality descriptions. The captioning model fuses the image and text features through cross-modal attention. Third, based on the quality descriptions, users can talk with ChatGPT to rate image quality scores or produce a radiological quality report. Our preliminary results demonstrate the feasibility of assessing image quality with large models. Remarkably, our IQAGPT outperforms GPT-4 and CLIP-IQA, as well as the multi-task classification and regression models that solely rely on images.

arXiv:1507.01251

Autoencoding the Retrieval Relevance of Medical Images

Content-based image retrieval (CBIR) of medical images is a crucial task that can contribute to a more reliable diagnosis if applied to big data. Recent advances in feature extraction and classification have enormously improved CBIR results for digital images. However, considering the increasing accessibility of big data in medical imaging, we are still in need of reducing both memory requirements and computational expenses of image retrieval systems. This work proposes to exclude the features of image blocks that exhibit a low encoding error when learned by a $n/p/n$ autoencoder ($p\!<\!n$). We examine the histogram of autoendcoding errors of image blocks for each image class to facilitate the decision which image regions, or roughly what percentage of an image perhaps, shall be declared relevant for the retrieval task. This leads to reduction of feature dimensionality and speeds up the retrieval process. To validate the proposed scheme, we employ local binary patterns (LBP) and support vector machines (SVM) which are both well-established approaches in CBIR research community. As well, we use IRMA dataset with 14,410 x-ray images as test data. The results show that the dimensionality of annotated feature vectors can be reduced by up to 50% resulting in speedups greater than 27% at expense of less than 1% decrease in the accuracy of retrieval when validating the precision and recall of the top 20 hits.

arXiv:2502.04263공식 코드 · Papers with Code

Cross the Gap: Exposing the Intra-modal Misalignment in CLIP via Modality Inversion

Pre-trained multi-modal Vision-Language Models like CLIP are widely used off-the-shelf for a variety of applications. In this paper, we show that the common practice of individually exploiting the text or image encoders of these powerful multi-modal models is highly suboptimal for intra-modal tasks like image-to-image retrieval. We argue that this is inherently due to the CLIP-style inter-modal contrastive loss that does not enforce any intra-modal constraints, leading to what we call intra-modal misalignment. To demonstrate this, we leverage two optimization-based modality inversion techniques that map representations from their input modality to the complementary one without any need for auxiliary data or additional trained adapters. We empirically show that, in the intra-modal tasks of image-to-image and text-to-text retrieval, approaching these tasks inter-modally significantly improves performance with respect to intra-modal baselines on more than fifteen datasets. Additionally, we demonstrate that approaching a native inter-modal task (e.g. zero-shot image classification) intra-modally decreases performance, further validating our findings. Finally, we show that incorporating an intra-modal term in the pre-training objective or narrowing the modality gap between the text and image feature embedding spaces helps reduce the intra-modal misalignment. The code is publicly available at: https://github.com/miccunifi/Cross-the-Gap.

arXiv:2210.03743공식 코드 · Papers with Code

Single Image Super-Resolution Based on Capsule Neural Networks

Single image super-resolution (SISR) is the process of obtaining one high-resolution version of a low-resolution image by increasing the number of pixels per unit area. This method has been actively investigated by the research community, due to the wide variety of real-world problems where it can be applied, from aerial and satellite imaging to compressed image and video enhancement. Despite the improvements achieved by deep learning in the field, the vast majority of the used networks are based on traditional convolutions, with the solutions focusing on going deeper and/or wider, and innovations coming from jointly employing successful concepts from other fields. In this work, we decided to step up from the traditional convolutions and adopt the concept of capsules. Since their overwhelming results both in image classification and segmentation problems, we question how suitable they are for SISR. We also verify that different solutions share most of their configurations, and argue that this trend leads to fewer explorations of network varieties. During our experiments, we check various strategies to improve results, ranging from new and different loss functions to changes in the capsule layers. Our network achieved good results with fewer convolutional-based layers, showing that capsules might be a concept worth applying in the image super-resolution problem.

arXiv:1709.02565

Segmentation and Classification of Cine-MR Images Using Fully Convolutional Networks and Handcrafted Features

Three-dimensional cine-MRI is of crucial importance for assessing the cardiac function. Features that describe the anatomy and function of cardiac structures (e.g. Left Ventricle (LV), Right Ventricle (RV), and Myocardium(MC)) are known to have significant diagnostic value and can be computed from 3D cine-MR images. However, these features require precise segmentation of cardiac structures. Among the fully automated segmentation methods, Fully Convolutional Networks (FCN) with Skip Connections have shown robustness in medical segmentation problems. In this study, we develop a complete pipeline for classification of subjects with cardiac conditions based on 3D cine-MRI. For the segmentation task, we develop a 2D FCN and introduce Parallel Paths (PP) as a way to exploit the 3D information of the cine-MR image. For the classification task, 125 features were extracted from the segmented structures, describing their anatomy and function. Next, a two-stage pipeline for feature selection using the LASSO method is developed. A subset of 20 features is selected for classification. Each subject is classified using an ensemble of Logistic Regression, Multi-Layer Perceptron, and Support Vector Machine classifiers through majority voting. The Dice Coefficient for segmentation was 0.95+-0.03, 0.89+-0.13, and 0.90+-0.03 for LV, RV, and MC respectively. The 8-fold cross validation accuracy for the classification task was 95.05% and 92.77% based on ground truth and the proposed methods segmentations respectively. The results show that the PPs increase the segmentation accuracy, by exploiting the spatial relations. Moreover, the classification algorithm and the features showed discriminability while keeping the sensitivity to segmentation error as low as possible.

arXiv:2503.11219

MEET: A Million-Scale Dataset for Fine-Grained Geospatial Scene Classification with Zoom-Free Remote Sensing Imagery

Accurate fine-grained geospatial scene classification using remote sensing imagery is essential for a wide range of applications. However, existing approaches often rely on manually zooming remote sensing images at different scales to create typical scene samples. This approach fails to adequately support the fixed-resolution image interpretation requirements in real-world scenarios. To address this limitation, we introduce the Million-scale finE-grained geospatial scEne classification dataseT (MEET), which contains over 1.03 million zoom-free remote sensing scene samples, manually annotated into 80 fine-grained categories. In MEET, each scene sample follows a scene-inscene layout, where the central scene serves as the reference, and auxiliary scenes provide crucial spatial context for finegrained classification. Moreover, to tackle the emerging challenge of scene-in-scene classification, we present the Context-Aware Transformer (CAT), a model specifically designed for this task, which adaptively fuses spatial context to accurately classify the scene samples. CAT adaptively fuses spatial context to accurately classify the scene samples by learning attentional features that capture the relationships between the center and auxiliary scenes. Based on MEET, we establish a comprehensive benchmark for fine-grained geospatial scene classification, evaluating CAT against 11 competitive baselines. The results demonstrate that CAT significantly outperforms these baselines, achieving a 1.88% higher balanced accuracy (BA) with the Swin-Large backbone, and a notable 7.87% improvement with the Swin-Huge backbone. Further experiments validate the effectiveness of each module in CAT and show the practical applicability of CAT in the urban functional zone mapping. The source code and dataset will be publicly available at https://jerrywyn.github.io/project/MEET.html.

arXiv:1804.10892

Local Learning with Deep and Handcrafted Features for Facial Expression Recognition

We present an approach that combines automatic features learned by convolutional neural networks (CNN) and handcrafted features computed by the bag-of-visual-words (BOVW) model in order to achieve state-of-the-art results in facial expression recognition. To obtain automatic features, we experiment with multiple CNN architectures, pre-trained models and training procedures, e.g. Dense-Sparse-Dense. After fusing the two types of features, we employ a local learning framework to predict the class label for each test image. The local learning framework is based on three steps. First, a k-nearest neighbors model is applied in order to select the nearest training samples for an input test image. Second, a one-versus-all Support Vector Machines (SVM) classifier is trained on the selected training samples. Finally, the SVM classifier is used to predict the class label only for the test image it was trained for. Although we have used local learning in combination with handcrafted features in our previous work, to the best of our knowledge, local learning has never been employed in combination with deep features. The experiments on the 2013 Facial Expression Recognition (FER) Challenge data set, the FER+ data set and the AffectNet data set demonstrate that our approach achieves state-of-the-art results. With a top accuracy of 75.42% on FER 2013, 87.76% on the FER+, 59.58% on AffectNet 8-way classification and 63.31% on AffectNet 7-way classification, we surpass the state-of-the-art methods by more than 1% on all data sets.

arXiv:1902.00236

Natural and Adversarial Error Detection using Invariance to Image Transformations

We propose an approach to distinguish between correct and incorrect image classifications. Our approach can detect misclassifications which either occur $\it{unintentionally}$ ("natural errors"), or due to $\it{intentional~adversarial~attacks}$ ("adversarial errors"), both in a single $\it{unified~framework}$. Our approach is based on the observation that correctly classified images tend to exhibit robust and consistent classifications under certain image transformations (e.g., horizontal flip, small image translation, etc.). In contrast, incorrectly classified images (whether due to adversarial errors or natural errors) tend to exhibit large variations in classification results under such transformations. Our approach does not require any modifications or retraining of the classifier, hence can be applied to any pre-trained classifier. We further use state of the art targeted adversarial attacks to demonstrate that even when the adversary has full knowledge of our method, the adversarial distortion needed for bypassing our detector is $\it{no~longer~imperceptible~to~the~human~eye}$. Our approach obtains state-of-the-art results compared to previous adversarial detection methods, surpassing them by a large margin.

arXiv:2002.08991

Comparing Different Deep Learning Architectures for Classification of Chest Radiographs

Chest radiographs are among the most frequently acquired images in radiology and are often the subject of computer vision research. However, most of the models used to classify chest radiographs are derived from openly available deep neural networks, trained on large image-datasets. These datasets routinely differ from chest radiographs in that they are mostly color images and contain several possible image classes, while radiographs are greyscale images and often only contain fewer image classes. Therefore, very deep neural networks, which can represent more complex relationships in image-features, might not be required for the comparatively simpler task of classifying grayscale chest radiographs. We compared fifteen different architectures of artificial neural networks regarding training-time and performance on the openly available CheXpert dataset to identify the most suitable models for deep learning tasks on chest radiographs. We could show, that smaller networks such as ResNet-34, AlexNet or VGG-16 have the potential to classify chest radiographs as precisely as deeper neural networks such as DenseNet-201 or ResNet-151, while being less computationally demanding.

arXiv:2503.17731

Co-op: Correspondence-based Novel Object Pose Estimation

We propose Co-op, a novel method for accurately and robustly estimating the 6DoF pose of objects unseen during training from a single RGB image. Our method requires only the CAD model of the target object and can precisely estimate its pose without any additional fine-tuning. While existing model-based methods suffer from inefficiency due to using a large number of templates, our method enables fast and accurate estimation with a small number of templates. This improvement is achieved by finding semi-dense correspondences between the input image and the pre-rendered templates. Our method achieves strong generalization performance by leveraging a hybrid representation that combines patch-level classification and offset regression. Additionally, our pose refinement model estimates probabilistic flow between the input image and the rendered image, refining the initial estimate to an accurate pose using a differentiable PnP layer. We demonstrate that our method not only estimates object poses rapidly but also outperforms existing methods by a large margin on the seven core datasets of the BOP Challenge, achieving state-of-the-art accuracy.

arXiv:2611.16534v1cs.CV

얼굴 제시 공격 탐지를 위한 문맥 예측 저순위 적응

Context-Predictive Low-Rank Adaptation for Face Presentation Attack Detection

얼굴 제시 공격 탐지에는 획득 장치, 대상자, 공격 매체의 변화에도 물리적 스푸핑 단서를 구별하는 표현이 필요하다. 대형 시각 인코더는 전이 가능한 표현을 제공하지만, 기존의 이진 미세 조정은 부분 관측 상황에서 국소 얼굴 증거가 어떻게 표현되는지를 명시적으로 제약하지 않은 채 최종 결정만을 지도한다. 우리는 사전 학습된 CLIP 이미지 인코더의 저순위 조정과 결합 임베딩 예측 목적 함수를 결합한 문맥 예측 적응 프레임워크인 CP-JEPA를 제시한다. 학생 모델은 마스킹된 공간 위치에서 전체 이미지 교사 특징을 예측하며, 마스킹되지 않은 분류, 마스킹된 분류, 예측 일관성이 실제/스푸프 과제와의 연결을 유지한다. 지수 이동 평균 교사는 진화하는 잠재 목표를 제공한다. 보조 구성 요소는 학습 중에만 사용되며, 추론에는 LoRA로 적응된 CLIP 시각 인코더와 이진 분류기만 필요하다. {우리는 CP-JEPA를 단일 소스 및 다중 소스 OCIM 전이, CelebA-Spoof 추가 학습, CelebA-Spoof로의 전이, 두 소스 학습, SiW-Mv2에서의 미확인 공격 평가에서 평가한다.} 표준 OCIM 벤치마크에서 CP-JEPA는 평균 HTER 3.74%와 평균 AUC 98.73%를 달성한다. {단일 소스 평가는 열두 개의 소스-타깃 조합에서 평균 HTER 11.36%를 달성한다.} 세 개의 OCIM 학습 소스에 CelebA-Spoof를 추가하면 CP-JEPA는 평균 HTER를 3.45%에서 2.35%로 낮추고 평균 AUC를 98.68%에서 99.32%로 높인다. SiW-Mv2에서는 제외된 공격 범주 전반에서 평균 HTER 3.36%를 달성한다. 통제된 손실 제거 실험은 잠재 예측, 마스킹된 분류, 예측 일관성의 기여를 검토한다. 결과는 문맥 예측 적응이 데이터셋 간 PAD 성능을 향상시킬 수 있음을 보여 주지만, 그 이득은 학습 소스와 평가 설정에 따라 달라진다.

arXiv:2611.01921v1cs.CV

의사 컬러 심전도 이미지를 이용한 Torsades de Pointes 위험의 비전-언어 분류: T파 형태와 색상 맥락은 도움이 되는가, 방해가 되는가?

Vision-language classification of Torsades de Pointes risk from pseudo-coloured ECG images: do T-wave morphology and colour context help or hinder?

맥락 프롬프팅은 비전 모델의 성능을 저렴하게 높이는 방법으로 널리 소개되지만, 그 근거는 대부분 프롬프트에 명시된 속성이 이미지에도 존재하여 두 채널이 교란되는 벤치마크에서 나온다. 우리는 도메인 적응된 비전-언어 모델인 BiomedCLIP을 사용하여, 의미적으로 복잡한 생체의학 신호 분류 과제에서 맥락이 갖는 중요성을 조사한다. QT 노모그램과 의사 컬러링 기법을 이용해 심전도(ECG) 신호를 색상 강화 이미지로 변환하여 Torsades de Pointes 위험을 표시한 선행 연구를 바탕으로, 네 가지 맥락 전략과 네 가지 T파 형태 레이블 집합을 교차시켜 제로샷 및 클래스 균형 선형 프로브로 평가하였으며, 9,789개 이미지에 대해 5-겹 층화 교차 검증을 수행하였다. 제로샷 추론은 사용할 수 없는 수준이다. 학습 후에는 색상 강화 이미지가 모든 지표에서 무채색 이미지보다 우수하였다. 균형 정확도는 0.903 대 0.825, 민감도는 0.914 대 0.827, 특이도는 0.892 대 0.823, ROC-AUC는 0.955 대 0.888, 매튜스 상관계수(MCC)는 0.437 대 0.311이다. 이미지 내 색상과 프롬프트 내 색상을 교차시키면 교란된 채널이 분리된다. 프롬프트가 색상을 명시하든 그렇지 않든 이미지는 MCC +0.126(균형 정확도 +0.078, ROC-AUC +0.067)을 기여하는 반면, 색상을 명시하는 것은 어떤 지표에서도 변화를 일으키지 않는다. 맥락은 양면적이다. 최상의 프롬프트 집합은 색상이 있는 경우 MCC가 0.008, 없는 경우 0.016 향상되지만, 최악의 집합은 각각 0.017과 0.013 하락한다. 이미지로 표현된 신호에서는 신호를 어떻게 설명하느냐보다 어떻게 그리느냐에 효과의 열쇠가 있으며, 종점이 간격 측정값인 경우 T파 형태를 명시하는 프롬프트는 잘못된 양을 기술하는 것이다.

arXiv:2109.07607

Partner-Assisted Learning for Few-Shot Image Classification

Few-shot Learning has been studied to mimic human visual capabilities and learn effective models without the need of exhaustive human annotation. Even though the idea of meta-learning for adaptation has dominated the few-shot learning methods, how to train a feature extractor is still a challenge. In this paper, we focus on the design of training strategy to obtain an elemental representation such that the prototype of each novel class can be estimated from a few labeled samples. We propose a two-stage training scheme, Partner-Assisted Learning (PAL), which first trains a partner encoder to model pair-wise similarities and extract features serving as soft-anchors, and then trains a main encoder by aligning its outputs with soft-anchors while attempting to maximize classification performance. Two alignment constraints from logit-level and feature-level are designed individually. For each few-shot task, we perform prototype classification. Our method consistently outperforms the state-of-the-art method on four benchmarks. Detailed ablation studies of PAL are provided to justify the selection of each component involved in training.

arXiv:2204.05281공식 코드 · Papers with Code

Physically Disentangled Representations

State-of-the-art methods in generative representation learning yield semantic disentanglement, but typically do not consider physical scene parameters, such as geometry, albedo, lighting, or camera. We posit that inverse rendering, a way to reverse the rendering process to recover scene parameters from an image, can also be used to learn physically disentangled representations of scenes without supervision. In this paper, we show the utility of inverse rendering in learning representations that yield improved accuracy on downstream clustering, linear classification, and segmentation tasks with the help of our novel Leave-One-Out, Cycle Contrastive loss (LOOCC), which improves disentanglement of scene parameters and robustness to out-of-distribution lighting and viewpoints. We perform a comparison of our method with other generative representation learning methods across a variety of downstream tasks, including face attribute classification, emotion recognition, identification, face segmentation, and car classification. Our physically disentangled representations yield higher accuracy than semantically disentangled alternatives across all tasks and by as much as 18%. We hope that this work will motivate future research in applying advances in inverse rendering and 3D understanding to representation learning.

arXiv:2208.11739

Rethinking Cost-sensitive Classification in Deep Learning via Adversarial Data Augmentation

Cost-sensitive classification is critical in applications where misclassification errors widely vary in cost. However, over-parameterization poses fundamental challenges to the cost-sensitive modeling of deep neural networks (DNNs). The ability of a DNN to fully interpolate a training dataset can render a DNN, evaluated purely on the training set, ineffective in distinguishing a cost-sensitive solution from its overall accuracy maximization counterpart. This necessitates rethinking cost-sensitive classification in DNNs. To address this challenge, this paper proposes a cost-sensitive adversarial data augmentation (CSADA) framework to make over-parameterized models cost-sensitive. The overarching idea is to generate targeted adversarial examples that push the decision boundary in cost-aware directions. These targeted adversarial samples are generated by maximizing the probability of critical misclassifications and used to train a model with more conservative decisions on costly pairs. Experiments on well-known datasets and a pharmacy medication image (PMI) dataset made publicly available show that our method can effectively minimize the overall cost and reduce critical errors, while achieving comparable performance in terms of overall accuracy.

arXiv:2311.03422

Efficient and Low-Footprint Object Classification using Spatial Contrast

Event-based vision sensors traditionally compute temporal contrast that offers potential for low-power and low-latency sensing and computing. In this research, an alternative paradigm for event-based sensors using localized spatial contrast (SC) under two different thresholding techniques, relative and absolute, is investigated. Given the slow maturity of spatial contrast in comparison to temporal-based sensors, a theoretical simulated output of such a hardware sensor is explored. Furthermore, we evaluate traffic sign classification using the German Traffic Sign dataset (GTSRB) with well-known Deep Neural Networks (DNNs). This study shows that spatial contrast can effectively capture salient image features needed for classification using a Binarized DNN with significant reduction in input data usage (at least 12X) and memory resources (17.5X), compared to high precision RGB images and DNN, with only a small loss (~2%) in macro F1-score. Binarized MicronNet achieves an F1-score of 94.4% using spatial contrast, compared to only 56.3% when using RGB input images. Thus, SC offers great promise for deployment in power and resource constrained edge computing environments.