PyTorchKR Papers

“Image Classification”

총 18,403편
태그·기간 등 상세 필터

카테고리 탐색

arXiv:2205.00449

Molecular Identification from AFM images using the IUPAC Nomenclature and Attribute Multimodal Recurrent Neural Networks

Despite being the main tool to visualize molecules at the atomic scale, AFM with CO-functionalized metal tips is unable to chemically identify the observed molecules. Here we present a strategy to address this challenging task using deep learning techniques. Instead of identifying a finite number of molecules following a traditional classification approach, we define the molecular identification as an image captioning problem. We design an architecture, composed of two multimodal recurrent neural networks, capable of identifying the structure and composition of an unknown molecule using a 3D-AFM image stack as input. The neural network is trained to provide the name of each molecule according to the IUPAC nomenclature rules. To train and test this algorithm we use the novel QUAM-AFM dataset, which contains almost 700,000 molecules and 165 million AFM images. The accuracy of the predictions is remarkable, achieving a high score quantified by the cumulative BLEU 4-gram, a common metric in language recognition studies.

arXiv:2406.15035

Improving Interpretability and Robustness for the Detection of AI-Generated Images

With growing abilities of generative models, artificial content detection becomes an increasingly important and difficult task. However, all popular approaches to this problem suffer from poor generalization across domains and generative models. In this work, we focus on the robustness of AI-generated image (AIGI) detectors. We analyze existing state-of-the-art AIGI detection methods based on frozen CLIP embeddings and show how to interpret them, shedding light on how images produced by various AI generators differ from real ones. Next we propose two ways to improve robustness: based on removing harmful components of the embedding vector and based on selecting the best performing attention heads in the image encoder model. Our methods increase the mean out-of-distribution (OOD) classification score by up to 6% for cross-model transfer. We also propose a new dataset for AIGI detection and use it in our evaluation; we believe this dataset will help boost further research. The dataset and code are provided as a supplement.

arXiv:2503.18997

Improving Food Image Recognition with Noisy Vision Transformer

Food image recognition is a challenging task in computer vision due to the high variability and complexity of food images. In this study, we investigate the potential of Noisy Vision Transformers (NoisyViT) for improving food classification performance. By introducing noise into the learning process, NoisyViT reduces task complexity and adjusts the entropy of the system, leading to enhanced model accuracy. We fine-tune NoisyViT on three benchmark datasets: Food2K (2,000 categories, ~1M images), Food-101 (101 categories, ~100K images), and CNFOOD-241 (241 categories, ~190K images). The performance of NoisyViT is evaluated against state-of-the-art food recognition models. Our results demonstrate that NoisyViT achieves Top-1 accuracies of 95%, 99.5%, and 96.6% on Food2K, Food-101, and CNFOOD-241, respectively, significantly outperforming existing approaches. This study underscores the potential of NoisyViT for dietary assessment, nutritional monitoring, and healthcare applications, paving the way for future advancements in vision-based food computing. Code for reproducing NoisyViT for food recognition is available at NoisyViT_Food.

arXiv:2611.14436v1cs.CV

설명 가능한 딥페이크 탐지를 위한 지표 정렬 그룹 상대 정책 최적화

Metric-Aligned Group Relative Policy Optimization for Explainable Deepfake Detection

딥페이크 탐지기는 보통 진짜/가짜 분류만으로 평가되지만, 실제 응용에서는 각 예측의 근거가 되는 시각적 증거에 대한 설명이 필요하다. 결합형 비전-언어 모델은 진위 판단에 어려움을 겪으며, 조작된 이미지에 대해 "진짜"라는 서술을 생성한다. 이를 해결하기 위해 ACM Multimedia 2026 설명 가능한 딥페이크 탐지 챌린지에 대한 제출물로서, XPlainVerse 벤치마크에서 학습 및 평가한 분리형 예측 후 설명 파이프라인을 제안한다. 먼저 비전 파운데이션 모델 앙상블이 이미지를 분류하고, 이어서 Qwen3-VL-8B 모델이 시각적 증거에 대한 판정 조건부 설명을 생성한다. 가짜 이미지의 경우, 두 번째 Qwen3-VL-8B가 이미지를 사용하지 않고 해당 텍스트를 비전문가용 짧은 요약으로 압축한다. 또한 챌린지 기준에 맞춰 두 설명기를 모두 최적화하기 위해 지표 정렬 그룹 상대 정책 최적화(GRPO)를 적용한다. 동일한 조건에서 GRPO는 복합 BERTScore-F1을 0.686에서 0.712로 높이고 근거 제시(개체 및 증거 F1)도 개선하며, 우리의 제출물은 복합 BERTScore-F1에서 1위, 종합 7위를 기록했다.

arXiv:1907.10346

Delving Deep into Liver Focal Lesion Detection: A Preliminary Study

Hepatocellular carcinoma (HCC) is the second most frequent cause of malignancy-related death and is one of the diseases with the highest incidence in the world. Because the liver is the only organ in the human body that is supplied by two major vessels: the hepatic artery and the portal vein, various types of malignant tumors can spread from other organs to the liver. And due to the liver masses' heterogeneous and diffusive shape, the tumor lesions are very difficult to be recognized, thus automatic lesion detection is necessary for the doctors with huge workloads. To assist doctors, this work uses the existing large-scale annotation medical image data to delve deep into liver lesion detection from multiple directions. To solve technical difficulties, such as the image-recognition task, traditional deep learning with convolution neural networks (CNNs) has been widely applied in recent years. However, this kind of neural network, such as Faster Regions with CNN features (R-CNN), cannot leverage the spatial information because it is applied in natural images (2D) rather than medical images (3D), such as computed tomography (CT) images. To address this issue, we propose a novel algorithm that is appropriate for liver CT imaging. Furthermore, according to radiologists' experience in clinical diagnosis and the characteristics of CT images of liver cancer, a liver cancer-detection framework with CNN, including image processing, feature extraction, region proposal, image registration, and classification recognition, was proposed to facilitate the effective detection of liver lesions.

arXiv:2004.03366

Knife and Threat Detectors

Despite rapid advances in image-based machine learning, the threat identification of a knife wielding attacker has not garnered substantial academic attention. This relative research gap appears less understandable given the high knife assault rate (>100,000 annually) and the increasing availability of public video surveillance to analyze and forensically document. We present three complementary methods for scoring automated threat identification using multiple knife image datasets, each with the goal of narrowing down possible assault intentions while minimizing misidentifying false positives and risky false negatives. To alert an observer to the knife-wielding threat, we test and deploy classification built around MobileNet in a sparse and pruned neural network with a small memory requirement (< 2.2 megabytes) and 95% test accuracy. We secondly train a detection algorithm (MaskRCNN) to segment the hand from the knife in a single image and assign probable certainty to their relative location. This segmentation accomplishes both localization with bounding boxes but also relative positions to infer overhand threats. A final model built on the PoseNet architecture assigns anatomical waypoints or skeletal features to narrow the threat characteristics and reduce misunderstood intentions. We further identify and supplement existing data gaps that might blind a deployed knife threat detector such as collecting innocuous hand and fist images as important negative training sets. When automated on commodity hardware and software solutions one original research contribution is this systematic survey of timely and readily available image-based alerts to task and prioritize crime prevention countermeasures prior to a tragic outcome.

arXiv:2303.17644

Vision-Language Modelling For Radiological Imaging and Reports In The Low Data Regime

This paper explores training medical vision-language models (VLMs) -- where the visual and language inputs are embedded into a common space -- with a particular focus on scenarios where training data is limited, as is often the case in clinical datasets. We explore several candidate methods to improve low-data performance, including: (i) adapting generic pre-trained models to novel image and text domains (i.e. medical imaging and reports) via unimodal self-supervision; (ii) using local (e.g. GLoRIA) & global (e.g. InfoNCE) contrastive loss functions as well as a combination of the two; (iii) extra supervision during VLM training, via: (a) image- and text-only self-supervision, and (b) creating additional positive image-text pairs for training through augmentation and nearest-neighbour search. Using text-to-image retrieval as a benchmark, we evaluate the performance of these methods with variable sized training datasets of paired chest X-rays and radiological reports. Combined, they significantly improve retrieval compared to fine-tuning CLIP, roughly equivalent to training with the data. A similar pattern is found in the downstream task classification of CXR-related conditions with our method outperforming CLIP and also BioVIL, a strong CXR VLM benchmark, in the zero-shot and linear probing settings. We conclude with a set of recommendations for researchers aiming to train vision-language models on other medical imaging modalities when training data is scarce. To facilitate further research, we will make our code and models publicly available.

arXiv:2312.12481

Unveiling Spaces: Architecturally meaningful semantic descriptions from images of interior spaces

There has been a growing adoption of computer vision tools and technologies in architectural design workflows over the past decade. Notable use cases include point cloud generation, visual content analysis, and spatial awareness for robotic fabrication. Multiple image classification, object detection, and semantic pixel segmentation models have become popular for the extraction of high-level symbolic descriptions and semantic content from two-dimensional images and videos. However, a major challenge in this regard has been the extraction of high-level architectural structures (walls, floors, ceilings windows etc.) from diverse imagery where parts of these elements are occluded by furniture, people, or other non-architectural elements. This project aims to tackle this problem by proposing models that are capable of extracting architecturally meaningful semantic descriptions from two-dimensional scenes of populated interior spaces. 1000 virtual classrooms are parametrically generated, randomized along key spatial parameters such as length, width, height, and door/window positions. The positions of cameras, and non-architectural visual obstructions (furniture/objects) are also randomized. A Generative Adversarial Network (GAN) for image-to-image translation (Pix2Pix) is trained on synthetically generated rendered images of these enclosures, along with corresponding image abstractions representing high-level architectural structure. The model is then tested on unseen synthetic imagery of new enclosures, and outputs are compared to ground truth using pixel-wise comparison for evaluation. A similar model evaluation is also carried out on photographs of existing indoor enclosures, to measure its performance in real-world settings.

arXiv:2112.06806

Automatic Multi-Class Cardiovascular Magnetic Resonance Image Quality Assessment using Unsupervised Domain Adaptation in Spatial and Frequency Domains

Population imaging studies rely upon good quality medical imagery before downstream image quantification. This study provides an automated approach to assess image quality from cardiovascular magnetic resonance (CMR) imaging at scale. We identify four common CMR imaging artefacts, including respiratory motion, cardiac motion, Gibbs ringing, and aliasing. The model can deal with images acquired in different views, including two, three, and four-chamber long-axis and short-axis cine CMR images. Two deep learning-based models in spatial and frequency domains are proposed. Besides recognising these artefacts, the proposed models are suitable to the common challenges of not having access to data labels. An unsupervised domain adaptation method and a Fourier-based convolutional neural network are proposed to overcome these challenges. We show that the proposed models reliably allow for CMR image quality assessment. The accuracies obtained for the spatial model in supervised and weakly supervised learning are 99.41+0.24 and 96.37+0.66 for the UK Biobank dataset, respectively. Using unsupervised domain adaptation can somewhat overcome the challenge of not having access to the data labels. The maximum achieved domain gap coverage in unsupervised domain adaptation is 16.86%. Domain adaptation can significantly improve a 5-class classification task and deal with considerable domain shift without data labels. Increasing the speed of training and testing can be achieved with the proposed model in the frequency domain. The frequency-domain model can achieve the same accuracy yet 1.548 times faster than the spatial model. This model can also be used directly on k-space data, and there is no need for image reconstruction.

arXiv:2607.09305v2cs.CV

분류에서 위치 추정 및 임상 검증까지: 태국 흉부 방사선 사진에서 흉부 질환 탐지를 위한 심층 학습 시스템의 대규모 개발

From Classification to Localization and Clinical Validation: Large-Scale Development of a Deep Learning System for Thoracic Disease Detection on Chest Radiographs in Thailand

흉부 방사선 촬영(CXR)은 가장 널리 사용되는 흉부 영상 기법이지만, 태국과 동남아시아 전반에 걸친 심각한 방사선 전문의 부족으로 인해 전문가 판독에 제약이 있다. 태국 데이터에 대한 심층 학습 모델의 현지 적응이 태국 인구에서의 정확도를 실질적으로 향상시키는 것으로 나타났다. 본 연구에서는 단일 모델 내에서 다중 레이블 흉부 질환 분류와 약지도 병변 위치 추정을 수행하는 심층 학습 시스템인 Inspectra CXR 버전 5의 흉부 방사선 사진 분석 모델의 개발 및 종합 검증을 제시한다. 이 아키텍처는 DenseNet-121 백본에 Attend-and-Compare Modules(ACM)와 Probabilistic Class Activation Map(PCAM) 집계 층을 결합하여, 질환별 분류 점수와 히트맵을 동시에 생성한다. 모델은 방콕 시리라즈 병원의 방사선 전문의 보고서가 쌍으로 연결된 874,858장의 정면 흉부 방사선 사진으로 개발되었다. 방사선 전문의가 검증한 19,871건의 도메인 내 테스트 세트에서 임상적으로 중요한 9가지 질환에 걸쳐 평균 AUROC 0.994(평균 민감도 92.4%, 특이도 98.6%)를 달성하였다. 태국 전역 13개 병원의 5,992건으로 구성된 독립적인 일반화 세트에서는 평균 AUROC 0.970을 기록하여, 기관 간 강건한 전이 능력을 입증하였다. 방사선 전문의가 주석을 단 4,549건을 대상으로 평가한 위치 추정에서는 이미지당 0.59건의 비병변 위치 추정을 허용하는 조건에서 평균 병변 위치 추정 비율(LLF) 77.9%를 달성하였다. 흉부 방사선 전문의 5명과의 사용성 평가에서 시스템은 분류 일치율 93.6%, 위치 추정 일치율 94.7%, 평균 System Usability Scale(SUS) 점수 89를 기록하였다. 이러한 결과는 현지 개발된 위치 추정 가능 CXR 시스템이 높은 정확도를 제공하고, 이질적인 태국 병원 환경에서 일반화되며, 임상 방사선 전문의의 신뢰를 얻을 수 있음을 보여준다.

arXiv:1804.11214

k-Nearest Neighbors by Means of Sequence to Sequence Deep Neural Networks and Memory Networks

k-Nearest Neighbors is one of the most fundamental but effective classification models. In this paper, we propose two families of models built on a sequence to sequence model and a memory network model to mimic the k-Nearest Neighbors model, which generate a sequence of labels, a sequence of out-of-sample feature vectors and a final label for classification, and thus they could also function as oversamplers. We also propose 'out-of-core' versions of our models which assume that only a small portion of data can be loaded into memory. Computational experiments show that our models on structured datasets outperform k-Nearest Neighbors, a feed-forward neural network, XGBoost, lightGBM, random forest and a memory network, due to the fact that our models must produce additional output and not just the label. On image and text datasets, the performance of our model is close to many state-of-the-art deep models. As an oversampler on imbalanced datasets, the sequence to sequence kNN model often outperforms Synthetic Minority Over-sampling Technique and Adaptive Synthetic Sampling.

arXiv:1707.09457공식 코드 · Papers with Code

Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints

Language is increasingly being used to define rich visual recognition problems with supporting image collections sourced from the web. Structured prediction models are used in these tasks to take advantage of correlations between co-occurring labels and visual input but risk inadvertently encoding social biases found in web corpora. In this work, we study data and models associated with multilabel object classification and visual semantic role labeling. We find that (a) datasets for these tasks contain significant gender bias and (b) models trained on these datasets further amplify existing bias. For example, the activity cooking is over 33% more likely to involve females than males in a training set, and a trained model further amplifies the disparity to 68% at test time. We propose to inject corpus-level constraints for calibrating existing structured prediction models and design an algorithm based on Lagrangian relaxation for collective inference. Our method results in almost no performance loss for the underlying recognition task but decreases the magnitude of bias amplification by 47.5% and 40.5% for multilabel classification and visual semantic role labeling, respectively.

arXiv:1808.02061

Semblance: A Rank-Based Kernel on Probability Spaces for Niche Detection

In data science, determining proximity between observations is critical to many downstream analyses such as clustering, information retrieval and classification. However, when the underlying structure of the data probability space is unclear, the function used to compute similarity between data points is often arbitrarily chosen. Here, we present a novel concept of proximity, Semblance, that uses the empirical distribution across all observations to inform the similarity between each pair. The advantage of Semblance lies in its distribution free formulation and its ability to detect niche features by placing greater emphasis on similarity between observation pairs that fall at the outskirts of the data distribution, as opposed to those that fall towards the center. We prove that Semblance is a valid Mercer kernel, thus allowing its principled use in kernel based learning machines. Semblance can be applied to any data modality, and we demonstrate its consistently improved performance against conventional methods through simulations and three real case studies from very different applications, viz. cell type classification using single cell RNA sequencing, selecting predictors of positive return on real estate investments, and image compression.

arXiv:1906.02896공식 코드 · Papers with Code

Adversarial Explanations for Understanding Image Classification Decisions and Improved Neural Network Robustness

For sensitive problems, such as medical imaging or fraud detection, Neural Network (NN) adoption has been slow due to concerns about their reliability, leading to a number of algorithms for explaining their decisions. NNs have also been found vulnerable to a class of imperceptible attacks, called adversarial examples, which arbitrarily alter the output of the network. Here we demonstrate both that these attacks can invalidate prior attempts to explain the decisions of NNs, and that with very robust networks, the attacks themselves may be leveraged as explanations with greater fidelity to the model. We show that the introduction of a novel regularization technique inspired by the Lipschitz constraint, alongside other proposed improvements, greatly improves an NN's resistance to adversarial examples. On the ImageNet classification task, we demonstrate a network with an Accuracy-Robustness Area (ARA) of 0.0053, an ARA 2.4x greater than the previous state of the art. Improving the mechanisms by which NN decisions are understood is an important direction for both establishing trust in sensitive domains and learning more about the stimuli to which NNs respond.

arXiv:1906.11904

Effective degrees of freedom for surface finish defect detection and classification

One of the primary concerns of product quality control in the automotive industry is an automated detection of defects of small sizes on specular car body surfaces. A new statistical learning approach is presented for surface finish defect detection based on spline smoothing method for feature extraction and $k$-nearest neighbour probabilistic classifier. Since the surfaces are specular, structured lightning reflection technique is applied for image acquisition. Reduced rank cubic regression splines are used to smooth the pixel values while the effective degrees of freedom of the obtained smooths serve as components of the feature vector. A key advantage of the approach is that it allows reaching near zero misclassification error rate when applying standard learning classifiers. We also propose probability based performance evaluation metrics as alternatives to the conventional metrics. The usage of those provides the means for uncertainty estimation of the predictive performance of a classifier. Experimental classification results on the images obtained from the pilot system located at Volvo GTO Cab plant in Ume{\aa}, Sweden, show that the proposed approach is much more efficient than the compared methods.

arXiv:2010.13737공식 코드 · Papers with Code

Real-Time Edge Classification: Optimal Offloading under Token Bucket Constraints

To deploy machine learning-based algorithms for real-time applications with strict latency constraints, we consider an edge-computing setting where a subset of inputs are offloaded to the edge for processing by an accurate but resource-intensive model, and the rest are processed only by a less-accurate model on the device itself. Both models have computational costs that match available compute resources, and process inputs with low-latency. But offloading incurs network delays, and to manage these delays to meet application deadlines, we use a token bucket to constrain the average rate and burst length of transmissions from the device. We introduce a Markov Decision Process-based framework to make offload decisions under these constraints, based on the local model's confidence and the token bucket state, with the goal of minimizing a specified error measure for the application. Beyond isolated decisions for individual devices, we also propose approaches to allow multiple devices connected to the same access switch to share their bursting allocation. We evaluate and analyze the policies derived using our framework on the standard ImageNet image classification benchmark.

arXiv:2012.01988

Wisdom of Committees: An Overlooked Approach To Faster and More Accurate Models

Committee-based models (ensembles or cascades) construct models by combining existing pre-trained ones. While ensembles and cascades are well-known techniques that were proposed before deep learning, they are not considered a core building block of deep model architectures and are rarely compared to in recent literature on developing efficient models. In this work, we go back to basics and conduct a comprehensive analysis of the efficiency of committee-based models. We find that even the most simplistic method for building committees from existing, independently pre-trained models can match or exceed the accuracy of state-of-the-art models while being drastically more efficient. These simple committee-based models also outperform sophisticated neural architecture search methods (e.g., BigNAS). These findings hold true for several tasks, including image classification, video classification, and semantic segmentation, and various architecture families, such as ViT, EfficientNet, ResNet, MobileNetV2, and X3D. Our results show that an EfficientNet cascade can achieve a 5.4x speedup over B7 and a ViT cascade can achieve a 2.3x speedup over ViT-L-384 while being equally accurate.

arXiv:2104.09375공식 코드 · Papers with Code

A Multi-Task Deep Learning Framework for Building Footprint Segmentation

The task of building footprint segmentation has been well-studied in the context of remote sensing (RS) as it provides valuable information in many aspects, however, difficulties brought by the nature of RS images such as variations in the spatial arrangements and in-consistent constructional patterns require studying further, since it often causes poorly classified segmentation maps. We address this need by designing a joint optimization scheme for the task of building footprint delineation and introducing two auxiliary tasks; image reconstruction and building footprint boundary segmentation with the intent to reveal the common underlying structure to advance the classification accuracy of a single task model under the favor of auxiliary tasks. In particular, we propose a deep multi-task learning (MTL) based unified fully convolutional framework which operates in an end-to-end manner by making use of joint loss function with learnable loss weights considering the homoscedastic uncertainty of each task loss. Experimental results conducted on the SpaceNet6 dataset demonstrate the potential of the proposed MTL framework as it improves the classification accuracy considerably compared to single-task and lesser compounded tasks.

arXiv:2111.14680

Graph Embedding via High Dimensional Model Representation for Hyperspectral Images

Learning the manifold structure of remote sensing images is of paramount relevance for modeling and understanding processes, as well as to encapsulate the high dimensionality in a reduced set of informative features for subsequent classification, regression, or unmixing. Manifold learning methods have shown excellent performance to deal with hyperspectral image (HSI) analysis but, unless specifically designed, they cannot provide an explicit embedding map readily applicable to out-of-sample data. A common assumption to deal with the problem is that the transformation between the high-dimensional input space and the (typically low) latent space is linear. This is a particularly strong assumption, especially when dealing with hyperspectral images due to the well-known nonlinear nature of the data. To address this problem, a manifold learning method based on High Dimensional Model Representation (HDMR) is proposed, which enables to present a nonlinear embedding function to project out-of-sample samples into the latent space. The proposed method is compared to manifold learning methods along with its linear counterparts and achieves promising performance in terms of classification accuracy of a representative set of hyperspectral images.

arXiv:2202.00678

Classification of Skin Cancer Images using Convolutional Neural Networks

Skin cancer is the most common human malignancy(American Cancer Society) which is primarily diagnosed visually, starting with an initial clinical screening and followed potentially by dermoscopic(related to skin) analysis, a biopsy and histopathological examination. Skin cancer occurs when errors (mutations) occur in the DNA of skin cells. The mutations cause the cells to grow out of control and form a mass of cancer cells. The aim of this study was to try to classify images of skin lesions with the help of convolutional neural networks. The deep neural networks show humongous potential for image classification while taking into account the large variability exhibited by the environment. Here we trained images based on the pixel values and classified them on the basis of disease labels. The dataset was acquired from an Open Source Kaggle Repository(Kaggle Dataset)which itself was acquired from ISIC(International Skin Imaging Collaboration) Archive. The training was performed on multiple models accompanied with Transfer Learning. The highest model accuracy achieved was over 86.65%. The dataset used is publicly available to ensure credibility and reproducibility of the aforementioned result.