PyTorchKR Papers

“Image Classification”

총 18,403편
태그·기간 등 상세 필터

카테고리 탐색

arXiv:2611.17176v1cs.CV

차분 모핑 공격 탐지를 위한 검증 기반 프롬프트 선택

Validation-Guided Prompt Selection for Differential Morphing Attack Detection

차분 모핑 공격 탐지(D-MAD)는 신뢰할 수 있는 실시간 촬영본을 얼굴 모프가 포함되었을 수 있는 문서 이미지와 비교한다. 우리는 레이블이 있는 검증 데이터가 모델 미세 조정이 아닌 프롬프트 선택을 통해 매개변수가 고정된 비전-언어 평가자를 개선할 수 있는지 조사한다. 버전 1은 50개의 후보 프롬프트를 생성하고, 놓친 모프 공격(MACER)을 우선시하면서 정상 분류 오류(BSCER), 응답 실패, 파싱 실패, 판정 불일치를 고려하는 보안 지향 손실을 사용하여 상위 순위의 지시문을 선택한다. 버전 2는 집계된 검증 피드백을 이용해 프롬프트를 재귀적으로 수정한다. LMA, MIPGAN-2, PIPE를 아우르는 피험자 분리 테스트 세트 160쌍에서, 선택된 버전 1 프롬프트는 모프 공격 분류 오류율(MACER) 19.17\%를 달성하였으며 관측된 정상 샘플 분류 오류율(BSCER)은 0.00\%였고, 통제된 수동 프롬프트의 MACER는 70.00\%였다. 대응 표본의 50.83 퍼센트포인트 차이는 95\% 피험자 부트스트랩 구간이 [43.33,58.33]이다. 재귀적 수정은 최상의 0세대 검증 손실을 넘어서지 못했으며, 보고된 수정 후손의 테스트 MACER는 31.67\%이다. 버전 1이 놓친 23개 공격은 모두 랜드마크 기반(LMA) 모프이다. 전반적으로 이 결과는 프롬프트 구성이 매개변수가 고정된 VLM 기반 D-MAD 성능에 상당한 영향을 줄 수 있음을 보여 주는 동시에, 10명의 피험자로 이루어진 소규모 검증 환경에서 재귀적 프롬프트 개선이 어렵다는 점을 부각한다.

arXiv:2607.28005v1cs.CV

카메라 트랩 이미지를 활용한 심층 학습 기반 계층적 곤충 분류

Deep learning-based hierarchical insect classification using camera trap imagery

곤충 개체 수의 감소로 신뢰할 수 있는 생물 다양성 모니터링의 필요성이 더욱 시급해지고 있으나, 곤충 생물 다양성 모니터링은 표준화된 데이터의 부재와 곤충학 전문가의 비용과 시간이 많이 소요되는 수작업 식별로 인해 제약을 받고 있습니다. 자동화된 비치명적 카메라 트랩의 데이터를 처리하는 심층 학습(deep learning) 기반 이미지 분류기는 곤충 생물 다양성 모니터링을 혁신하고 확장할 수 있는 잠재력을 지니고 있습니다. 그러나 전문가가 라벨링한 데이터셋의 확보, 다양한 분류학적 수준에 걸쳐 우수하게 일반화되는 모델 아키텍처 개발, 극도로 불균형한 데이터에 대한 모델 학습 등 여러 과제가 여전히 남아 있습니다. 또한 계층적 데이터는 더 세부적인 분류학적 수준이 불확실할 때 신뢰도가 더 높은 조잡한(상위) 수준의 예측으로 기본 설정되도록 모델을 설계함으로써 이점을 얻을 수 있습니다. 본 논문에서는 심층 학습 기반의 계층적 분류 모델을 통해 이러한 문제들을 해결합니다. 먼저, 1,801개의 카메라 트랩 비디오 녹화물에서 추출하고 5개 계층 수준 및 34개 클래스로 라벨링한 약 100만 장의 곤충 이미지로 구성된 수작업 큐레이션된 롱테일(long-tailed) 데이터셋을 제시합니다. 나아가, 클래스 균형 가중치를 적용하여 5개 수준의 가변 깊이 계층 구조에 맞춘 계층적 분류 모델 아키텍처를 도입합니다. 제안하는 모델은 생물학적 분류 체계를 활용하여 특정 세분성(granularity)별 시각적 특징을 추출함으로써 비계층적 분류기에 비해 성능을 향상시키며, 신뢰도 임계값(T = 0.6)을 충족하는 가장 깊은 분류학적 수준까지 계층적으로 일관된 예측을 수행합니다. 우리 모델은 테스트 데이터의 5개 계층 수준 전반에서 수준별 80~99%의 정확도를 달성했습니다. 나아가 ...

arXiv:2610.13675v1eess.IV

멀티모달 의사결정 모델이 유방 초음파 이미지를 분류할 수 있는가? 프롬프트와 데이터셋 전반에 걸친 CLEF 및 CLEF-Flash 벤치마킹

Can Multimodal Decision Models Classify Breast Ultrasound Images? Benchmarking CLEF and CLEF-Flash Across Prompts and Datasets

멀티모달 의사결정 모델은 사전에 정의된 대안들에 대한 확률을 반환하여 이미지 분류를 위한 구조화된 인터페이스를 제공한다. 본 연구는 Dataset B, BUSI, BUS-BRA에서 수집한 2818개의 유방 초음파 이미지에 대해 CLEF 및 CLEF-Flash를 사용한 추론을 벤치마킹한다. Dataset B와 BUS-BRA는 이진 양성-악성 분류를 사용하는 반면, BUSI는 정상, 양성, 악성 범주를 포함한다. 모델 파라미터는 변경하지 않은 채 유지되며, 원본 이미지는 사용자 정의 전처리 없이 입력된다. 세 가지 프롬프트는 최소한의 라벨, 명시적인 분류 태스크, 또는 초음파 검사 기준을 제공한다. 본 연구에서는 정확도, 매크로 F1, 균형 정확도, 매크로 정밀도, 처리량, 지연 시간을 보고한다. 임상 프롬프트를 적용한 CLEF-Flash는 세 데이터셋 모두에서 가장 높은 정확도를 달성하여 Dataset B에서 74.85%, BUSI에서 66.28%, BUS-BRA에서 51.63%를 기록했다. 그러나 BUSI에서는 태스크 유도 프롬프트를 적용한 CLEF가 가장 높은 매크로 F1(64.51%)과 균형 정확도(68.72%)를 달성했다. CLEF-Flash는 일치 조건으로 비교한 모든 경우에서 더 낮은 지연 시간을 보였다. 임상 지침 프롬프트는 최소 프롬프트 대비 CLEF-Flash의 정확도를 향상시켰으나, 모든 데이터셋에서 CLEF의 정확도는 감소시켰다. 혼동 행렬을 통해 양성을 악성으로 오분류하는 오류가 빈번함을 확인했으며, 특히 BUS-BRA에서는 모든 구성이 다수 클래스 예측기의 정확도인 67.63%에 미치지 못했다. 이러한 결과는 성능이 데이터셋과 프롬프트에 따라 달라짐을 보여주며, 유방 초음파 분석에 멀티모달 의사결정 모델을 적용하기 전에 클래스별 평가가 선행되어야 함을 시사한다.

arXiv:2001.06372

BigEarthNet Dataset with A New Class-Nomenclature for Remote Sensing Image Understanding

This paper presents BigEarthNet that is a large-scale Sentinel-2 multispectral image dataset with a new class nomenclature to advance deep learning (DL) studies in remote sensing (RS). BigEarthNet is made up of 590,326 image patches annotated with multi-labels provided by the CORINE Land Cover (CLC) map of 2018 based on its most thematic detailed Level-3 class nomenclature. Initial research demonstrates that some CLC classes are challenging to be accurately described by considering only Sentinel-2 images. To increase the effectiveness of BigEarthNet, in this paper we introduce an alternative class-nomenclature to allow DL models for better learning and describing the complex spatial and spectral information content of the Sentinel-2 images. This is achieved by interpreting and arranging the CLC Level-3 nomenclature based on the properties of Sentinel-2 images in a new nomenclature of 19 classes. Then, the new class-nomenclature of BigEarthNet is used within state-of-the-art DL models in the context of multi-label classification. Results show that the models trained from scratch on BigEarthNet outperform those pre-trained on ImageNet, especially in relation to some complex classes including agriculture, other vegetated and natural environments. All DL models are made publicly available at http://bigearth.net/#downloads, offering an important resource to guide future progress on RS image analysis.

arXiv:2111.05684공식 코드 · Papers with Code

Learning to ignore: rethinking attention in CNNs

Recently, there has been an increasing interest in applying attention mechanisms in Convolutional Neural Networks (CNNs) to solve computer vision tasks. Most of these methods learn to explicitly identify and highlight relevant parts of the scene and pass the attended image to further layers of the network. In this paper, we argue that such an approach might not be optimal. Arguably, explicitly learning which parts of the image are relevant is typically harder than learning which parts of the image are less relevant and, thus, should be ignored. In fact, in vision domain, there are many easy-to-identify patterns of irrelevant features. For example, image regions close to the borders are less likely to contain useful information for a classification task. Based on this idea, we propose to reformulate the attention mechanism in CNNs to learn to ignore instead of learning to attend. Specifically, we propose to explicitly learn irrelevant information in the scene and suppress it in the produced representation, keeping only important attributes. This implicit attention scheme can be incorporated into any existing attention mechanism. In this work, we validate this idea using two recent attention methods Squeeze and Excitation (SE) block and Convolutional Block Attention Module (CBAM). Experimental results on different datasets and model architectures show that learning to ignore, i.e., implicit attention, yields superior performance compared to the standard approaches.

arXiv:2304.11592

Broken Rail Detection With Texture Image Processing Using Two-Dimensional Gray Level Co-occurrence Matrix

Application of electronic railway systems as well as the implication of Automatic Train Control (ATC) System has increased the safety of rail transportation. However, one of the most important causes of accidents on the railway is rail damage and breakage. In this paper, we have proposed a method that the rail region is first recognized from the observation area, then by investigating the image texture processing data, the types of rail defects including cracks, wear, peeling, disintegration, and breakage are detected. In order to reduce the computational cost, the image is changed from the RGB color spectrum to the gray spectrum. Image texture processing data is obtained by the two-dimensional Gray Levels Co-occurrence Matrix (GLCM) at different angles; this data demonstrates second-order features of the images. Large data of features has a negative effect on the overall accuracy of the classifiers. To tackle this issue and acquire faster response, Principal Component Analysis (PCA) algorithm is used, before entering the band into the classifier. Then the features extracted from the images are compared by three different classifiers including Support Vector Machine (SVM), Random Forest (RF), and K-Nearest Neighbor (KNN) classification. The results obtained from this method indicate that the Random Forest classifier has better performance (accuracy 97%, precision 96%, and recall 96%) than other classifiers.

arXiv:2608.00361v1cs.CV

광학 현미경을 통한 입자 및 섬유 특성 분석을 위한 인공지능

Artificial Intelligence for the Characterization of Particles and Fibers by Optical Microscopy

입자 및 섬유 분산액의 광학 현미경 검사는 시료의 형태, 화학적 조성, 배율 및 조명 조건의 영향을 받는 미세한 시각적 단서를 해석하는 과정을 수반합니다. 본 연구에서는 의미론적 앵커(semantic anchor)를 사용하여 현미경 이미지로부터 의미론적으로 풍부한 이미지 임베딩을 추출하는 인공지능(AI) 증류(distillation) 프레임워크를 소개합니다. 멀티모달 티처(multimodal teacher)는 각 이미지의 시각적 임베딩을 조명 모달리티, 배율, 시료의 정체성 및 형태를 나타내는 세 가지 텍스트 임베딩과 결합합니다. LongCLIP의 확장된 컨텍스트 텍스트 인코더로 생성된 이 벡터는 2304차원의 블록 구조화된 티처 벡터를 형성하며, 그 구성 블록들은 학습 및 추론 전반에 걸쳐 물리적으로 해석 가능한 상태를 유지합니다. 다층 퍼셉트론(MLP) 디코더를 갖춘 스튜던트 비전 트랜스포머(ViT)는 이미지만으로 이 티처 벡터를 재구성하도록 학습되며, 티처의 블록 구조에 대한 좌표 수준의 충실도를 강제하는 평균 절대 오차(L1) 손실을 최소화합니다. 티처 임베딩 공간의 HDBSCAN 클러스터링에서 파생된 의사 클래스(pseudo-class)에 대한 교차 엔트로피 항은 붕괴 방지 정규화기 역할을 하여, 대조적 네거티브 마이닝 없이도 클러스터 간 분리를 강제합니다. 추론 시 스튜던트는 이미지 입력만으로 작동하여 티처 벡터의 전체 의미론적 내용을 복원하는 컴팩트한 임베딩을 생성합니다. 이 프레임워크는 leave-one-out 최근접 이웃 검색 하에서 세분화된 시료 설명 레이블에 대해 약 80%의 의사 클래스 검증 정확도와 75%의 Recall@1(재현율@1)을 달성합니다. 이러한 결과는 의미론적 앵커링을 통해 비전 전용 스튜던트가 이미지 전용 학습보다 더 풍부하고 해석 가능한 표현을 획득할 수 있음을 보여주며, 이질적인 입자 및 섬유 분산액의 검색, 분류 및 탐색적 분석에 직접 적용할 수 있습니다.

arXiv:1703.10217

Smartphone Based Colorimetric Detection via Machine Learning

We report the application of machine learning to smartphone based colorimetric detection of pH values. The strip images were used as the training set for Least Squares-Support Vector Machine (LS-SVM) classifier algorithms that were able to successfully classify the distinct pH values. The difference in the obtained image formats was found not to significantly affect the performance of the proposed machine learning approach. Moreover, the influence of the illumination conditions on the perceived color of pH strips was investigated and further experiments were carried out to study effect of color change on the learning model. Test results on JPEG, RAW and RAW-corrected image formats captured in different lighting conditions lead to perfect classification accuracy, sensitivity and specificity, which proves that the colorimetric detection using machine learning based systems is able to adapt to various experimental conditions and is a great candidate for smartphone based sensing in paper-based colorimetric assays.

arXiv:1511.01804

Wood Species Recognition Based on SIFT Keypoint Histogram

Traditionally, only experts who are equipped with professional knowledge and rich experience are able to recognize different species of wood. Applying image processing techniques for wood species recognition can not only reduce the expense to train qualified identifiers, but also increase the recognition accuracy. In this paper, a wood species recognition technique base on Scale Invariant Feature Transformation (SIFT) keypoint histogram is proposed. We use first the SIFT algorithm to extract keypoints from wood cross section images, and then k-means and k-means++ algorithms are used for clustering. Using the clustering results, an SIFT keypoints histogram is calculated for each wood image. Furthermore, several classification models, including Artificial Neural Networks (ANN), Support Vector Machine (SVM) and K-Nearest Neighbor (KNN) are used to verify the performance of the method. Finally, through comparing with other prevalent wood recognition methods such as GLCM and LBP, results show that our scheme achieves higher accuracy.

arXiv:1405.4930

Adapted Approach for Fruit Disease Identification using Images

Diseases in fruit cause devastating problem in economic losses and production in agricultural industry worldwide. In this paper, an adaptive approach for the identification of fruit diseases is proposed and experimentally validated. The image processing based proposed approach is composed of the following main steps; in the first step K-Means clustering technique is used for the defect segmentation, in the second step some state of the art features are extracted from the segmented image, and finally images are classified into one of the classes by using a Multi-class Support Vector Machine. We have considered diseases of apple as a test case and evaluated our approach for three types of apple diseases namely apple scab, apple blotch and apple rot. Our experimental results express that the proposed solution can significantly support accurate detection and automatic identification of fruit diseases. The classification accuracy for the proposed solution is achieved up to 93%.

arXiv:1906.06148공식 코드 · Papers with Code

A Partially Reversible U-Net for Memory-Efficient Volumetric Image Segmentation

One of the key drawbacks of 3D convolutional neural networks for segmentation is their memory footprint, which necessitates compromises in the network architecture in order to fit into a given memory budget. Motivated by the RevNet for image classification, we propose a partially reversible U-Net architecture that reduces memory consumption substantially. The reversible architecture allows us to exactly recover each layer's outputs from the subsequent layer's ones, eliminating the need to store activations for backpropagation. This alleviates the biggest memory bottleneck and enables very deep (theoretically infinitely deep) 3D architectures. On the BraTS challenge dataset, we demonstrate substantial memory savings. We further show that the freed memory can be used for processing the whole field-of-view (FOV) instead of patches. Increasing network depth led to higher segmentation accuracy while growing the memory footprint only by a very small fraction, thanks to the partially reversible architecture.

arXiv:2004.13453공식 코드 · Papers with Code

DRU-net: An Efficient Deep Convolutional Neural Network for Medical Image Segmentation

Residual network (ResNet) and densely connected network (DenseNet) have significantly improved the training efficiency and performance of deep convolutional neural networks (DCNNs) mainly for object classification tasks. In this paper, we propose an efficient network architecture by considering advantages of both networks. The proposed method is integrated into an encoder-decoder DCNN model for medical image segmentation. Our method adds additional skip connections compared to ResNet but uses significantly fewer model parameters than DenseNet. We evaluate the proposed method on a public dataset (ISIC 2018 grand-challenge) for skin lesion segmentation and a local brain MRI dataset. In comparison with ResNet-based, DenseNet-based and attention network (AttnNet) based methods within the same encoder-decoder network structure, our method achieves significantly higher segmentation accuracy with fewer number of model parameters than DenseNet and AttnNet. The code is available on GitHub (GitHub link: https://github.com/MinaJf/DRU-net).

arXiv:2104.03488

Deep Features for training Support Vector Machine

Features play a crucial role in computer vision. Initially designed to detect salient elements by means of handcrafted algorithms, features are now often learned by different layers in Convolutional Neural Networks (CNNs). This paper develops a generic computer vision system based on features extracted from trained CNNs. Multiple learned features are combined into a single structure to work on different image classification tasks. The proposed system was experimentally derived by testing several approaches for extracting features from the inner layers of CNNs and using them as inputs to SVMs that are then combined by sum rule. Dimensionality reduction techniques are used to reduce the high dimensionality of inner layers. The resulting vision system is shown to significantly boost the performance of standard CNNs across a large and diverse collection of image data sets. An ensemble of different topologies using the same approach obtains state-of-the-art results on a virus data set.

arXiv:2201.12212공식 코드 · Papers with Code

Möbius Convolutions for Spherical CNNs

M\"obius transformations play an important role in both geometry and spherical image processing - they are the group of conformal automorphisms of 2D surfaces and the spherical equivalent of homographies. Here we present a novel, M\"obius-equivariant spherical convolution operator which we call M\"obius convolution, and with it, develop the foundations for M\"obius-equivariant spherical CNNs. Our approach is based on a simple observation: to achieve equivariance, we only need to consider the lower-dimensional subgroup which transforms the positions of points as seen in the frames of their neighbors. To efficiently compute M\"obius convolutions at scale we derive an approximation of the action of the transformations on spherical filters, allowing us to compute our convolutions in the spectral domain with the fast Spherical Harmonic Transform. The resulting framework is both flexible and descriptive, and we demonstrate its utility by achieving promising results in both shape classification and image segmentation tasks.

arXiv:2201.12078공식 코드 · Papers with Code

You Only Cut Once: Boosting Data Augmentation with a Single Cut

We present You Only Cut Once (YOCO) for performing data augmentations. YOCO cuts one image into two pieces and performs data augmentations individually within each piece. Applying YOCO improves the diversity of the augmentation per sample and encourages neural networks to recognize objects from partial information. YOCO enjoys the properties of parameter-free, easy usage, and boosting almost all augmentations for free. Thorough experiments are conducted to evaluate its effectiveness. We first demonstrate that YOCO can be seamlessly applied to varying data augmentations, neural network architectures, and brings performance gains on CIFAR and ImageNet classification tasks, sometimes surpassing conventional image-level augmentation by large margins. Moreover, we show YOCO benefits contrastive pre-training toward a more powerful representation that can be better transferred to multiple downstream tasks. Finally, we study a number of variants of YOCO and empirically analyze the performance for respective settings. Code is available at GitHub.

arXiv:2307.10638

Quantized Feature Distillation for Network Quantization

Neural network quantization aims to accelerate and trim full-precision neural network models by using low bit approximations. Methods adopting the quantization aware training (QAT) paradigm have recently seen a rapid growth, but are often conceptually complicated. This paper proposes a novel and highly effective QAT method, quantized feature distillation (QFD). QFD first trains a quantized (or binarized) representation as the teacher, then quantize the network using knowledge distillation (KD). Quantitative results show that QFD is more flexible and effective (i.e., quantization friendly) than previous quantization methods. QFD surpasses existing methods by a noticeable margin on not only image classification but also object detection, albeit being much simpler. Furthermore, QFD quantizes ViT and Swin-Transformer on MS-COCO detection and segmentation, which verifies its potential in real world deployment. To the best of our knowledge, this is the first time that vision transformers have been quantized in object detection and image segmentation tasks.

arXiv:2605.24602v5cs.CV

어텐션 분산으로 유발된 시각적 흐림 보정을 통한 환각 감소: 알고리즘과 이론

Correcting Visual Blur Induced by Attention Distraction to Reduce Hallucinations: Algorithm and Theory

멀티모달 대규모 언어 모델(MLLM)에서 발생하는 객체 환각(object hallucination)이 인간의 주의 분산 시 시각적 선명도가 저하되는 현상과 유사하게, 멀티헤드 어텐션의 공간적 불일치와 디코딩 과정에서 이미지 토큰에 대한 어텐션의 시간적 소실로 나타남을 규명하였다. 이를 바탕으로 크로스헤드 어텐션 강화와 동적 과거 어텐션 강화를 결합한 훈련 불필요 모듈 AFIP를 제안하여 다양한 벤치마크에서 환각 억제 효과를 검증하였다.

arXiv:2608.25575v1cs.CV

MLLMCLIP: 강건한 비전-언어 표현을 위한 MLLM의 특징 수준 지식 증류

MLLMCLIP: Feature-Level Distillation of MLLM for Robust Vision-Language Representations

CLIP과 같은 사전학습 비전-언어 모델은 제로샷(zero-shot) 인식에서 뛰어난 성능을 보이지만, 특히 속성-객체 및 관계 구조와 같은 구성성(compositionality) 처리에는 종종 실패합니다. 최근 연구들은 대규모 언어 모델과 텍스트-투-이미지(text-to-image) 모델의 캐스케이드로 생성된 합성 하드 네거티브(hard negative)로 학습을 증강하여 이 문제를 완화하지만, 이는 상당한 파이프라인 오버헤드를 유발합니다. 대신 본 논문에서는 합성 데이터를 완전히 배제하고 생성형 멀티모달 대규모 언어 모델(MLLM) 교사로부터 판별형 CLIP 학생 모델로 멀티모달 지식을 직접 전달하는 이종 지식 증류(distillation) 프레임워크인 MLLMCLIP을 제안합니다. 두 패러다임 간의 아키텍처 불일치를 해소하기 위해 어텐션(attention) 기반 층별 토큰 선택 기법과 CKA 기반 증류 손실(loss)을 도입합니다. 이전의 CLIP 개선 방법들과 비교하여 MLLMCLIP은 최고 수준의 구성적 정확도를 달성하는 동시에 표준 제로샷 분류 및 이미지-텍스트 검색에서도 일관된 성능 향상을 보이며, 특징 수준의 지식 증류가 구성적 비전-언어 표현 능력과 일반적인 비전-언어 표현 능력을 모두 강화함을 입증합니다.

arXiv:2405.09194

Flexible image analysis for law enforcement agencies with deep neural networks to determine: where, who and what

Due to the increasing need for effective security measures and the integration of cameras in commercial products, a hugeamount of visual data is created today. Law enforcement agencies (LEAs) are inspecting images and videos to findradicalization, propaganda for terrorist organizations and illegal products on darknet markets. This is time consuming.Instead of an undirected search, LEAs would like to adapt to new crimes and threats, and focus only on data from specificlocations, persons or objects, which requires flexible interpretation of image content. Visual concept detection with deepconvolutional neural networks (CNNs) is a crucial component to understand the image content. This paper has fivecontributions. The first contribution allows image-based geo-localization to estimate the origin of an image. CNNs andgeotagged images are used to create a model that determines the location of an image by its pixel values. The secondcontribution enables analysis of fine-grained concepts to distinguish sub-categories in a generic concept. The proposedmethod encompasses data acquisition and cleaning and concept hierarchies. The third contribution is the recognition ofperson attributes (e.g., glasses or moustache) to enable query by textual description for a person. The person-attributeproblem is treated as a specific sub-task of concept classification. The fourth contribution is an intuitive image annotationtool based on active learning. Active learning allows users to define novel concepts flexibly and train CNNs with minimalannotation effort. The fifth contribution increases the flexibility for LEAs in the query definition by using query expansion.Query expansion maps user queries to known and detectable concepts. Therefore, no prior knowledge of the detectableconcepts is required for the users. The methods are validated on data with varying locations (popular and non-touristiclocations), varying person attributes (CelebA dataset), and varying number of annotations.

arXiv:2406.05184공식 코드 · Papers with Code

The Unmet Promise of Synthetic Training Images: Using Retrieved Real Images Performs Better

Generative text-to-image models enable us to synthesize unlimited amounts of images in a controllable manner, spurring many recent efforts to train vision models with synthetic data. However, every synthetic image ultimately originates from the upstream data used to train the generator. Does the intermediate generator provide additional information over directly training on relevant parts of the upstream data? Grounding this question in the setting of image classification, we compare finetuning on task-relevant, targeted synthetic data generated by Stable Diffusion -- a generative model trained on the LAION-2B dataset -- against finetuning on targeted real images retrieved directly from LAION-2B. We show that while synthetic data can benefit some downstream tasks, it is universally matched or outperformed by real data from the simple retrieval baseline. Our analysis suggests that this underperformance is partially due to generator artifacts and inaccurate task-relevant visual details in the synthetic images. Overall, we argue that targeted retrieval is a critical baseline to consider when training with synthetic data -- a baseline that current methods do not yet surpass. We release code, data, and models at https://github.com/scottgeng00/unmet-promise.