PyTorchKR Papers

“Image Classification”

총 18,403편
태그·기간 등 상세 필터

카테고리 탐색

arXiv:2506.08900공식 코드 · Papers with Code

MIRAGE: Multimodal foundation model and benchmark for comprehensive retinal OCT image analysis

Artificial intelligence (AI) has become a fundamental tool for assisting clinicians in analyzing ophthalmic images, such as optical coherence tomography (OCT). However, developing AI models often requires extensive annotation, and existing models tend to underperform on independent, unseen data. Foundation models (FMs), large AI models trained on vast unlabeled datasets, have shown promise in overcoming these challenges. Nonetheless, available FMs for ophthalmology lack extensive validation, especially for segmentation tasks, and focus on a single imaging modality. In this context, we propose MIRAGE, a novel multimodal FM for the analysis of OCT and scanning laser ophthalmoscopy (SLO) images. Additionally, we propose a new evaluation benchmark with OCT/SLO classification and segmentation tasks. The comparison with general and specialized FMs and segmentation methods shows the superiority of MIRAGE in both types of tasks, highlighting its suitability as a basis for the development of robust AI systems for retinal OCT image analysis. Both MIRAGE and the evaluation benchmark are publicly available: https://github.com/j-morano/MIRAGE.

arXiv:1810.05964

Perceptual Image Quality Assessment through Spectral Analysis of Error Representations

In this paper, we analyze the statistics of error signals to assess the perceived quality of images. Specifically, we focus on the magnitude spectrum of error images obtained from the difference of reference and distorted images. Analyzing spectral statistics over grayscale images partially models interference in spatial harmonic distortion exhibited by the visual system but it overlooks color information, selective and hierarchical nature of visual system. To overcome these shortcomings, we introduce an image quality assessment algorithm based on the Spectral Understanding of Multi-scale and Multi-channel Error Representations, denoted as SUMMER. We validate the quality assessment performance over 3 databases with around 30 distortion types. These distortion types are grouped into 7 main categories as compression artifact, image noise, color artifact, communication error, blur, global and local distortions. In total, we benchmark the performance of 17 algorithms along with the proposed algorithm using 5 performance metrics that measure linearity, monotonicity, accuracy, and consistency. In addition to experiments with standard performance metrics, we analyze the distribution of objective and subjective scores with histogram difference metrics and scatter plots. Moreover, we analyze the classification performance of quality assessment algorithms along with their statistical significance tests. Based on our experiments, SUMMER significantly outperforms majority of the compared methods in all benchmark categories

arXiv:2103.04736공식 코드 · Papers with Code

Content-Aware Detection of Temporal Metadata Manipulation

Most pictures shared online are accompanied by temporal metadata (i.e., the day and time they were taken), which makes it possible to associate an image content with real-world events. Maliciously manipulating this metadata can convey a distorted version of reality. In this work, we present the emerging problem of detecting timestamp manipulation. We propose an end-to-end approach to verify whether the purported time of capture of an outdoor image is consistent with its content and geographic location. We consider manipulations done in the hour and/or month of capture of a photograph. The central idea is the use of supervised consistency verification, in which we predict the probability that the image content, capture time, and geographical location are consistent. We also include a pair of auxiliary tasks, which can be used to explain the network decision. Our approach improves upon previous work on a large benchmark dataset, increasing the classification accuracy from 59.0% to 81.1%. We perform an ablation study that highlights the importance of various components of the method, showing what types of tampering are detectable using our approach. Finally, we demonstrate how the proposed method can be employed to estimate a possible time-of-capture in scenarios in which the timestamp is missing from the metadata.

arXiv:2104.13188공식 코드 · Papers with Code

Rethinking BiSeNet For Real-time Semantic Segmentation

BiSeNet has been proved to be a popular two-stream network for real-time segmentation. However, its principle of adding an extra path to encode spatial information is time-consuming, and the backbones borrowed from pretrained tasks, e.g., image classification, may be inefficient for image segmentation due to the deficiency of task-specific design. To handle these problems, we propose a novel and efficient structure named Short-Term Dense Concatenate network (STDC network) by removing structure redundancy. Specifically, we gradually reduce the dimension of feature maps and use the aggregation of them for image representation, which forms the basic module of STDC network. In the decoder, we propose a Detail Aggregation module by integrating the learning of spatial information into low-level layers in single-stream manner. Finally, the low-level features and deep features are fused to predict the final segmentation results. Extensive experiments on Cityscapes and CamVid dataset demonstrate the effectiveness of our method by achieving promising trade-off between segmentation accuracy and inference speed. On Cityscapes, we achieve 71.9% mIoU on the test set with a speed of 250.4 FPS on NVIDIA GTX 1080Ti, which is 45.2% faster than the latest methods, and achieve 76.8% mIoU with 97.0 FPS while inferring on higher resolution images.

arXiv:2505.02949

Gone With the Bits: Revealing Racial Bias in Low-Rate Neural Compression for Facial Images

Neural compression methods are gaining popularity due to their superior rate-distortion performance over traditional methods, even at extremely low bitrates below 0.1 bpp. As deep learning architectures, these models are prone to bias during the training process, potentially leading to unfair outcomes for individuals in different groups. In this paper, we present a general, structured, scalable framework for evaluating bias in neural image compression models. Using this framework, we investigate racial bias in neural compression algorithms by analyzing nine popular models and their variants. Through this investigation, we first demonstrate that traditional distortion metrics are ineffective in capturing bias in neural compression models. Next, we highlight that racial bias is present in all neural compression models and can be captured by examining facial phenotype degradation in image reconstructions. We then examine the relationship between bias and realism in the decoded images and demonstrate a trade-off across models. Finally, we show that utilizing a racially balanced training set can reduce bias but is not a sufficient bias mitigation strategy. We additionally show the bias can be attributed to compression model bias and classification model bias. We believe that this work is a first step towards evaluating and eliminating bias in neural image compression models.

arXiv:2610.01778v1cs.CV

GIFTBench: 이미지 위조 국소화에서의 일반화 진단 및 모델 설계 안내

GIFTBench: Diagnosing Generalization in Image Forgery Localization and Informing Model Design

이미지 위조 국소화(image forgery localization, IFL)에 대한 신뢰할 수 있는 평가(evaluation)는 다양한 분포 변화 하에서 모델(model)을 평가할 것을 요구하지만, 기존 벤치마크(benchmark)들은 제한된 조작 조건만을 다루거나 교차 데이터셋 평가에서 여러 요인을 뒤얽히게 만드는 경우가 많습니다. 결과적으로 종합적인 성능만으로는 국소화 일반화(generalization)에 대한 불완전한 시각만을 제공하게 됩니다. 본 연구에서는 조작 출처, 의미론적 대상, 편집 연산, 합성 복잡도에 걸쳐 픽셀 수준 주석이 포함된 115,013개의 조작 이미지로 구성된 다축 벤치마크인 GIFTBench를 소개합니다. GIFTBench는 축별 전이 분석과 12개의 외부 데이터셋(dataset)에 대한 평가를 지원합니다. 본 벤치마크의 진단 연구는 비대칭적 교차 출처 전이, 재현율(recall) 위주의 실패, 그리고 의미론적·연산적·합성적 변화에 걸친 이종적 성능 저하를 밝혀냅니다. 진단을 넘어, GIFTBench의 규모와 다양성은 기존 IFL 데이터셋보다 실질적으로 훨씬 더 넓은 학습(training) 분포를 제공합니다. GIFTBench에서 대표적인 국소화 모델들을 학습시키면 외부 데이터셋으로의 종합적인 전이 성능이 일관되게 향상되며, 이는 본 벤치마크가 평가 도구뿐만 아니라 도메인 간 국소화를 위한 효과적인 학습 자원으로도 기능함을 보여줍니다. 이러한 진단 결과를 바탕으로, 우리는 분류(classification)에 적응된 표현과 다중 심도·다중 스케일 공간 특징, 학습된 층(layer) 융합, 선택적 조대 스케일 조건화를 결합한 탐지 및 국소화 프레임워크인 ForenScope를 추가로 개발했습니다. 실험 결과, 이미지 수준의 탐지 능력을 유지하면서도 향상된 교차 데이터셋 국소화 성능을 보였습니다. GIFTBench 데이터셋 쇼케이스 페이지는 https://giftbench-preview.doudoudouya337.chatgpt.site 에서 확인할 수 있습니다.

arXiv:2001.10966

Developing a gender classification approach in human face images using modified local binary patterns and tani-moto based nearest neighbor algorithm

Human identification is a much attention problem in computer vision. Gender classification plays an important role in human identification as preprocess step. So far, various methods have been proposed to solve this problem. Absolutely, classification accuracy is the main challenge for researchers in gender classification. But, some challenges such as rotation, gray scale variations, pose, illumination changes may be occurred in smart phone image capturing. In this respect, a multi step approach is proposed in this paper to classify genders in human face images based on improved local binary patters (MLBP). LBP is a texture descriptor, which extract local contrast and local spatial structure information. Some issues such as noise sensitivity, rotation sensitivity and low discriminative features can be considered as disadvantages of the basic LBP. MLBP handle disadvantages using a new theory to categorize extracted binary patterns of basic LBP. The proposed approach includes two stages. First of all, a feature vector is extracted for human face images based on MLBP. Next, non linear classifiers can be used to classify gender. In this paper nearest neighborhood classifier is evaluated based on Tani-Moto metric as distance measure. In the result part, two databases, self-collected and ICPR are used as human face database. Results are compared by some state-ofthe-art algorithms in this literature that shows the high quality of the proposed approach in terms of accuracy rate. Some of other main advantages of the proposed approach are rotation invariant, low noise sensitivity, size invariant and low computational complexity. The proposed approach decreases the computational complexity of smartphone applications because of reducing the number of database comparisons. It can also improve performance of the synchronous applications in the smarphones because of memory and CPU usage reduction.

arXiv:2108.04555공식 코드 · Papers with Code

Deep Joint Learning of Pathological Region Localization and Alzheimer's Disease Diagnosis

The identification of Alzheimer's disease (AD) and its early stages using structural magnetic resonance imaging (MRI) has been attracting the attention of researchers. Various data-driven approaches have been introduced to capture subtle and local morphological changes of the brain accompanied by the disease progression. One of the typical approaches for capturing subtle changes is patch-level feature representation. However, the predetermined regions to extract patches can limit classification performance by interrupting the exploration of potential biomarkers. In addition, the existing patch-level analyses have difficulty explaining their decision-making. To address these problems, we propose the BrainBagNet with a position-based gate (PG-BrainBagNet), a framework for jointly learning pathological region localization and AD diagnosis in an end-to-end manner. In advance, as all scans are aligned to a template in image processing, the position of brain images can be represented through the 3D Cartesian space shared by the overall MRI scans. The proposed method represents the patch-level response from whole-brain MRI scans and discriminative brain-region from position information. Based on the outcomes, the patch-level class evidence is calculated, and then the image-level prediction is inferred by a transparent aggregation. The proposed models were evaluated on the ADNI datasets. In five-fold cross-validation, the classification performance of the proposed method outperformed that of the state-of-the-art methods in both AD diagnosis (AD vs. normal control) and mild cognitive impairment (MCI) conversion prediction (progressive MCI vs. stable MCI) tasks. In addition, changes in the identified discriminant regions and patch-level class evidence according to the patch size used for model training are presented and analyzed.

arXiv:2209.08770

S$^3$R: Self-supervised Spectral Regression for Hyperspectral Histopathology Image Classification

Benefited from the rich and detailed spectral information in hyperspectral images (HSI), HSI offers great potential for a wide variety of medical applications such as computational pathology. But, the lack of adequate annotated data and the high spatiospectral dimensions of HSIs usually make classification networks prone to overfit. Thus, learning a general representation which can be transferred to the downstream tasks is imperative. To our knowledge, no appropriate self-supervised pre-training method has been designed for histopathology HSIs. In this paper, we introduce an efficient and effective Self-supervised Spectral Regression (S$^3$R) method, which exploits the low rank characteristic in the spectral domain of HSI. More concretely, we propose to learn a set of linear coefficients that can be used to represent one band by the remaining bands via masking out these bands. Then, the band is restored by using the learned coefficients to reweight the remaining bands. Two pre-text tasks are designed: (1)S$^3$R-CR, which regresses the linear coefficients, so that the pre-trained model understands the inherent structures of HSIs and the pathological characteristics of different morphologies; (2)S$^3$R-BR, which regresses the missing band, making the model to learn the holistic semantics of HSIs. Compared to prior arts i.e., contrastive learning methods, which focuses on natural images, S$^3$R converges at least 3 times faster, and achieves significant improvements up to 14% in accuracy when transferring to HSI classification tasks.

arXiv:2405.15563

Heterogeneous virus classification using a functional deep learning model based on transmission electron microscopy images (Preprint)

Viruses are submicroscopic agents that can infect all kinds of lifeforms and use their hosts' living cells to replicate themselves. Despite having some of the simplest genetic structures among all living beings, viruses are highly adaptable, resilient, and given the right conditions, are capable of causing unforeseen complications in their hosts' bodies. Due to their multiple transmission pathways, high contagion rate, and lethality, viruses are the biggest biological threat faced by animal and plant species. It is often challenging to promptly detect the presence of a virus in a possible host's body and accurately determine its type using manual examination techniques; however, it can be done using computer-based automatic diagnosis methods. Most notably, the analysis of Transmission Electron Microscopy (TEM) images has been proven to be quite successful in instant virus identification. Using TEM images collected from a recently published dataset, this article proposes a deep learning-based classification model to identify the type of virus within those images correctly. The methodology of this study includes two coherent image processing techniques to reduce the noise present in the raw microscopy images. Experimental results show that it can differentiate among the 14 types of viruses present in the dataset with a maximum of 97.44% classification accuracy and F1-score, which asserts the effectiveness and reliability of the proposed method. Implementing this scheme will impart a fast and dependable way of virus identification subsidiary to the thorough diagnostic procedures.

arXiv:2607.26043v1cs.LG

유방촬영술 전이학습의 재고: 유방암 선별검사 및 병변 진단을 위한 데이터셋 기반 전이학습(DITL) 프레임워크

Re-thinking Mammography Transfer Learning: The Dataset-Informed Transfer Learning (DITL) Framework for Breast Cancer Screening and Lesion Diagnosis

유방촬영술(mammography)에서 분류(classification) 성능을 향상하는 것은 소규모 정제 데이터셋과 대규모 임상 코호트 전반에서 여전히 지속적인 과제로 남아 있습니다. 기존 전이학습(transfer learning) 방식은 데이터셋(dataset) 고유의 특성을 간과하는 경우가 많으며, 최근의 이웃 정보 기반(neighborhood-informed) 방법들은 경직된 수식과 함께 협소한 작업에만 국한되어 대규모 인구 집단 수준의 데이터셋으로의 확장성이 제한되었습니다. 이러한 문제를 해결하기 위해, 본 논문에서는 데이터셋에서 파생된 난이도 신호와 이웃 기반 트리플렛 지도(triplet supervision)를 단일 목적함수로 통합한 데이터셋 기반 전이학습(Dataset-Informed Transfer Learning, DITL) 프레임워크를 제안합니다. DITL은 두 가지 적응형 구성 요소를 도입합니다: (i) 자기지도(self-supervised) 특징(feature) 공간에서 k-최근접 이웃(k-nearest neighbor) 레이블 순도에 따라 샘플별 가중치를 부여하는 적응형 난이도 가중 교차 엔트로피(Adaptive Difficulty-Weighted Cross-Entropy, A-DWCE), 그리고 (ii) 학습 가능한 마진(margin)을 사용하여 클래스 내 응집도와 클래스 간 분리를 강화하는 적응형 이웃 표현 트리플렛(Adaptive Neighborhood Representation Triplet, A-NR-Triplet)입니다. 포컬 손실(focal loss)과 달리 DITL은 하이퍼파라미터(hyperparameter) 튜닝이 필요 없고 경험적 가중치 부여 및 고정 마진을 제거하며 연산 오버헤드가 거의 없어, 견고하고 확장 가능한 최적화 전략을 제공합니다. 대규모 VinDR-Mammo 데이터셋에서 DITL은 전반적인 이미지 유방 밀도 분류 부문 최고 성능(SOTA)을 달성하였으며, 정확도, F1-score, AUC 전반에 걸쳐 유의미한 향상(p < 0.0001)을 보였습니다. 대규모 코호트를 넘어 DITL은 소규모 관심 영역(ROI) 데이터셋에서도 일관되고 통계적으로 유의미한 성능 향상(p < 0.0001)을 제공합니다. 소규모 병변 분석과 대규모 밀도 추정을 연결함으로써, DITL은 유방암 선별검사부터 진단에 이르는 전 과정을 포괄하며 임상적으로 유의미하고 확장 가능하며 일반화 능력을 갖춘 유방촬영술 분류 프레임워크를 구축합니다.

arXiv:2609.05070v1cs.CV

MultiAttenGastro: 위장관 내시경 분류를 위한 다차원 어텐션 증강

MultiAttenGastro: Multi-Dimensional Attention Augmentation for Gastrointestinal Endoscopy Classification

자동화된 위장관(GI) 내시경 분류는 종종 자연 이미지 사전학습(pretraining)과 큰 차이를 보이는 다양한 모달리티와 클래스 분포 전반에 걸쳐 일반화할 수 있는 모델을 필요로 합니다. 본 논문에서는 병렬 1차원 채널, 2차원 공간, 3차원 문맥 헤드를 갖춘 플러그 앤 플레이 방식의 어텐션(attention) 프레임워크인 MultiAttenGastro를 제안하고, 5개의 공개 GI 데이터셋(dataset)에서 8개의 CNN 및 트랜스포머(transformer) 백본을 대상으로 최초의 체계적인 교차 데이터셋 평가(80회의 백본-데이터셋 실행)를 제시합니다. 연구 결과 어텐션의 효과는 보편적이지 않으며 ImageNet 피처와 타깃 분포 간의 표현 격차(representational gap)를 따르는 것으로 나타났습니다. 즉, MultiAttenGastro는 Kvasir-Capsule(14개 클래스 WCE, 큰 격차, 최고 매크로 F1 98.33%)에서 8개 백본 중 6개를 향상시켰으나, 격차가 작은 Kvasir-v2 벤치마크에서는 일관되게 부정적인 결과(0/8)를 보였고, 중간 정도의 격차를 가진 데이터셋에서는 엇갈린 결과를 보였습니다. 가장 효과가 컸던 사례(Kvasir-Capsule, ConvNeXt-Tiny)에 대한 5개 시드(seed) 아블레이션 분석 결과, 이러한 향상은 방향성 측면에서는 일관되지만 통계적으로 결정적이지는 않았으며(대응표본 $t$ 검정: $p=0.47$, 윌콕슨 검정: $p=0.63$), 개별 어텐션 헤드는 단독으로는 균일하게 유익하지 않고 이들의 조합만이 긍정적인 평균 효과를 내는 것으로 나타났습니다. 중심화 커널 정렬(CKA) 분석은 이 패턴을 표현 중복성(representational redundancy)과 연결짓습니다. 도메인 격차가 클 때 낮은 헤드 간 CKA는 프레임워크의 유일하게 일관된 성능 향상과 일치하는 반면, 격차가 작을 때의 높은 중복성은 성능 저하와 일치합니다. 우리는 MultiAttenGastro가 엄격하게 우월한 아키텍처적 선택이라는 주장보다는, 다차원 어텐션이 GI 내시경 분류에 언제, 왜 도움이 되는지에 대한 증거로서 유의하지 않은 차이를 포함한 이러한 결과를 보고합니다.

arXiv:1611.05328

Image Credibility Analysis with Effective Domain Transferred Deep Networks

Numerous fake images spread on social media today and can severely jeopardize the credibility of online content to public. In this paper, we employ deep networks to learn distinct fake image related features. In contrast to authentic images, fake images tend to be eye-catching and visually striking. Compared with traditional visual recognition tasks, it is extremely challenging to understand these psychologically triggered visual patterns in fake images. Traditional general image classification datasets, such as ImageNet set, are designed for feature learning at the object level but are not suitable for learning the hyper-features that would be required by image credibility analysis. In order to overcome the scarcity of training samples of fake images, we first construct a large-scale auxiliary dataset indirectly related to this task. This auxiliary dataset contains 0.6 million weakly-labeled fake and real images collected automatically from social media. Through an AdaBoost-like transfer learning algorithm, we train a CNN model with a few instances in the target training set and 0.6 million images in the collected auxiliary set. This learning algorithm is able to leverage knowledge from the auxiliary set and gradually transfer it to the target task. Experiments on a real-world testing set show that our proposed domain transferred CNN model outperforms several competing baselines. It obtains superiror results over transfer learning methods based on the general ImageNet set. Moreover, case studies show that our proposed method reveals some interesting patterns for distinguishing fake and authentic images.

arXiv:2107.04886

Hierarchical Self-Supervised Learning for Medical Image Segmentation Based on Multi-Domain Data Aggregation

A large labeled dataset is a key to the success of supervised deep learning, but for medical image segmentation, it is highly challenging to obtain sufficient annotated images for model training. In many scenarios, unannotated images are abundant and easy to acquire. Self-supervised learning (SSL) has shown great potentials in exploiting raw data information and representation learning. In this paper, we propose Hierarchical Self-Supervised Learning (HSSL), a new self-supervised framework that boosts medical image segmentation by making good use of unannotated data. Unlike the current literature on task-specific self-supervised pretraining followed by supervised fine-tuning, we utilize SSL to learn task-agnostic knowledge from heterogeneous data for various medical image segmentation tasks. Specifically, we first aggregate a dataset from several medical challenges, then pre-train the network in a self-supervised manner, and finally fine-tune on labeled data. We develop a new loss function by combining contrastive loss and classification loss and pretrain an encoder-decoder architecture for segmentation tasks. Our extensive experiments show that multi-domain joint pre-training benefits downstream segmentation tasks and outperforms single-domain pre-training significantly. Compared to learning from scratch, our new method yields better performance on various tasks (e.g., +0.69% to +18.60% in Dice scores with 5% of annotated data). With limited amounts of training data, our method can substantially bridge the performance gap w.r.t. denser annotations (e.g., 10% vs.~100% of annotated data).

arXiv:2110.13670

W-Net: A Two-Stage Convolutional Network for Nucleus Detection in Histopathology Image

Pathological diagnosis is the gold standard for cancer diagnosis, but it is labor-intensive, in which tasks such as cell detection, classification, and counting are particularly prominent. A common solution for automating these tasks is using nucleus segmentation technology. However, it is hard to train a robust nucleus segmentation model, due to several challenging problems, the nucleus adhesion, stacking, and excessive fusion with the background. Recently, some researchers proposed a series of automatic nucleus segmentation methods based on point annotation, which can significant improve the model performance. Nevertheless, the point annotation needs to be marked by experienced pathologists. In order to take advantage of segmentation methods based on point annotation, further alleviate the manual workload, and make cancer diagnosis more efficient and accurate, it is necessary to develop an automatic nucleus detection algorithm, which can automatically and efficiently locate the position of the nucleus in the pathological image and extract valuable information for pathologists. In this paper, we propose a W-shaped network for automatic nucleus detection. Different from the traditional U-Net based method, mapping the original pathology image to the target mask directly, our proposed method split the detection task into two sub-tasks. The first sub-task maps the original pathology image to the binary mask, then the binary mask is mapped to the density mask in the second sub-task. After the task is split, the task's difficulty is significantly reduced, and the network's overall performance is improved.

arXiv:2310.05969공식 코드 · Papers with Code

Automated Chest X-Ray Report Generator Using Multi-Model Deep Learning Approach

Reading and interpreting chest X-ray images is one of the most radiologist's routines. However, it still can be challenging, even for the most experienced ones. Therefore, we proposed a multi-model deep learning-based automated chest X-ray report generator system designed to assist radiologists in their work. The basic idea of the proposed system is by utilizing multi binary-classification models for detecting multi abnormalities, with each model responsible for detecting one abnormality, in a single image. In this study, we limited the radiology abnormalities detection to only cardiomegaly, lung effusion, and consolidation. The system generates a radiology report by performing the following three steps: image pre-processing, utilizing deep learning models to detect abnormalities, and producing a report. The aim of the image pre-processing step is to standardize the input by scaling it to 128x128 pixels and slicing it into three segments, which covers the upper, lower, and middle parts of the lung. After pre-processing, each corresponding model classifies the image, resulting in a 0 (zero) for no abnormality detected and a 1 (one) for the presence of an abnormality. The prediction outputs of each model are then concatenated to form a 'result code'. The 'result code' is used to construct a report by selecting the appropriate pre-determined sentence for each detected abnormality in the report generation step. The proposed system is expected to reduce the workload of radiologists and increase the accuracy of chest X-ray diagnosis.

arXiv:2501.01973

INFELM: In-depth Fairness Evaluation of Large Text-To-Image Models

The rapid development of large language models (LLMs) and large vision models (LVMs) have propelled the evolution of multi-modal AI systems, which have demonstrated the remarkable potential for industrial applications by emulating human-like cognition. However, they also pose significant ethical challenges, including amplifying harmful content and reinforcing societal biases. For instance, biases in some industrial image generation models highlighted the urgent need for robust fairness assessments. Most existing evaluation frameworks focus on the comprehensiveness of various aspects of the models, but they exhibit critical limitations, including insufficient attention to content generation alignment and social bias-sensitive domains. More importantly, their reliance on pixel-detection techniques is prone to inaccuracies. To address these issues, this paper presents INFELM, an in-depth fairness evaluation on widely-used text-to-image models. Our key contributions are: (1) an advanced skintone classifier incorporating facial topology and refined skin pixel representation to enhance classification precision by at least 16.04%, (2) a bias-sensitive content alignment measurement for understanding societal impacts, (3) a generalizable representation bias evaluation for diverse demographic groups, and (4) extensive experiments analyzing large-scale text-to-image model outputs across six social-bias-sensitive domains. We find that existing models in the study generally do not meet the empirical fairness criteria, and representation bias is generally more pronounced than alignment errors. INFELM establishes a robust benchmark for fairness assessment, supporting the development of multi-modal AI systems that align with ethical and human-centric principles.

arXiv:2503.24267

FakeScope: Large Multimodal Expert Model for Transparent AI-Generated Image Forensics

The rapid and unrestrained advancement of generative artificial intelligence (AI) presents a double-edged sword: while enabling unprecedented creativity, it also facilitates the generation of highly convincing deceptive content, undermining societal trust. As image generation techniques become increasingly sophisticated, detecting synthetic images is no longer just a binary task: it necessitates interpretable, context-aware methodologies that enhance trustworthiness and transparency. However, existing detection models primarily focus on classification, offering limited explanatory insights into image authenticity. In this work, we propose FakeScope, an expert multimodal model (LMM) tailored for AI-generated image forensics, which not only identifies AI-synthetic images with high accuracy but also provides rich, interpretable, and query-driven forensic insights. We first construct FakeChain dataset that contains linguistic authenticity reasoning based on visual trace evidence, developed through a novel human-machine collaborative framework. Building upon it, we further present FakeInstruct, the largest multimodal instruction tuning dataset containing 2 million visual instructions tailored to enhance forensic awareness in LMMs. FakeScope achieves state-of-the-art performance in both closed-ended and open-ended forensic scenarios. It can distinguish synthetic images with high accuracy while offering coherent and insightful explanations, free-form discussions on fine-grained forgery attributes, and actionable enhancement strategies. Notably, despite being trained exclusively on qualitative hard labels, FakeScope demonstrates remarkable zero-shot quantitative capability on detection, enabled by our proposed token-based probability estimation strategy. Furthermore, FakeScope exhibits strong generalization and in-the-wild ability, ensuring its applicability in real-world scenarios.

arXiv:2606.21113v1cs.CV

제한 데이터 환경의 이미지 생성 및 데이터 증강을 위한 객체 중심 데이터셋 리소스

Object-Centric Dataset Resources for Constrained-Data Image Generation and Augmentation

객체 중심 이미지 생성은 라벨이 적은 예시만 있는 설정에서 중요하며, 여기에는 스마트시티 장면에서의 보행자 분석, 교통 표지판 검사, 도메인 특화 객체 검출 등이 포함된다. 합성 이미지는 데이터셋이 객체 구조, 바운딩 박스, 시각적 다양성, 현실적인 맥락을 보존할 때 학습과 평가에가장 유용하다. 기존 이미지 데이터셋은 일반적으로 분류, 검출, 장면 이해를 목표로 하며, 클래스별 데이터가 제한된 환경에서의 통제된 객체 중심 생성 및 증강보다는 그쪽에 초점을 맞춘다. 우리는 Cityscapes-Pedestrian, TrafficSigns, COCO PottedPlant의 세 객체 중심 데이터셋 리소스를 공유 가능한 형태로 모은 컬렉션을 제시한다. 이 컬렉션은 세 영역에 걸쳐 256×256 객체 중심 크롭과 바운딩 박스 주석을 표준화한다: 프라이버시 블러와 폐색이 있는 밀집 보행자 장면, 대비가 높은 깨끗한 교통 표지판, 맥락이 다양한 화분 장면. 이 릴리스에는 TrafficSigns 샘플 3,009개, Cityscapes-Pedestrian 매니페스트 레코드 2,156개, COCO PottedPlant 매니페스트 레코드 7,679개가 포함된다. 더 큰 COCO 기반 매니페스트는 맥락적·다중 인스턴스 다양성을 보존하며, 동일 크기 부분집합은 통제된 비교를 위해 고정된 난수 시드로 추출할 수 있다. 릴리스는 재배포가 허용되는 경우 TrafficSigns 데이터를 직접 제공하며, Cityscapes 및 COCO 기반 부분집합에 대해서는 스크립트, 매니페스트, 박스 수준 주석 테이블, 체크섬, 재구성 문서를 함께 제공한다. Latzi/object-centric-low-data-datasets GitHub 저장소와 Zenodo DOI 10.5281/zenodo.20573001을 통해 이용할 수 있다. 이 컬렉션은 라벨 및 분할 검사, 부분집합 생성, 업스트림 데이터로부터의 재구성, 공유 레코드에 대한 객체 중심 이미지 생성 또는 합성 데이터 증강 방법의 평가를 지원한다.

arXiv:1707.08525

A Guided Spatial Transformer Network for Histology Cell Differentiation

Identification and counting of cells and mitotic figures is a standard task in diagnostic histopathology. Due to the large overall cell count on histological slides and the potential sparse prevalence of some relevant cell types or mitotic figures, retrieving annotation data for sufficient statistics is a tedious task and prone to a significant error in assessment. Automatic classification and segmentation is a classic task in digital pathology, yet it is not solved to a sufficient degree. We present a novel approach for cell and mitotic figure classification, based on a deep convolutional network with an incorporated Spatial Transformer Network. The network was trained on a novel data set with ten thousand mitotic figures, about ten times more than previous data sets. The algorithm is able to derive the cell class (mitotic tumor cells, non-mitotic tumor cells and granulocytes) and their position within an image. The mean accuracy of the algorithm in a five-fold cross-validation is 91.45%. In our view, the approach is a promising step into the direction of a more objective and accurate, semi-automatized mitosis counting supporting the pathologist.