Metallic materials such as brass, copper, and aluminum are used in numerous applications, including industrial manufacturing. The vibration characteristics of these objects are unique and can be used to identify these objects from a distance. This research presents a methodology for detecting and classifying these metallic objects using the vibration dynamics induced by their micro-Doppler signatures. The proposed approach utilizes image processing techniques to extract pivotal features from spectrograms. These spectrograms originate from micro-Doppler signatures of data collected during controlled laboratory experiments where signals were transmitted towards vibrating metal sheets, and the ensuing reflections were recorded using a software-defined radio (SDR). The spectrogram data was augmented using geometric transformation to train a convolutional neural network (CNN) based machine learning model for object classification. The results indicate that the proposed CNN model achieved an accuracy of more than 95% in classifying metals into brass, copper, and aluminum. This research could be used to understand the foundations of classifying spectrogram images using micro-Doppler signatures for its applications towards enhancing the sensing capabilities in industrial and defense applications.
Few-shot learning (FSL) is a challenging task in machine learning, demanding a model to render discriminative classification by using only a few labeled samples. In the literature of FSL, deep models are trained in a manner of metric learning to provide metric in a feature space which is well generalizable to classify samples of novel classes; in the space, even a few amount of labeled training examples can construct an effective classifier. In this paper, we propose a novel FSL loss based on \emph{geometric mean} to embed discriminative metric into deep features. In contrast to the other losses such as utilizing arithmetic mean in softmax-based formulation, the proposed method leverages geometric mean to aggregate pair-wise relationships among samples for enhancing discriminative metric across class categories. The proposed loss is not only formulated in a simple form but also is thoroughly analyzed in theoretical ways to reveal its favorable characteristics which are favorable for learning feature metric in FSL. In the experiments on few-shot image classification tasks, the method produces competitive performance in comparison to the other losses.
Multimodal fake news detection typically demands complex architectures and substantial computational resources, posing deployment challenges in real-world settings. We introduce UNITE-FND, a novel framework that reframes multimodal fake news detection as a unimodal text classification task. We propose six specialized prompting strategies with Gemini 1.5 Pro, converting visual content into structured textual descriptions, and enabling efficient text-only models to preserve critical visual information. To benchmark our approach, we introduce Uni-Fakeddit-55k, a curated dataset family of 55,000 samples each, each processed through our multimodal-to-unimodal translation framework. Experimental results demonstrate that UNITE-FND achieves 92.52% accuracy in binary classification, surpassing prior multimodal models while reducing computational costs by over 10x (TinyBERT variant: 14.5M parameters vs. 250M+ in SOTA models). Additionally, we propose a comprehensive suite of five novel metrics to evaluate image-to-text conversion quality, ensuring optimal information preservation. Our results demonstrate that structured text-based representations can replace direct multimodal processing with minimal loss of accuracy, making UNITE-FND a practical and scalable alternative for resource-constrained environments.
자기지도 학습(SSL)은 대규모 비레이블 전체 슬라이드 이미지(WSI)로부터 전이 가능한 표현을 학습하는 효과적인 패러다임으로 부상하고 있다. 그러나 기존의 SSL 방법들은 주로 일반적인 시각적 특징을 학습하며, 질환 특성화에 핵심적인 병리 특이적 형태학적 패턴을 명시적으로 포착하는 데 종종 실패한다. 이러한 한계를 해결하기 위해, 우리는 병리 인식 프로토타입 증류를 갖춘 Tiny Vision Transformer인 TVT-PAPD를 제안한다. 이 자기지도 병리 표현 학습 프레임워크는 Tiny Vision Transformer(TVT)와 새로운 병리 인식 프로토타입 증류(PAPD) 모듈을 통합한다. PAPD는 학습 가능한 병리 프로토타입 뱅크를 활용하여 대표적인 조직 형태학적 패턴을 발견하고 보존하며, 의미론적으로 유사한 병리 영역이 일관되고 판별적인 표현을 학습하도록 유도한다. 제안된 프레임워크는 9,000만 개의 파라미터로 계산 효율성을 유지하면서 병리 인식 특징 학습을 향상시킨다. Cancer Genome Atlas(TCGA) 저등급 신경교종(LGG)/교모세포종(GBM) 데이터셋 및 Indian Pathology Brain(IPD-Brain) 데이터셋에 대한 실험에서, TVT-PAPD는 LGG-GBM 분류에서 각각 93.02% 및 90.23%의 가중 F1-점수를 달성하면서 독립적인 신경교종 데이터셋 전반에 걸쳐 강력한 코호트 간 일반화 능력을 보인다.
컴퓨터 비전(CV)과 머신러닝(ML)은 문화유산(CH) 유물 분석에 새로운 도구를 제공하지만, 모델을 구성하고 학습하고 평가하는 데 필요한 배경지식이 부족한 CH 분야 전문가에게 CV/ML 파이프라인은 여전히 대체로 접근하기 어렵다. 본 논문에서는 이러한 격차를 해소하는 오픈소스 CV 플랫폼 AmalthAI를 제안하며, 이를 통해 ML 비전문가인 CH 전문가가 고고학적으로 의미 있는 발견을 독립적으로 도출하고 검증할 수 있도록 한다. 이 인터페이스는 분류, 분할(segmentation), 객체 검출을 위한 데이터셋 관리, 학습, 추론을 포괄하며, 확장 가능한 학습과 하이퍼파라미터 탐색은 Kubeflow와 Katib이 담당한다. Grad-CAM은 예측의 근거가 되는 이미지 영역을 국소화하고, 비전-언어 모델(VLM)은 전문가 검토를 위해 해당 영역에 대한 텍스트 설명을 추가한다. 고고학 데이터는 국가 소유이거나 권리 제약이 있어 기관의 관리 밖으로 반출할 수 없는 경우가 많으므로, AmalthAI는 자체 호스팅이 가능한 배포 방식을 통해 민감한 데이터가 구내에 보관되도록 보장한다. 우리는 점토 직물 압흔(clay textile imprints)으로 구성된 자체 데이터셋을 기반으로 한 고고학 사용 사례에서 이 플랫폼을 시험했으며, 여기서 CH 전문가들은 가설 검증을 위해 분할 모델과 분류 모델을 학습하고 검증했다. 구현 코드는 https://github.com/TEXTaiLES/AmalthAI 에서 제공한다.
실세계 데이터셋은 심각한 클래스 불균형을 보이는 경우가 많으며, 이는 특히 희귀 클래스에서 심층 학습(deep learning) 모델의 성능을 저하시킵니다. 이진 및 다중 클래스 환경에서는 불균형에 대한 연구가 활발히 이루어졌지만, 복잡한 레이블 동시 발생(co-occurrence)으로 인해 다중 레이블(multilabel) 문제에서는 훨씬 더 큰 난제가 됩니다. 심전도(ECG) 분류와 같은 임상 응용 분야에서는 이러한 난제와 함께 두 번째 결정적인 과제가 결부되는데, 이는 위음성(false negative)으로 인한 막대한 비용을 줄이기 위해 유병률과 무관하게 고위험 질환의 우선순위를 지정해야 한다는 점입니다. 본 연구에서는 학습 인스턴스에 대한 분포를 최적화하여 모델의 서로 다른 클래스에 대한 노출을 제어하는 학습 가능한 샘플러인 다중 레이블 클래스 우선순위화 확률론적 샘플러(Multilabel Class-Prioritized Probabilistic Sampler, ML-CLAPPS)를 소개합니다. 이 접근 방식은 다중 레이블 불균형을 해결하는 동시에 사용자 정의 중요도 벡터(importance vector, IV)를 통해 중요한 질환의 우선순위화를 가능하게 합니다. 실세계 다중 레이블 ECG 이미지 데이터셋을 사용한 실증적 평가는 ML-CLAPPS가 최고 수준(SOTA)의 성능을 달성함을 입증합니다. 우리는 본 모델이 지정된 IV에 클래스 노출도를 일치시키며, 특정 클래스의 중요도를 높이면 해당 클래스의 분류 성능이 상응하여 향상됨으로써 임상적으로 중대한 질환의 명시적인 우선순위화가 가능함을 보여줍니다.
Foundation models, pretrained on extensive datasets, have significantly advanced machine learning by providing robust and transferable embeddings applicable to various domains, including medical imaging diagnostics. This study evaluates the utility of embeddings derived from both general-purpose and medical domain-specific foundation models for training lightweight adapter models in multi-class radiography classification, focusing specifically on tube placement assessment. A dataset comprising 8842 radiographs classified into seven distinct categories was employed to extract embeddings using six foundation models: DenseNet121, BiomedCLIP, Med-Flamingo, MedImageInsight, Rad-DINO, and CXR-Foundation. Adapter models were subsequently trained using classical machine learning algorithms. Among these combinations, MedImageInsight embeddings paired with an support vector machine adapter yielded the highest mean area under the curve (mAUC) at 93.8%, followed closely by Rad-DINO (91.1%) and CXR-Foundation (89.0%). In comparison, BiomedCLIP and DenseNet121 exhibited moderate performance with mAUC scores of 83.0% and 81.8%, respectively, whereas Med-Flamingo delivered the lowest performance at 75.1%. Notably, most adapter models demonstrated computational efficiency, achieving training within one minute and inference within seconds on CPU, underscoring their practicality for clinical applications. Furthermore, fairness analyses on adapters trained on MedImageInsight-derived embeddings indicated minimal disparities, with gender differences in performance within 2% and standard deviations across age groups not exceeding 3%. These findings confirm that foundation model embeddings-especially those from MedImageInsight-facilitate accurate, computationally efficient, and equitable diagnostic classification using lightweight adapters for radiographic image analysis.
In this thesis we examined several multimodal feature extraction and learning methods for retrieval and classification purposes. We reread briefly some theoretical results of learning in Section 2 and reviewed several generative and discriminative models in Section 3 while we described the similarity kernel in Section 4. We examined different aspects of the multimodal image retrieval and classification in Section 5 and suggested methods for identifying quality assessments of Web documents in Section 6. In our last problem we proposed similarity kernel for time-series based classification. The experiments were carried over publicly available datasets and source codes for the most essential parts are either open source or released. Since the used similarity graphs (Section 4.2) are greatly constrained for computational purposes, we would like to continue work with more complex, evolving and capable graphs and apply for different problems such as capturing the rapid change in the distribution (e.g. session based recommendation) or complex graphs of the literature work. The similarity kernel with the proper metrics reaches and in many cases improves over the state-of-the-art. Hence we may conclude generative models based on instance similarities with multiple modes is a generally applicable model for classification and regression tasks ranging over various domains, including but not limited to the ones presented in this thesis. More generally, the Fisher kernel is not only unique in many ways but one of the most powerful kernel functions. Therefore we may exploit the Fisher kernel in the future over widely used generative models, such as Boltzmann Machines [Hinton et al., 1984], a particular subset, the Restricted Boltzmann Machines and Deep Belief Networks [Hinton et al., 2006]), Latent Dirichlet Allocation [Blei et al., 2003] or Hidden Markov Models [Baum and Petrie, 1966] to name a few.
The recent popularity of deep neural networks (DNNs) has generated a lot of research interest in performing DNN-related computation efficiently. However, the primary focus is usually very narrow and limited to (i) inference -- i.e. how to efficiently execute already trained models and (ii) image classification networks as the primary benchmark for evaluation. Our primary goal in this work is to break this myopic view by (i) proposing a new benchmark for DNN training, called TBD (TBD is short for Training Benchmark for DNNs), that uses a representative set of DNN models that cover a wide range of machine learning applications: image classification, machine translation, speech recognition, object detection, adversarial networks, reinforcement learning, and (ii) by performing an extensive performance analysis of training these different applications on three major deep learning frameworks (TensorFlow, MXNet, CNTK) across different hardware configurations (single-GPU, multi-GPU, and multi-machine). TBD currently covers six major application domains and eight different state-of-the-art models. We present a new toolchain for performance analysis for these models that combines the targeted usage of existing performance analysis tools, careful selection of new and existing metrics and methodologies to analyze the results, and utilization of domain specific characteristics of DNN training. We also build a new set of tools for memory profiling in all three major frameworks; much needed tools that can finally shed some light on precisely how much memory is consumed by different data structures (weights, activations, gradients, workspace) in DNN training. By using our tools and methodologies, we make several important observations and recommendations on where the future research and optimization of DNN training should be focused.
Background: When using deep learning models, there are many possible vulnerabilities and some of the most worrying are the adversarial inputs, which can cause wrong decisions with minor perturbations. Therefore, it becomes necessary to retrain these models against adversarial inputs, as part of the software testing process addressing the vulnerability to these inputs. Furthermore, for an energy efficient testing and retraining, data scientists need support on which are the best guidance metrics and optimal dataset configurations. Aims: We examined four guidance metrics for retraining convolutional neural networks and three retraining configurations. Our goal is to improve the models against adversarial inputs regarding accuracy, resource utilization and time from the point of view of a data scientist in the context of image classification. Method: We conducted an empirical study in two datasets for image classification. We explore: (a) the accuracy, resource utilization and time of retraining convolutional neural networks by ordering new training set by four different guidance metrics (neuron coverage, likelihood-based surprise adequacy, distance-based surprise adequacy and random), (b) the accuracy and resource utilization of retraining convolutional neural networks with three different configurations (from scratch and augmented dataset, using weights and augmented dataset, and using weights and only adversarial inputs). Results: We reveal that retraining with adversarial inputs from original weights and by ordering with surprise adequacy metrics gives the best model w.r.t. the used metrics. Conclusions: Although more studies are necessary, we recommend data scientists to use the above configuration and metrics to deal with the vulnerability to adversarial inputs of deep learning models, as they can improve their models against adversarial inputs without using many inputs.
Constructing model-agnostic group equivariant networks, such as equitune (Basu et al., 2023b) and its generalizations (Kim et al., 2023), can be computationally expensive for large product groups. We address this problem by providing efficient model-agnostic equivariant designs for two related problems: one where the network has multiple inputs each with potentially different groups acting on them, and another where there is a single input but the group acting on it is a large product group. For the first design, we initially consider a linear model and characterize the entire equivariant space that satisfies this constraint. This characterization gives rise to a novel fusion layer between different channels that satisfies an invariance-symmetry (IS) constraint, which we call an IS layer. We then extend this design beyond linear models, similar to equitune, consisting of equivariant and IS layers. We also show that the IS layer is a universal approximator of invariant-symmetric functions. Inspired by the first design, we use the notion of the IS property to design a second efficient model-agnostic equivariant design for large product groups acting on a single input. For the first design, we provide experiments on multi-image classification where each view is transformed independently with transformations such as rotations. We find equivariant models are robust to such transformations and perform competitively otherwise. For the second design, we consider three applications: language compositionality on the SCAN dataset to product groups; fairness in natural language generation from GPT-2 to address intersectionality; and robust zero-shot image classification with CLIP. Overall, our methods are simple and general, competitive with equitune and its variants, while also being computationally more efficient.
This paper presents one-bit supervision, a novel setting of learning with fewer labels, for image classification. Instead of training model using the accurate label of each sample, our setting requires the model to interact with the system by predicting the class label of each sample and learn from the answer whether the guess is correct, which provides one bit (yes or no) of information. An intriguing property of the setting is that the burden of annotation largely alleviates in comparison to offering the accurate label. There are two keys to one-bit supervision, which are (i) improving the guess accuracy and (ii) making good use of the incorrect guesses. To achieve these goals, we propose a multi-stage training paradigm and incorporate negative label suppression into an off-the-shelf semi-supervised learning algorithm. Theoretical analysis shows that one-bit annotation is more efficient than full-bit annotation in most cases and gives the conditions of combining our approach with active learning. Inspired by this, we further integrate the one-bit supervision framework into the self-supervised learning algorithm which yields an even more efficient training schedule. Different from training from scratch, when self-supervised learning is used for initialization, both hard example mining and class balance are verified effective in boosting the learning performance. However, these two frameworks still need full-bit labels in the initial stage. To cast off this burden, we utilize unsupervised domain adaptation to train the initial model and conduct pure one-bit annotations on the target dataset. In multiple benchmarks, the learning efficiency of the proposed approach surpasses that using full-bit, semi-supervised supervision.
Weakly supervised localization aims at finding target object regions using only image-level supervision. However, localization maps extracted from classification networks are often not accurate due to the lack of fine pixel-level supervision. In this paper, we propose to leverage pixel-level similarities across different objects for learning more accurate object locations in a complementary way. Particularly, two kinds of constraints are proposed to prompt the consistency of object features within the same categories. The first constraint is to learn the stochastic feature consistency among discriminative pixels that are randomly sampled from different images within a batch. The discriminative information embedded in one image can be leveraged to benefit its counterpart with inter-image communication. The second constraint is to learn the global consistency of object features throughout the entire dataset. We learn a feature center for each category and realize the global feature consistency by forcing the object features to approach class-specific centers. The global centers are actively updated with the training process. The two constraints can benefit each other to learn consistent pixel-level features within the same categories, and finally improve the quality of localization maps. We conduct extensive experiments on two popular benchmarks, i.e., ILSVRC and CUB-200-2011. Our method achieves the Top-1 localization error rate of 45.17% on the ILSVRC validation set, surpassing the current state-of-the-art method by a large margin. The code is available at https://github.com/xiaomengyc/I2C.
Analyzing imaging and hyperspectral data is crucial across scientific fields, including biology, medicine, chemistry, and physics. The primary goal is to transform high-resolution or high-dimensional data into an interpretable format to generate actionable insights, aiding decision-making and advancing knowledge. Currently, this task relies on complex, human-designed workflows comprising iterative steps such as denoising, spatial sampling, keypoint detection, feature generation, clustering, dimensionality reduction, and physics-based deconvolutions. The introduction of machine learning over the past decade has accelerated tasks like image segmentation and object detection via supervised learning, and dimensionality reduction via unsupervised methods. However, both classical and NN-based approaches still require human input, whether for hyperparameter tuning, data labeling, or both. The growing use of automated imaging tools, from atomically resolved imaging to biological applications, demands unsupervised methods that optimize data representation for human decision-making or autonomous experimentation. Here, we discuss advances in reward-based workflows, which adopt expert decision-making principles and demonstrate strong transfer learning across diverse tasks. We represent image analysis as a decision-making process over possible operations and identify desiderata and their mappings to classical decision-making frameworks. Reward-driven workflows enable a shift from supervised, black-box models sensitive to distribution shifts to explainable, unsupervised, and robust optimization in image analysis. They can function as wrappers over classical and DCNN-based methods, making them applicable to both unsupervised and supervised workflows (e.g., classification, regression for structure-property mapping) across imaging and hyperspectral data.
비전-언어 모델(VLM)로 의료 영상을 이해할 때는 대개 추론 토큰과 답변 토큰을 자기회귀 방식으로 생성한다. 이 불필요한 토큰들로 인해 응답 길이에 따라 추론 비용이 커진다. 자유 형식 텍스트에서 답을 추출해야 하므로 피할 수 있는 지연과 결정 오류도 발생한다. 본 연구는 의료 영상, 질문, 답변 후보를 한 번의 멀티모달 순전파로 곧바로 결정에 대응시키는 시스템 1 비전-언어 의사결정 모델 MedJev를 소개한다. 자기회귀 생성은 사용하지 않는다. 빠른 추론을 위해 상위 언어 층과 생성 출력 경로를 제거하며, 시각 처리 경로 전체와 모든 시각 토큰은 유지한다. 이처럼 줄인 백본에서 의사결정 헤드는 사전학습된 쿼리-키 매칭으로 각 후보에 점수를 매기고, 가벼운 다층 퍼셉트론(MLP) 잔차로 점수를 다듬는다. 이로써 사전학습한 매칭 구조를 과제에 특화된 비선형 판별과 결합한다. Qwen3.5-4B 기반 MedJev-4B는 의료 영상 분류 및 시각 질의응답 벤치마크 네 개 모두에서 원래 모델보다 더 정확하고 빨랐다. 절대 정확도 기준으로 7.93%포인트 높아졌고 종단 간 처리 속도는 6.04$\times$ 향상됐다. 최신 모델 20개와 추가로 비교한 결과, MedJev-4B는 매개변수 40억~90억 개 범위의 범용 및 의료 VLM 가운데 성능과 추론 속도가 최고 수준이었다. 멀티모달 입력으로 직접 의사결정을 하는 방식이 의료 영상을 빠르고 효과적으로 이해하는 데 유용함을 보여 준다.
Today's general-purpose deep convolutional neural networks (CNN) for image classification and object detection are trained offline on large static datasets. Some applications, however, will require training in real-time on live video streams with a human-in-the-loop. We refer to this class of problem as Time-ordered Online Training (ToOT) - these problems will require a consideration of not only the quantity of incoming training data, but the human effort required to tag and use it. In this paper, we define training benefit as a metric to measure the effectiveness of a sequence in using each user interaction. We demonstrate and evaluate a system tailored to performing ToOT in the field, capable of training an image classifier on a live video stream through minimal input from a human operator. We show that by exploiting the time-ordered nature of the video stream through optical flow-based object tracking, we can increase the effectiveness of human actions by about 8 times.
Convolutional Networks (ConvNets) are powerful models that learn hierarchies of visual features, which could also be used to obtain image representations for transfer learning. The basic pipeline for transfer learning is to first train a ConvNet on a large dataset (source task) and then use feed-forward units activation of the trained ConvNet as image representation for smaller datasets (target task). Our key contribution is to demonstrate superior performance of multiple ConvNet layer features over single ConvNet layer features. Combining multiple ConvNet layer features will result in more complex feature space with some features being repetitive. This requires some form of feature selection. We use AdaBoost with single stumps to implicitly select only distinct features that are useful towards classification from concatenated ConvNet features. Experimental results show that using multiple ConvNet layer activation features instead of single ConvNet layer features consistently will produce superior performance. Improvements becomes significant as we increase the distance between source task and the target task.
Vector-quantized local features frequently used in bag-of-visual-words approaches are the backbone of popular visual recognition systems due to both their simplicity and their performance. Despite their success, bag-of-words-histograms basically contain low-level image statistics (e.g., number of edges of different orientations). The question remains how much visual information is "lost in quantization" when mapping visual features to code words? To answer this question, we present an in-depth analysis of the effect of local feature quantization on human recognition performance. Our analysis is based on recovering the visual information by inverting quantized local features and presenting these visualizations with different codebook sizes to human observers. Although feature inversion techniques are around for quite a while, to the best of our knowledge, our technique is the first visualizing especially the effect of feature quantization. Thereby, we are now able to compare single steps in common image classification pipelines to human counterparts.
Understanding how cities visually differ from each others is interesting for planners, residents, and historians. We investigate the interpretation of deep features learned by convolutional neural networks (CNNs) for city recognition. Given a trained city recognition network, we first generate weighted masks using the known Grad-CAM technique and to select the most discriminate regions in the image. Since the image classification label is the city name, it contains no information of objects that are class-discriminate, we investigate the interpretability of deep representations with two methods. (i) Unsupervised method is used to cluster the objects appearing in the visual explanations. (ii) A pretrained semantic segmentation model is used to label objects in pixel level, and then we introduce statistical measures to quantitatively evaluate the interpretability of discriminate objects. The influence of network architectures and random initializations in training, is studied on the interpretability of CNN features for city recognition. The results suggest that network architectures would affect the interpretability of learned visual representations greater than different initializations.
It has been recently shown that the hidden variables of convolutional neural networks make for an efficient perceptual similarity metric that accurately predicts human judgment on relative image similarity assessment. First, we show that such learned perceptual similarity metrics (LPIPS) are susceptible to adversarial attacks that dramatically contradict human visual similarity judgment. While this is not surprising in light of neural networks' well-known weakness to adversarial perturbations, we proceed to show that self-ensembling with an infinite family of random transformations of the input --- a technique known not to render classification networks robust --- is enough to turn the metric robust against attack, while retaining predictive power on human judgments. Finally, we study the geometry imposed by our our novel self-ensembled metric (E-LPIPS) on the space of natural images. We find evidence of "perceptual convexity" by showing that convex combinations of similar-looking images retain appearance, and that discrete geodesics yield meaningful frame interpolation and texture morphing, all without explicit correspondences.