The effective diagnosis of acute and hard-to-heal wounds is crucial for wound care practitioners to provide effective patient care. Poor clinical outcomes are often linked to infection, peripheral vascular disease, and increasing wound depth, which collectively exacerbate these comorbidities. However, diagnostic tools based on Artificial Intelligence (AI) speed up the interpretation of medical images and improve early detection of disease. In this article, we propose a multi-modal AI model based on transfer learning (TL), which combines two state-of-the-art architectures, Xception and GMRNN, for wound classification. The multi-modal network is developed by concatenating the features extracted by a transfer learning algorithm and location features to classify the wound types of diabetic, pressure, surgical, and venous ulcers. The proposed method is comprehensively compared with deep neural networks (DNN) for medical image analysis. The experimental results demonstrate a notable wound-class classifications (containing only diabetic, pressure, surgical, and venous) vary from 78.77 to 100\% in various experiments. The results presented in this study showcase the exceptional accuracy of the proposed methodology in accurately classifying the most commonly occurring wound types using wound images and their corresponding locations.
유방 조영술(mammography) 및 흉부 방사선 촬영(chest radiography)과 같은 다중 뷰 영상화는 임상 진료의 표준 구성 요소입니다. 그러나 의료 이미지는 종종 정렬되지 않은(unregistered) 상태이며, 진단적으로 관련된 소견을 모호하게 만들 수 있는 뷰 특이적 아티팩트(artifacts)나 관련 없는 배경 단서를 포함하고 있습니다. 많은 기존 방법들은 뷰별 표현(representation)을 직접 융합하여, 이러한 관련 없는 내용이 융합된 임베딩(embedding)을 오염시키고 다양한 뷰 구성에서의 견고성(robustness)을 저하시킵니다. 우리는 다중 뷰 분류를 위해 융합 전 패치 토큰(token)을 정제하는, 최적 전송(optimal transport, OT)을 기반으로 한 신뢰도 인지 잠재 허브 토큰 정렬 모듈인 OTCHA를 제안합니다. OTCHA는 뷰 간에 공유되는 학습 가능한 잠재 허브 토큰 세트를 도입합니다. 각 뷰에 대해 특징(feature) 유사성과 기하학(geometry)을 함께 고려하여 패치 토큰과 허브 토큰 사이의 OT 계획(OT plan)을 계산하고, 부분 일치를 가능하게 하며 관련 없는 토큰을 버리기 위해 토큰 조건부 더스트빈(token-conditional dustbins)으로 OT 공식을 보강합니다. 결과적으로 생성된 전송 계획은 토큰 수준의 일치 신뢰도를 제공하며, 이는 허브 매개 메시지 전달(message passing)을 게이팅(gating)하고 정제를 안정화하기 위해 새로운 최적 전송 기반 표현 정렬 손실(loss)에 가중치를 부여합니다. 세 가지 다중 뷰 의료 이미지 데이터셋(dataset)에 대한 실험은 다양한 해부학적 구조와 뷰 구성에 걸쳐 경쟁 기준선(baselines) 대비 일관된 개선을 입증합니다. 우리의 코드는 https://github.com/labhai/OTCHA 에서 확인할 수 있습니다.
생성 모델이 지속적으로 발전함에 따라, AI 생성 이미지 탐지기는 이전 도메인에서 습득한 지식을 보존하면서 새로 등장하는 생성 도메인에 점진적으로 적응해야 합니다. 이러한 지속 학습(continual learning) 환경은 포렌식 흔적이 미세하고 생성기마다 고유하여, 탐지기가 치명적 망각(catastrophic forgetting)에 매우 취약해지기 때문에 특히 어렵습니다. 기존 방법들은 특징(feature) 표현을 안정화하는 방식으로 이 문제를 주로 다루며, 망각을 표현 수준의 문제로 암묵적으로 취급합니다. 본 논문에서는 이러한 관점이 불완전함을 보입니다. 특징 표현이 판별력을 유지하더라도 새로운 도메인에 대해 분류(classification) 헤드가 지속적으로 최적화됨에 따라 결정 경계(decision boundary)가 점진적으로 표류할 수 있음을 입증합니다. 이 두 가지 효과는 결합하여 이중 저하(Dual Degradation)라고 명명된 복합 실패 모드를 발생시킵니다. 이러한 문제를 극복하기 위해, 본 논문에서는 표현 및 결정 수준의 망각을 함께 완화하는 디커플링된 지속 탐지 프레임워크인 DECODE를 제안합니다. 구체적으로, 다양한 포렌식 표현을 보존하기 위한 부분공간 다양성 정규화(Subspace Diversity Regularization, SDR)와 수동 하이퍼파라미터(hyperparameter) 튜닝 없이 각 어댑터 병합 후 공유 분류 헤드를 재보정하는 닫힌 형식 결정 정렬(Closed-Form Decision Alignment, CDA)을 도입합니다. 19개 생성 도메인에 대한 광범위한 실험 결과, DECODE는 단 0.39%의 망각만으로 99.36%의 평균 정확도를 달성하며, 나아가 11개의 보지 못한 생성기에 대해서도 95.36%의 정확도로 일반화됨을 보여줍니다.
심층 신경망(deep neural network)은 자연스러운 태스크(task)에 대해 일반화하는 해(solution)를 학습할 수도 있고, 임의의 매핑(mapping)을 암기할 수도 있다. 우리는 전자를 이해하기 위해서는 상대적으로 훨씬 덜 연구된 후자를 이해하는 것이 필요하다고 주장한다. 일반화하는 네트워크와 암기하는 네트워크는 그 행동이나 학습 동역학(training dynamics)에서 어떤 방식으로 다른가? 암기하는 네트워크는 단순히 입력-출력 매핑이 독립적으로 저장되는 룩업 테이블(lookup table)처럼 동작하는가? 우리는 무작위화된 레이블(label) 또는 픽셀(pixel)로 이미지 분류 태스크를 학습한 CNN을 대상으로 이러한 질문들을 조사한다. 먼저 이러한 암기 네트워크가 보지 않은 이미지에 대해 어떻게 동작하는지를 연구한다. 우리는 동일한 무작위 레이블로 학습되었더라도 독립적으로 초기화된 네트워크들은 서로 다른 예측 함수를 학습한다는 것을 발견했다: 이들은 일반화하는 네트워크와 달리 테스트 예제에 대해 서로 다른 예측을 한다. 다음으로 우리는 학습 데이터셋을 보간(interpolate)하는 데 필요한 네트워크 용량(capacity)과 학습 시간을 통해, 데이터셋 크기가 암기에 미치는 영향을 살펴본다. 직관과 반대로, 우리는 입력-출력 쌍이 공통된 구조를 전혀 공유하지 않는 경우에도, 주어진 예제를 완전히 암기하는 데 필요한 제시 횟수(number of presentations)가 데이터셋 크기가 커질수록 오히려 감소할 수 있음을 발견했다. 암기하는 네트워크는 보편적인 해를 학습하지는 않지만, 그 기저 메커니즘에 대한 추가 연구를 요구하는 흥미로운 행동들을 보인다.
In recent years, deep learning has successfully been applied to automate a wide variety of tasks in diagnostic histopathology. However, fast and reliable localization of small-scale regions-of-interest (ROI) has remained a key challenge, as discriminative morphologic features often occupy only a small fraction of a gigapixel-scale whole-slide image (WSI). In this paper, we propose a sparse WSI analysis method for the rapid identification of high-power ROI for WSI-level classification. We develop an evaluation framework inspired by the early classification literature, in order to quantify the tradeoff between diagnostic performance and inference time for sparse analytic approaches. We test our method on a common but time-consuming task in pathology - that of diagnosing gastric intestinal metaplasia (GIM) on hematoxylin and eosin (H&E)-stained slides from endoscopic biopsy specimens. GIM is a well-known precursor lesion along the pathway to development of gastric cancer. We performed a thorough evaluation of the performance and inference time of our approach on a test set of GIM-positive and GIM-negative WSI, finding that our method successfully detects GIM in all positive WSI, with a WSI-level classification area under the receiver operating characteristic curve (AUC) of 0.98 and an average precision (AP) of 0.95. Furthermore, we show that our method can attain these metrics in under one minute on a standard CPU. Our results are applicable toward the goal of developing neural networks that can easily be deployed in clinical settings to support pathologists in quickly localizing and diagnosing small-scale morphologic features in WSI.
When training a deep neural network for image classification, one can broadly distinguish between two types of latent features of images that will drive the classification. We can divide latent features into (i) "core" or "conditionally invariant" features $X^\text{core}$ whose distribution $X^\text{core}\vert Y$, conditional on the class $Y$, does not change substantially across domains and (ii) "style" features $X^{\text{style}}$ whose distribution $X^{\text{style}} \vert Y$ can change substantially across domains. Examples for style features include position, rotation, image quality or brightness but also more complex ones like hair color, image quality or posture for images of persons. Our goal is to minimize a loss that is robust under changes in the distribution of these style features. In contrast to previous work, we assume that the domain itself is not observed and hence a latent variable. We do assume that we can sometimes observe a typically discrete identifier or "$\mathrm{ID}$ variable". In some applications we know, for example, that two images show the same person, and $\mathrm{ID}$ then refers to the identity of the person. The proposed method requires only a small fraction of images to have $\mathrm{ID}$ information. We group observations if they share the same class and identifier $(Y,\mathrm{ID})=(y,\mathrm{id})$ and penalize the conditional variance of the prediction or the loss if we condition on $(Y,\mathrm{ID})$. Using a causal framework, this conditional variance regularization (CoRe) is shown to protect asymptotically against shifts in the distribution of the style variables. Empirically, we show that the CoRe penalty improves predictive accuracy substantially in settings where domain changes occur in terms of image quality, brightness and color while we also look at more complex changes such as changes in movement and posture.
Cryo-electron microscopy (EM) single particle reconstruction is an entirely general technique for 3D structure determination of macromolecular complexes. However, because the images are taken at low electron dose, it is extremely hard to visualize the individual particle with low contrast and high noise level. In this paper, we propose a novel approach called multi-frequency vector diffusion maps (MFVDM) to improve the efficiency and accuracy of cryo-EM 2D image classification and denoising. This framework incorporates different irreducible representations of the estimated alignment between similar images. In addition, we propose a graph filtering scheme to denoise the images using the eigenvalues and eigenvectors of the MFVDM matrices. Through both simulated and publicly available real data, we demonstrate that our proposed method is efficient and robust to noise compared with the state-of-the-art cryo-EM 2D class averaging and image restoration algorithms.
We propose flexgrid2vec, a novel approach for image representation learning. Existing visual representation methods suffer from several issues, including the need for highly intensive computation, the risk of losing in-depth structural information and the specificity of the method to certain shapes or objects. flexgrid2vec converts an image to a low-dimensional feature vector. We represent each image with a graph of flexible, unique node locations and edge distances. flexgrid2vec is a multi-channel GCN that learns features of the most representative image patches. We have investigated both spectral and non-spectral implementations of the GCN node-embedding. Specifically, we have implemented flexgrid2vec based on different node-aggregation methods, such as vector summation, concatenation and normalisation with eigenvector centrality. We compare the performance of flexgrid2vec with a set of state-of-the-art visual representation learning models on binary and multi-class image classification tasks. Although we utilise imbalanced, low-size and low-resolution datasets, flexgrid2vec shows stable and outstanding results against well-known base classifiers. flexgrid2vec achieves 96.23% on CIFAR-10, 83.05% on CIFAR-100, 94.50% on STL-10, 98.8% on ASIRRA and 89.69% on the COCO dataset.
It has been shown that the activations invoked by an image within the top layers of a large convolutional neural network provide a high-level descriptor of the visual content of the image. In this paper, we investigate the use of such descriptors (neural codes) within the image retrieval application. In the experiments with several standard retrieval benchmarks, we establish that neural codes perform competitively even when the convolutional neural network has been trained for an unrelated classification task (e.g.\ Image-Net). We also evaluate the improvement in the retrieval performance of neural codes, when the network is retrained on a dataset of images that are similar to images encountered at test time. We further evaluate the performance of the compressed neural codes and show that a simple PCA compression provides very good short codes that give state-of-the-art accuracy on a number of datasets. In general, neural codes turn out to be much more resilient to such compression in comparison other state-of-the-art descriptors. Finally, we show that discriminative dimensionality reduction trained on a dataset of pairs of matched photographs improves the performance of PCA-compressed neural codes even further. Overall, our quantitative experiments demonstrate the promise of neural codes as visual descriptors for image retrieval.
Research connecting text and images has recently seen several breakthroughs, with models like CLIP, DALL-E 2, and Stable Diffusion. However, the connection between text and other visual modalities, such as lidar data, has received less attention, prohibited by the lack of text-lidar datasets. In this work, we propose LidarCLIP, a mapping from automotive point clouds to a pre-existing CLIP embedding space. Using image-lidar pairs, we supervise a point cloud encoder with the image CLIP embeddings, effectively relating text and lidar data with the image domain as an intermediary. We show the effectiveness of LidarCLIP by demonstrating that lidar-based retrieval is generally on par with image-based retrieval, but with complementary strengths and weaknesses. By combining image and lidar features, we improve upon both single-modality methods and enable a targeted search for challenging detection scenarios under adverse sensor conditions. We also explore zero-shot classification and show that LidarCLIP outperforms existing attempts to use CLIP for point clouds by a large margin. Finally, we leverage our compatibility with CLIP to explore a range of applications, such as point cloud captioning and lidar-to-image generation, without any additional training. Code and pre-trained models are available at https://github.com/atonderski/lidarclip.
We propose an approach to detect flying objects such as UAVs and aircrafts when they occupy a small portion of the field of view, possibly moving against complex backgrounds, and are filmed by a camera that itself moves. Solving such a difficult problem requires combining both appearance and motion cues. To this end we propose a regression-based approach to motion stabilization of local image patches that allows us to achieve effective classification on spatio-temporal image cubes and outperform state-of-the-art techniques. As the problem is relatively new, we collected two challenging datasets for UAVs and Aircrafts, which can be used as benchmarks for flying objects detection and vision-guided collision avoidance.
Dense local descriptors and machine learning have been used with success in several applications, like classification of textures, steganalysis, and forgery detection. We develop a new image forgery detector building upon some descriptors recently proposed in the steganalysis field suitably merging some of such descriptors, and optimizing a SVM classifier on the available training set. Despite the very good performance, very small forgeries are hardly ever detected because they contribute very little to the descriptors. Therefore we also develop a simple, but extremely specific, copy-move detector based on region matching and fuse decisions so as to reduce the missing detection rate. Overall results appear to be extremely encouraging.
Memes often merge visuals with brief text to share humor or opinions, yet some memes contain harmful messages such as hate speech. In this paper, we introduces MemeBLIP2, a light weight multimodal system that detects harmful memes by combining image and text features effectively. We build on previous studies by adding modules that align image and text representations into a shared space and fuse them for better classification. Using BLIP-2 as the core vision-language model, our system is evaluated on the PrideMM datasets. The results show that MemeBLIP2 can capture subtle cues in both modalities, even in cases with ironic or culturally specific content, thereby improving the detection of harmful material.
Deep convolutional neural networks have recently achieved great success on image aesthetics assessment task. In this paper, we propose an efficient method which takes the global, local and scene-aware information of images into consideration and exploits the composite features extracted from corresponding pretrained deep learning models to classify the derived features with support vector machine. Contrary to popular methods that require fine-tuning or training a new model from scratch, our training-free method directly takes the deep features generated by off-the-shelf models for image classification and scene recognition. Also, we analyzed the factors that could influence the performance from two aspects: the architecture of the deep neural network and the contribution of local and scene-aware information. It turns out that deep residual network could produce more aesthetics-aware image representation and composite features lead to the improvement of overall performance. Experiments on common large-scale aesthetics assessment benchmarks demonstrate that our method outperforms the state-of-the-art results in photo aesthetics assessment.
We address representation learning for large-scale instance-level image retrieval. Apart from backbone, training pipelines and loss functions, popular approaches have focused on different spatial pooling and attention mechanisms, which are at the core of learning a powerful global image representation. There are different forms of attention according to the interaction of elements of the feature tensor (local and global) and the dimensions where it is applied (spatial and channel). Unfortunately, each study addresses only one or two forms of attention and applies it to different problems like classification, detection or retrieval. We present global-local attention module (GLAM), which is attached at the end of a backbone network and incorporates all four forms of attention: local and global, spatial and channel. We obtain a new feature tensor and, by spatial pooling, we learn a powerful embedding for image retrieval. Focusing on global descriptors, we provide empirical evidence of the interaction of all forms of attention and improve the state of the art on standard benchmarks.
Small on-device models have been successfully trained with user-level differential privacy (DP) for next word prediction and image classification tasks in the past. However, existing methods can fail when directly applied to learn embedding models using supervised training data with a large class space. To achieve user-level DP for large image-to-embedding feature extractors, we propose DP-FedEmb, a variant of federated learning algorithms with per-user sensitivity control and noise addition, to train from user-partitioned data centralized in the datacenter. DP-FedEmb combines virtual clients, partial aggregation, private local fine-tuning, and public pretraining to achieve strong privacy utility trade-offs. We apply DP-FedEmb to train image embedding models for faces, landmarks and natural species, and demonstrate its superior utility under same privacy budget on benchmark datasets DigiFace, EMNIST, GLD and iNaturalist. We further illustrate it is possible to achieve strong user-level DP guarantees of $\epsilon<4$ while controlling the utility drop within 5%, when millions of users can participate in training.
Cervical spine fractures constitute a critical medical emergency, with the potential for lifelong paralysis or even fatality if left untreated or undetected. Over time, these fractures can deteriorate without intervention. To address the lack of research on the practical application of deep learning techniques for the detection of spine fractures, this study leverages a dataset containing both cervical spine fractures and non-fractured computed tomography images. This paper introduces a two-stage pipeline designed to identify the presence of cervical vertebrae in each image slice and pinpoint the location of fractures. In the first stage, a multi-input network, incorporating image and image metadata, is trained. This network is based on the Global Context Vision Transformer, and its performance is benchmarked against popular deep learning image classification model. In the second stage, a YOLOv8 model is trained to detect fractures within the images, and its effectiveness is compared to YOLOv5. The obtained results indicate that the proposed algorithm significantly reduces the workload of radiologists and enhances the accuracy of fracture detection.
3D Shape represented as point cloud has achieve advancements in multimodal pre-training to align image and language descriptions, which is curial to object identification, classification, and retrieval. However, the discrete representations of point cloud lost the object's surface shape information and creates a gap between rendering results and 2D correspondences. To address this problem, we propose GS-CLIP for the first attempt to introduce 3DGS (3D Gaussian Splatting) into multimodal pre-training to enhance 3D representation. GS-CLIP leverages a pre-trained vision-language model for a learned common visual and textual space on massive real world image-text pairs and then learns a 3D Encoder for aligning 3DGS optimized per object. Additionally, a novel Gaussian-Aware Fusion is proposed to extract and fuse global explicit feature. As a general framework for language-image-3D pre-training, GS-CLIP is agnostic to 3D backbone networks. Experiments on challenging shows that GS-CLIP significantly improves the state-of-the-art, outperforming the previously best results.
신경 세포자동기(NCA)는 극히 작은 모델 크기에서도 정확하고 견고한 추론을 가능하게 하는 새로운 유형의 신경망 아키텍처이다. 최근 NCA는 이미지 분석, 합성 이미지 생성, 시뮬레이션 등 다양한 작업에서 합성곱 및 어텐션 기반 아키텍처에 대한 흥미로운 저자원 대안으로 발전해 왔다. 급속한 발전과 연구 관심의 증가는 이 신흥 기술에 대한 포괄적인 검토를 필요로 한다. 본 리뷰는 NCA의 기초, 의료 영상에 대한 응용, 그리고 최신 동향에 대한 통찰을 개괄적으로 제공한다. 우리는 최초로 제안된 NCA 아키텍처에 대한 최근 수정 사항을 효율성과 정확도 측면에서 분석한다. 또한 의료 영상 분석, 세분화(segmentation), 분류, 정합(registration), 깊이 추정, 이미지 합성에 초점을 맞추어 실제 시나리오에서의 실용적 응용을 검토한다. 마지막으로 NCA의 여러 장점과 연구 공백을 식별하고, 제한된 환경이거나 견고성 또는 효율적인 데이터 처리에 특별한 요구가 있는 영역에서의 의료 응용을 위한 NCA의 미래 기회를 분석하며 결론을 내린다.
Weakly supervised learning with noisy data has drawn attention in the medical imaging community due to the sparsity of high-quality disease labels. However, little is known about the limitations of such weakly supervised learning and the effect of these constraints on disease classification performance. In this paper, we test the effects of such weak supervision by examining model tolerance for three conditions. First, we examined model tolerance for noisy data by incrementally increasing error in the labels within the training data. Second, we assessed the impact of dataset size by varying the amount of training data. Third, we compared performance differences between binary and multi-label classification. Results demonstrated that the model could endure up to 10% added label error before experiencing a decline in disease classification performance. Disease classification performance steadily rose as the amount of training data was increased for all disease classes, before experiencing a plateau in performance at 75% of training data. Last, the binary model outperformed the multilabel model in every disease category. However, such interpretations may be misleading, as the binary model was heavily influenced by co-occurring diseases and may not have learned the specific features of the disease in the image. In conclusion, this study may help the medical imaging community understand the benefits and risks of weak supervision with noisy labels. Such studies demonstrate the need to build diverse, large-scale datasets and to develop explainable and responsible AI.