The Caltech-UCSD Birds-200-2011 (CUB-200-2011) dataset is the most widely-used dataset for fine-grained visual categorization task. It contains 11,788 images of 200 subcategories belonging to birds, 5,994 for training and 5,794 for testing. Each image has detailed annotations: 1 subcategory label, 15 part locations, 312 binary attributes and 1 bounding box. The textual information comes from Reed et al.. They expand the CUB-200-2011 dataset by collecting fine-grained natural language descriptions. Ten single-sentence descriptions are collected for each image. The natural language descriptions are collected through the Amazon Mechanical Turk (AMT) platform, and are required at least 10 words, without any information of subcategories and actions. Source: Fine-grained Visual-textual Representation Learning Image Source: http://www.vision.caltech.edu/visipedia/CUB-200-2011.html

The Food-101 dataset consists of 101 food categories with 750 training and 250 test images per category, making a total of 101k images. The labels for the test images have been manually cleaned, while the training set contains some noise. Source: Combining Weakly and Webly Supervised Learning for Classifying Food Images Image Source: https://data.vision.ee.ethz.ch/cvl/datasets_extra/food-101/

Science Question Answering (ScienceQA) is a new benchmark that consists of 21,208 multimodal multiple choice questions with diverse science topics and annotations of their answers with corresponding lectures and explanations. Out of the questions in ScienceQA, 10,332 (48.7%) have an image context, 10,220 (48.2%) have a text context, and 6,532 (30.8%) have both. Most questions are annotated with grounded lectures (83.9%) and detailed explanations (90.5%). The lecture and explanation provide general external knowledge and specific reasons, respectively, for arriving at the correct answer. To the best of our knowledge, ScienceQA is the first large-scale multimodal dataset that annotates lectures and explanations for the answers. ScienceQA, in contrast to previous datasets, has richer domain diversity from three subjects: natural science, language science, and social science. Questions in each subject are categorized first by the topic (Biology, Physics, Chemistry, etc.), then by the category (Plants, Cells, Animals, etc.), and finally by the skill (Classify fruits and vegetables as plant parts, Identify countries of Africa, etc.). ScienceQA features 26 topics, 127 categories, and 379 skills that cover a wide range of domains.

We propose VALSE (Vision And Language Structured Evaluation), a novel benchmark designed for testing general-purpose pretrained vision and language (V&L) models for their visio-linguistic grounding capabilities on specific linguistic phenomena. VALSE offers a suite of six tests covering various linguistic constructs. Solving these requires models to ground linguistic phenomena in the visual modality, allowing more fine-grained evaluations than hitherto possible. We expect VALSE to serve as an important benchmark to measure future progress of pretrained V&L models from a linguistic perspective, complementing the canonical task-centred V&L evaluations.

Recent accelerations in multi-modal applications have been made possible with the plethora of image and text data available online. However, the scarcity of similar data in the medical field, specifically in histopathology, has halted similar progress. To enable similar representation learning for histopathology, we turn to YouTube, an untapped resource of videos, offering 1,087 hours of valuable educational histopathology videos from expert clinicians. From YouTube, we curate Quilt: a large-scale vision-language dataset consisting of 768,826 image and text pairs. Quilt was automatically curated using a mixture of models, including large language models), handcrafted algorithms, human knowledge databases, and automatic speech recognition. In comparison, the most comprehensive datasets curated for histopathology amass only around 200K samples. We combine Quilt with datasets, from other sources, including Twitter, research papers, and the internet in general, to create an even larger dataset: Quilt-1M, with 1M paired image-text samples, marking it as the largest vision-language histopathology dataset to date. We demonstrate the value of Quilt-1M by fine-tuning a pre-trained CLIP model. Our model outperforms state-of-the-art models on both zero-shot and linear probing tasks for classifying new pathology images across 13 diverse patch-level datasets of 8 different sub-pathologies and cross-modal retrieval tasks.

Clinical diagnosis of the eye is performed over multifarious data modalities including scalar clinical labels, vectorized biomarkers, two-dimensional fundus images, and three-dimensional Optical Coherence Tomography (OCT) scans. While the clinical labels, fundus images and OCT scans are instrumental measurements, the vectorized biomarkers are interpreted attributes from the other measurements. Clinical practitioners use all these data modalities for diagnosing and treating eye diseases like Diabetic Retinopathy (DR) or Diabetic Macular Edema (DME). Enabling usage of machine learning algorithms within the ophthalmic medical domain requires research into the relationships and interactions between these relevant data modalities. Existing datasets are limited in that: (i) they view the problem as disease prediction without assessing biomarkers, and (ii) they do not consider the explicit relationship among all four data modalities over the treatment period. In this paper, we introduce the Ophthalmic Labels for Investigating Visual Eye Semantics (OLIVES) dataset that addresses the above limitations. This is the first OCT and fundus dataset that includes clinical labels, biomarker labels, and time-series patient treatment information from associated clinical trials. The dataset consists of 1268 fundus eye images each with 49 OCT scans, and 16 biomarkers, along with 3 clinical labels and a disease diagnosis of DR or DME. In total, there are 96 eyes' data averaged over a period of at least two years with each eye treated for an average of 66 weeks and 7 injections. OLIVES dataset has advantages in other fields of machine learning research including self-supervised learning as it provides alternate augmentation schemes that are medically grounded.

The dataset contains single-shot videos taken from moving cameras in underwater environments. The first shard of a new Marine Video Kit dataset is presented to serve for video retrieval and other computer vision challenges. In addition to basic meta-data statistics, we present several insights based on low-level features as well as semantic annotations of selected keyframes. 1379 videos with a length from 2 s to 4.95 min, with the mean and median duration of each video is 29.9 s, and 25.4 s, respectively. We capture data from 11 different regions and countries during the time from 2011 to 2022.

As part of an ongoing worldwide effort to comprehend and monitor insect biodiversity, we present the BIOSCAN-5M Insect dataset to the machine learning community. BIOSCAN-5M is a comprehensive dataset containing multi-modal information for over 5 million insect specimens, and it significantly expands existing image-based biological datasets by including taxonomic labels, raw nucleotide barcode sequences, assigned barcode index numbers, geographical information, and specimen size. Every record has both image and DNA data. Each record of the BIOSCAN-5M dataset contains six primary attributes: RGB image DNA barcode sequence Barcode Index Number (BIN) Biological taxonomic classification Geographical information Specimen size

##MUTE This is the first open-source Bengali Hateful Meme dataset, consisting of around 4200 memes annotated with two labels: hate and not hate.

LUMA is a multimodal dataset that consists of audio, image, and text modalities. It allows controlled injection of uncertainties into the data and is mainly intended for studying uncertainty quantification in multimodal classification settings. This repository provides the Audio and Text modalities. The image modality consists of images from CIFAR-10/100 datasets. To download the image modality and compile the dataset with a specified amount of uncertainties, please use the LUMA compilation tool.

Multimodal object recognition is still an emerging field. Thus, publicly available datasets are still rare and of small size. This dataset was developed to help fill this void and presents multimodal data for 63 objects with some visual and haptic ambiguity. The dataset contains visual, kinesthetic and tactile (audio/vibrations) data. To completely solve sensory ambiguity, sensory integration/fusion would be required. This report describes the creation and structure of the dataset. The first section explains the underlying approach used to capture the visual and haptic properties of the objects. The second section describes the technical aspects (experimental setup) needed for the collection of the data. The third section introduces the objects, while the final section describes the structure and content of the dataset.

Boombox is a multi-modal dataset for visual reconstruction from acoustic vibrations. Involves dropping objects into a box and capturing resulting images and vibrations. Used for training ML systems that predict images from vibration. Potential application domain: Computer Vision, Multimodal Perception, Vision and Sound, Sight from Sound, Robotics, Deep Learning, and Machine Learning.

Correlated Corrupted Dataset is an evaluation set that consists of realistic visible-infrared (V-I) corruptions allowing for models' corruption robustness evaluation. Initially proposed for multimodal person re-identification, our dataset can also be used for the evaluation of V-I cross-modal approaches. Corruptions of the visible modality are the twenty corruptions proposed by Chen & al. in the "Benchmarks for Corruption Invariant Person Re-identification" paper. Corruptions of the infrared modalities have been proposed in our paper, introducing 19 corruptions that respect the infrared modality encoding. In practice, for co-located visible-infrared cameras, weather-related corruptions should, for example, affect each camera. Also, blur-related corruption would likely occur in both visible and infrared cameras. This dataset tackles this aspect by considering the eventual correlations that may occur from one modality camera to another.

We construct Gaze-CIFAR-10, a gaze-augmented image dataset based on the standard CIFAR-10 benchmark, enhanced with human eye-tracking annotations collected using the HTC VIVE Pro Eye headset. The original CIFAR-10 dataset consists of 60,000 color images across 10 categories, each with a resolution of $32 \times 32$ pixels. To enable reliable human gaze tracking, all images are upsampled to $1024 \times 1024$ using the Real-ESRGAN model. For each image, we collect a sequence of 176 eye-gaze coordinates, normalized to the range $[0, 224]$ (with the lower-left corner of the image as the origin) to match the input resolution of Vision Transformers (ViTs). Gaze data were collected from 20 participants using an HTC VIVE Pro Eye headset. The dataset contains 50,000 training images and 10,000 test images, with corresponding synchronized gaze trajectories.

We provide a custom synthetic bimodal dataset, called GeBiD, designed specifically for the comparison of the joint- and cross-generative capabilities of Multimodal Variational Autoencoders. It comprises RGB images of geometric primitives and textual descriptions. The dataset offers 5 levels of difficulty (based on the number of attributes) to find the minimal functioning scenario for each model. Moreover, its rigid structure enables automatic qualitative evaluation of the generated samples.

Vision-language supervised fine-tuning effectively enhances VLLM performance, but existing visual instruction tuning datasets have limitations: 1. Instruction Annotation Quality: Despite strong performance, advanced VLLMs may generate instructions with inaccuracies, such as hallucinations. 2. Instruction and Image Diversity: Limited instruction types and lack of diverse image data impact the model's ability to generate varied and realistic outputs. MMInstruct Dataset To address these challenges, we created the MMInstruct dataset, featuring: - 973K instructions from 24 domains - Four instruction types: Judgement, Multiple-Choice, Long Visual Question Answering, and Short Visual Question Answering.

Dataset for multimodal skills assessment focusing on assessing piano player’s skill level. Annotations include player's skills level, and song difficulty level. Bounding box annotations around pianists' hands are also provided.

Recent advances in large language models have led to the development of multimodal LLMs (MLLMs), which take both image data and text as an input. Virtually all of these models have been announced within the past year, leading to a significant need for benchmarks evaluating the abilities of these models to reason truthfully and accurately on a diverse set of tasks. When Google announced Gemini (Gemini Team et al., 2023), they showcased its ability to solve rebuses—wordplay puzzles which involve creatively adding and subtracting letters from words derived from text and images. The diversity of rebuses allows for a broad evaluation of multimodal reasoning capabilities, including image recognition, multi- step reasoning, and understanding the human creator’s intent. We present REBUS: a collection of 333 hand-crafted rebuses spanning 13 diverse cate- gories, including hand-drawn and digital images created by nine contributors. Samples are presented in Table 1. Notably, GPT-4V, the most powerful model we evaluated, answered only 24% of puzzles correctly, highlighting the poor capabilities of MLLMs in new and unex- pected domains to which human reasoning generalizes with comparative ease. Open-source models perform even worse, with a median accuracy below 1%. We notice that models often give faithless explanations, fail to change their minds after an initial approach doesn’t work, and remain highly uncalibrated on their own abilities.

Uncorrelated Corrupted Dataset is an evaluation set that consists of realistic visible-infrared (V-I) corruptions allowing for models' corruption robustness evaluation. Initially proposed for multimodal person re-identification, our dataset can also be used for the evaluation of V-I cross-modal approaches. Corruptions of the visible modality are the twenty corruptions proposed by Chen & al. in the "Benchmarks for Corruption Invariant Person Re-identification" paper. Corruptions of the infrared modalities have been proposed in our paper, introducing 19 corruptions that respect the infrared modality encoding. In practice, the corruptions are applied randomly and independently to the visible and the infrared cameras, making it more suited to a not co-located camera setting.

WebLI (Web Language Image) is a web-scale multilingual image-text dataset, designed to support Google’s vision-language research, such as the large-scale pre-training for image understanding, image captioning, visual question answering, object detection etc. The dataset is built from the public web, including image bytes, image-associated texts (alt-text, OCR, page title), 109 languages and many other features. The dataset is deduplicated on 68 common vision/vision-language tasks, and has no user or personally identifiable data with careful RAI considerations. source: PaLI: A Jointly-Scaled Multilingual Language-Image Model

This dataset endeavors to fill the research void by presenting a meticulously curated collection of misogynistic memes in a code-mixed language of Hindi and English. It introduces two sub-tasks: the first entails a binary classification to determine the presence of misogyny in a meme, while the second task involves categorizing the misogynistic memes into multiple labels, including Objectification, Prejudice, and Humiliation. For more Information and Citation: Singh, A., Sharma, D., & Singh, V. K. (2024). MIMIC: Misogyny Identification in Multimodal Internet Content in Hindi-English Code-Mixed Language. ACM Transactions on Asian and Low-Resource Language Information Processing. (https://doi.org/10.1145/3656169)

Mudestreda Multimodal Device State Recognition Dataset obtained from real industrial milling device with Time Series and Image Data for Classification, Regression, Anomaly Detection, Remaining Useful Life (RUL) estimation, Signal Drift measurement, Zero Shot Flank Took Wear, and Feature Engineering purposes. The official dataset used in the paper "Multimodal Isotropic Neural Architecture with Patch Embedding" ICONIP23. Official Minape repository: https://github.com/hubtru/Minape Official Mudestreda dataset: https://zenodo.org/records/8238653 Conference paper: https://link.springer.com/chapter/10.1007/978-981-99-8079-6_14 Mudestreda (MD) | Size 512 Samples (Instances, Observations)| Modalities 4 | Classes 3 | Future research: Regression, Remaining Useful Life (RUL) estimation, Signal Drift detection, Anomaly Detection, Multivariate Time Series Prediction, and Feature Engineering. ## Overview * Task: Uni/Multi-Modal Classification * Domain: Industrial Flank Tool Wear of the Milling Machine * Input (sample): 4 Images: 1 Tool Image, 3 Spectrograms (X, Y, Z axis) * Output: Machine state classes: Sharp, Used, Dulled * Evaluation: Accuracies, Precision, Recal, F1-score, ROC curve * Each tool's wear is categorized sequentially: Sharp → Used → Dulled. * The dataset includes measurements from ten tools: T1 to T10. * Data splitting options include random or chronological distribution, without shuffling. * Options: * Original data or Augmented data * Random distribution or Tool Distribution ## Use Cases: | Input | Model | Output | |-----------------|---------------|-----------------------| | Use Cases: | | | | 4 Images (1 Tool Image, 3 Spectrograms (X, Y, Z)) | Classification Model | Class (Flank Tool Wear: Sharp, Used, Dulled) | | 3 Spectrograms (X,Y,Z axis) | Classification Model | Class (Flank Tool Wear) | | 1 Tool Image | Classification Model | Image Class (Flank Tool Wear) | | Future Work: | | | | [1, ..., 4] Images | Model | Remaining Useful Life (RUL) estimation | | [1, ..., 4] Images | Monitoring Model | Fault and Anomaly Detection | | [1, ..., 4] Images | Forecasting Model | Multivariate Time Series Prediction | | [1, ..., 3] Spectrograms | Model | Signal Drift measurement | | [1, ..., 4] Images | Regression Model | Zero-Shot Flank Tool Wear (in µm, 10e-6 meter) | | [1, ..., 4] Images | Feature Engineering | Diagnostic Feature Designer If you use Mudestreda dataset cite the work Minape @ ICONIP2023. ## Cite the Paper: If you reference the papr or you use Mudestreda dataset cite the work Minape @ ICONIP2023 @inproceedings{truchan2023multimodal, title={Multimodal Isotropic Neural Architecture with Patch Embedding}, author={Truchan, Hubert and Naumov, Evgenii and Abedin, Rezaul and Palmer, Gregory and Ahmadi, Zahra}, booktitle={International Conference on Neural Information Processing}, pages={173--187}, year={2023}, organization={Springer} }

Facial landmark detection is a cornerstone in many facial analysis tasks such as face recognition, drowsiness detection, and facial expression recognition. Numerous methodologies were introduced to achieve accurate and efficient facial landmark localization in visual images. However, there are only several works that address facial landmark detection in thermal images. The main challenge is the limited number of annotated datasets. In this work, we present a thermal face dataset with annotated face bounding boxes and facial landmarks. The dataset contains 2,556 thermal images of 142 individuals, where each thermal image is paired with the corresponding visual image. To the best of our knowledge, our dataset is the largest in terms of the number of individuals. In addition, our dataset can be employed for tasks such as thermal-to-visual image translation, thermal-visual face recognition, and others. We trained two models for the facial landmark detection task to show the efficacy of our dataset. The first model is a classic machine learning model based on an ensemble of regression trees. The second model is a deep learning model based on the U-net architecture. The dataset, annotations, source code, and pre-trained models are publicly available to advance research in thermal face analysis.

Human activity recognition and clinical biomechanics are challenging problems in physical telerehabilitation medicine. However, most publicly available datasets on human body movements cannot be used to study both problems in an out-of-the-lab movement acquisition setting. The objective of the VIDIMU dataset is to pave the way towards affordable patient tracking solutions for remote daily life activities recognition and kinematic analysis. The VIDIMU dataset includes 54 healthy young adults that were recorded on video and 16 of them were simultaneously recorded using custom IMUs. For each subject, 13 activities were registered using a low-resolution video camera and five Inertial Measurement Units (IMUs). Inertial sensors were placed in the lower or the upper limbs of the subject, respectively for activities that involve movement with the lower or the upper body. Video recordings were postprocessed using the state-of-the-art pose estimator BodyTrack (similar to OpenPose, and included in NVIDIA Maxine-AR-SDK) to provide a sequence of 3D joint positions for each movement. Raw IMU recordings were post-processed to compute joint angles by inverse kinematics with OpenSim. For recordings including simultaneous acquisition of video and IMU data types, these signals were used for data file synchronization. Collected data can be further used in applications related to human activity recognition and biomechanics related experiments in simulated home-like settings.