Image Captioning

The COCO (Common Objects in Context) dataset is a large-scale object detection, segmentation, and captioning dataset. It is designed to encourage research on a wide variety of object categories and is commonly used for benchmarking computer vision models. It is an essential dataset for researchers and developers working on object detection, segmentation, and pose estimation tasks.

The Flickr30k dataset contains 31,000 images collected from Flickr, together with 5 reference sentences provided by human annotators. Source: Guiding Long-Short Term Memory for Image Caption Generation Image Source: Dual-Path Convolutional Image-Text Embedding with Instance Loss

Automatic image captioning is the task of producing a natural-language utterance (usually a sentence) that correctly reflects the visual content of an image. Up to this point, the resource most used for this task was the MS-COCO dataset, containing around 120,000 images and 5-way image-caption annotations (produced by paid annotators). Google's Conceptual Captions dataset has more than 3 million images, paired with natural-language captions. In contrast with the curated style of the MS-COCO images, Conceptual Captions images and their raw descriptions are harvested from the web, and therefore represent a wider variety of styles. The raw descriptions are harvested from the Alt-text HTML attribute associated with web images. The authors developed an automatic pipeline that extracts, filters, and transforms candidate image/caption pairs, with the goal of achieving a balance of cleanliness, informativeness, fluency, and learnability of the resulting captions. Source: Conceptual Captions Image Source: Sharma et al

The VizWiz-VQA dataset originates from a natural visual question answering setting where blind people each took an image and recorded a spoken question about it, together with 10 crowdsourced answers per visual question. The proposed challenge addresses the following two tasks for this dataset: predict the answer to a visual question and (2) predict whether a visual question cannot be answered. Source: https://vizwiz.org/tasks-and-datasets/vqa/ Image Source: https://vizwiz.org/tasks-and-datasets/vqa/

COCO Captions contains over one and a half million captions describing over 330,000 images. For the training and validation images, five independent human generated captions are be provided for each image. Source: Microsoft COCO Captions: Data Collection and Evaluation Server

The Hateful Memes data set is a multimodal dataset for hateful meme detection (image + text) that contains 10,000+ new multimodal examples created by Facebook AI. Images were licensed from Getty Images so that researchers can use the data set to support their work.

The nocaps benchmark consists of 166,100 human-generated captions describing 15,100 images from the OpenImages validation and test sets. Source: nocaps: novel object captioning at scale Image Source: https://nocaps.org/

ReferIt3D provides two large-scale and complementary visio-linguistic datasets: i) Sr3D, which contains 83.5K template-based utterances leveraging spatial relations among fine-grained object classes to localize a referred object in a scene, and ii) Nr3D which contains 41.5K natural, free-form, utterances collected by deploying a 2-player object reference game in 3D scenes. This dataset can be used for 3D visual grounding and 3D dense captioning tasks.

Contains 145k captions for 28k images. The dataset challenges a model to recognize text, relate it to its visual context, and decide what part of the text to copy or paraphrase, requiring spatial, semantic, and visual reasoning between multiple text tokens and visual entities, such as objects. Source: TextCaps: a Dataset for Image Captioning with Reading Comprehension

Click to add a brief description of the dataset (Markdown and LaTeX enabled). Provide: * a high-level explanation of the dataset characteristics * explain motivations and summary of its content * potential use cases of the dataset

Winoground is a dataset for evaluating the ability of vision and language models to conduct visio-linguistic compositional reasoning. Given two images and two captions, the goal is to match them correctly -- but crucially, both captions contain a completely identical set of words, only in a different order. The dataset was carefully hand-curated by expert annotators and is labeled with a rich set of fine-grained tags to assist in analyzing model performance.

The ReferIt dataset contains 130,525 expressions for referring to 96,654 objects in 19,894 images of natural scenes. Source: BiLingUNet: Image Segmentation by Modulating Top-Down and Bottom-Up Visual Processing with Referring Expressions Image Source: http://tamaraberg.com/referitgame/

The Remote Sensing Image Captioning Dataset (RSICD) is a dataset for remote sensing image captioning task. It contains more than ten thousands remote sensing images which are collected from Google Earth, Baidu Map, MapABC and Tianditu. The images are fixed to 224X224 pixels with various resolutions. The total number of remote sensing images is 10921, with five sentences descriptions per image. Source: https://github.com/201528014227051/RSICD_optimal Image Source: https://github.com/201528014227051/RSICD_optimal

We propose Localized Narratives, a new form of multimodal image annotations connecting vision and language. We ask annotators to describe an image with their voice while simultaneously hovering their mouse over the region they are describing. Since the voice and the mouse pointer are synchronized, we can localize every single word in the description. This dense visual grounding takes the form of a mouse trace segment per word and is unique to our data. We annotated 849k images with Localized Narratives: the whole COCO, Flickr30k, and ADE20K datasets, and 671k images of Open Images, all of which we make publicly available. We provide an extensive analysis of these annotations showing they are diverse, accurate, and efficient to produce. We also demonstrate their utility on the application of controlled image captioning.

Contains 51,583 descriptions of 11,046 objects from 800 ScanNet scenes. ScanRefer is the first large-scale effort to perform object localization via natural language expression directly in 3D. Source: ScanRefer: 3D Object Localization in RGB-D Scans using Natural Language

A new challenge set for multimodal classification, focusing on detecting hate speech in multimodal memes. Source: The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes

A new large-scale dataset for referring expressions, based on MS-COCO. Source: Generation and Comprehension of Unambiguous Object Descriptions

Dataset contains 33,010 molecule-description pairs split into 80%/10%/10% train/val/test splits. The goal of the task is to retrieve the relevant molecule for a natural language description. It is defined as follows: To push the boundaries of multimodal models, we present a new IR task: \textbf{Text2Mol}. Given a text query and list of molecules without any reference textual information (represented, for example, as SMILES strings, graphs, or other equivalent representations) retrieve the molecule corresponding to the query. From a text description of a molecule, the model must incorporate the information in the description into a semantic representation which can be used to directly retrieve the molecule. This requires the integration of two very different types of information: the structured knowledge represented by text and the chemical properties present in molecular graphs. We assume there is only one correct (relevant) molecule for each description, so we consider two measures for this task: Hits@1 and mean reciprocal rank (MRR). 80% of the data is used for training. Retrieval is done against the entire corpus of molecules (train, val, test).

UT Zappos50K is a large shoe dataset consisting of 50,025 catalog images collected from Zappos.com. The images are divided into 4 major categories — shoes, sandals, slippers, and boots — followed by functional types and individual brands. The shoes are centered on a white background and pictured in the same orientation for convenient analysis. Source: UT Zappos50K

The Image Paragraph Captioning dataset allows researchers to benchmark their progress in generating paragraphs that tell a story about an image. The dataset contains 19,561 images from the Visual Genome dataset. Each image contains one paragraph. The training/val/test sets contains 14,575/2,487/2,489 images. Since all the images are also part of the Visual Genome dataset, each image also contains 50 region descriptions (short phrases describing parts of an image), 35 objects, 26 attributes and 21 relationships and 17 question-answer pairs. Source: A Hierarchical Approach for Generating Descriptive Image Paragraphs Image Source: https://cs.stanford.edu/people/ranjaykrishna/im2p/index.html

Research in massively multilingual image captioning has been severely hampered by a lack of high-quality evaluation datasets. In this paper we present the Crossmodal-3600 dataset (XM3600 in short), a geographically-diverse set of 3600 images annotated with human-generated reference captions in 36 languages. The images were selected from across the world, covering regions where the 36 languages are spoken, and annotated with captions that achieve consistency in terms of style across all languages, while avoiding annotation artifacts due to direct translation. We apply this benchmark to model selection for massively multilingual image captioning models, and show strong correlation results with human evaluations when using XM3600 as golden references for automatic metrics.

The SentiCap dataset contains several thousand images with captions with positive and negative sentiments. These sentimental captions are constructed by the authors by re-writing factual descriptions. In total there are 2000+ sentimental captions. Source: SentiCap: Generating Image Descriptions with Sentiments

DIOR-RSVG is a large-scale benchmark dataset of remote sensing data (RSVG). It aims to localize the referred objects in remote sensing (RS) images with the guidance of natural language. This new dataset includes image/expression/box triplets for training and evaluating visual grounding models. Source: RSVG: Exploring Data and Models for Visual Grounding on Remote Sensing Data Image Source: https://arxiv.org/pdf/2210.12634v1.pdf

FlickrStyle10K is collected and built on Flickr30K image caption dataset. The original FlickrStyle10K dataset has 10,000 pairs of images and stylized captions including humorous and romantic styles. However, only 7,000 pairs from the official training set are now publicly accessible. The dataset can be downloaded via https://zhegan27.github.io/Papers/FlickrStyle_v0.9.zip

WIDER is a dataset for complex event recognition from static images. As of v0.1, it contains 61 event categories and around 50574 images annotated with event class labels.

COCO-CN is a bilingual image description dataset enriching MS-COCO with manually written Chinese sentences and tags. The new dataset can be used for multiple tasks including image tagging, captioning and retrieval, all in a cross-lingual setting. Source: COCO-CN

IU X-ray (Demner-Fushman et al., 2016) is a set of chest X-ray images paired with their corresponding diagnostic reports. The dataset contains 7,470 pairs of images and reports. A simplified version of IU X-Ray through CPIR-MR prompting was presented at AAAI 2025: AAAI Article Link. Dataset Link: SMR IU X-Ray.

SCICAP is a large-scale image captioning dataset that contains real-world scientific figures and captions. SCICAP was constructed using more than two million images from over 290,000 papers collected and released by arXiv. Image source: https://arxiv.org/pdf/2110.11624v1.pdf

WHOOPS! Is a dataset and benchmark for visual commonsense. The dataset is comprised of purposefully commonsense-defying images created by designers using publicly-available image generation tools like Midjourney. It contains commonsense-defying image from a wide range of reasons, deviations from expected social norms and everyday knowledge.

Consists of over 39,000 images originating from people who are blind that are each paired with five captions. Source: Captioning Images Taken by People Who Are Blind

Object HalBench is a benchmark used to evaluate the performance of Language Models, particularly those that are multimodal (i.e., they can process and generate both text and images). It's designed to test how well these models can avoid "hallucinations" - generating text that is not factually grounded in the images they're processing¹. For instance, the OmniLMM-12B model, which is a state-of-the-art open-source Language Model, has been reported to outperform GPT-4V on the Object HalBench¹. This model is aligned via a technique called multimodal RLHF (Reinforcement Learning from Human Feedback) for trustworthy behavior¹. This means it's designed to generate outputs that are more reliable and factually accurate, particularly when dealing with multimodal inputs¹. (1) openbmb/OmniLMM-12B · Hugging Face. https://huggingface.co/openbmb/OmniLMM-12B. (2) OmniLMM：准确、高效的开源多模态大模型 - 知乎. https://zhuanlan.zhihu.com/p/681251797. (3) RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from - arXiv.org. https://arxiv.org/html/2312.00849v2. (4) README.md · openbmb/MiniCPM-V-2 at main - Hugging Face. https://huggingface.co/openbmb/MiniCPM-V-2/blob/main/README.md. (5) undefined. https://github.com/OpenBMB/OmniLMM.git.

A collection that allows researchers to approach the extremely challenging problem of description generation using relatively simple non-parametric methods and produces surprisingly effective results. Source: Im2Text: Describing Images Using 1 Million Captioned Photographs

This dataset is a collection of spoken language instructions for a robotic system to pick and place common objects. Text instructions and corresponding object images are provided. The dataset consists of situations where the robot is instructed by the operator to pick up a specific object and move it to another location: for example, Move the blue and white tissue box to the top right bin. This dataset consists of RGBD images, bounding box annotations, destination box annotations, and text instructions. Source: https://github.com/pfnet-research/picking-instruction Image Source: https://github.com/pfnet-research/picking-instruction

CITE is a crowd-sourced resource for multimodal discourse: this resource characterises inferences in image-text contexts in the domain of cooking recipes in the form of coherence relations. Source: CITE: A Corpus of Image-Text Discourse Relations

Open Images is a computer vision dataset covering ~9 million images with labels spanning thousands of object categories. A subset of 1.9M includes diverse annotations types. - 15,851,536 boxes on 600 classes - 2,785,498 instance segmentations on 350 classes - 3,284,280 relationship annotations on 1,466 relationships - 675,155 localized narratives (synchronized voice, mouse trace, and text caption) - 66,391,027 point-level annotations on 5,827 classes - 61,404,966 image-level labels on 20,638 classes Images are under a CC BY 2.0 license, annotations under CC BY 4.0 license.

Peir Gross (Jing et al., 2018) was collected with descriptions in the Gross sub-collection from PEIR digital library, resulting in 7.442 image-caption pairs from 21 different sub-categories. Each caption contains only one sentence.