Papers with Code 아카이브

Image Captioning

태스크의 하위 분류와 논문·코드·평가·기법·데이터셋을 탐색하세요.

하위 태스크
8
평가표
34
보존된 평가 기록
419

Image Captioning is the task of describing the content of an image in words. This task lies at the intersection of computer vision and natural language processing. Most image captioning systems use an encoder-decoder framework, where an input image is encoded into an intermediate representation of the information in the image, and then decoded into a descriptive text sequence. The most popular benchmarks are nocaps and COCO, and models are typically evaluated according to a BLEU or CIDER metric.

( Image credit: Reflective Decoding Network for Image Captioning, ICCV'19)

데이터셋

출처에 같은 태스크 이름으로 연결된 데이터셋입니다. 동명 태스크가 포함될 수 있으므로, 비교할 때는 평가 결과의 조건도 확인하세요.

BanglaLekhaImageCaptions

This dataset consists of images and annotations in Bengali. The images are human annotated in Bengali by two adult native Bengali speakers. All popular image captioning datasets have a predominant western cultural bias with the annotations done in English. Using such datasets to train an image captioning system assumes that a good English to target language translation system exists and that the original dataset had elements of the target culture. Both these assumptions are false, leading to the need of a culturally relevant dataset in Bengali, to generate appropriate image captions of images relevant to the Bangladeshi and wider subcontinental context. The dataset presented consists of 9,154 images.

PoMo

PoMo consists of more than 231K sentences with post-modifiers and associated facts extracted from Wikidata for around 57K unique entities. Source: PoMo: Generating Entity-Specific Post-Modifiers in Context

SkyEye-968k

SkyEyeGPT: Unifying Remote Sensing Vision-Language Tasks via Instruction Tuning with Large Language Model

Tencent ML-Images

Tencent ML-Images is a large open-source multi-label image database, including 17,609,752 training and 88,739 validation image URLs, which are annotated with up to 11,166 categories. Source: Tencent ML-Images

Concadia

Concadia is a publicly available Wikipedia-based corpus, which consists of 96,918 images with corresponding English-language descriptions, captions, and surrounding context.

Image Editing Request Dataset

A new language-guided image editing dataset that contains a large number of real image pairs with corresponding editing instructions. Source: Expressing Visual Relationships via Language

Polaris

The Polaris dataset offers a large-scale, diverse benchmark for evaluating metrics for image captioning, surpassing existing datasets in terms of size, caption diversity, number of human judgments, and granularity of the evaluations. It includes 131,020 generated captions and 262,040 reference captions. The generated captions have a vocabulary of 3,154 unique words and the reference captions have a vocabulary of 22,275 unique words.

STAIR Captions

STAIR Captions is a large-scale dataset containing 820,310 Japanese captions. This dataset can be used for caption generation, multimodal retrieval, and image generation. Source: STAIR Captions

Hephaestus

Hephaestus is the first large-scale InSAR dataset. Driven by volcanic unrest detection, it provides 19,919 unique satellite frames annotated with a diverse set of labels. Moreover, each sample is accompanied by a textual description of its contents. The goal of this dataset is to boost research on exploitation of interferometric data enabling the application of state-of-the-art computer vision+NLP methods. Furthermore, the annotated dataset is bundled with a large archive of unlabeled frames to enable large-scale self-supervised learning. The final size of the dataset amounts to 110,573 interferograms.

Image and Video Advertisements

The Image and Video Advertisements collection consists of an image dataset of 64,832 image ads, and a video dataset of 3,477 ads. The data contains rich annotations encompassing the topic and sentiment of the ads, questions and answers describing what actions the viewer is prompted to take and the reasoning that the ad presents to persuade the viewer ("What should I do according to this ad, and why should I do it? "), and symbolic references ads make (e.g. a dove symbolizes peace). Source: Automatic Understanding of Image and Video Advertisements

Kvasir-VQA

The Kvasir-VQA dataset is an extended dataset derived from the HyperKvasir and Kvasir-Instrument datasets, augmented with question-and-answer annotations. This dataset is designed to facilitate advanced machine learning tasks in gastrointestinal (GI) diagnostics, including image captioning, Visual Question Answering (VQA) and text-based generation of synthetic medical images.

OpenCHAIR

OpenCHAIR is a benchmark for evaluating open-vocabulary hallucinations in image captioning models. By leveraging the linguistic knowledge of LLMs, OpenCHAIR is able to perform fine-grained hallucination measurements, as well as significantly increase the amount of objects that can be measured (especially when compared to the existing benchmark, CHAIR). To exploit the LLM's full potential we construct a new dataset by generating 5000 captions with highly diverse objects and let a powerful text-to-image model generate images for them. We find that we are not just able to increase the benchmark's diversity, but also improve the evaluation accuracy with respect to CHAIR's. For more info see https://assafbk.github.io/mocha/

SCapRepo

A screenshot-caption dataset containing 135k pairs of screenshots and captions extracted from Google Play.

WikiScenes

The WikiScenes dataset consists of paired images and language descriptions capturing world landmarks and cultural sites, with associated 3D models and camera poses. WikiScenes is derived from the massive public catalog of freely-licensed crowdsourced data in the Wikimedia Commons project, which contains a large variety of images with captions and other metadata. The dataset contains two forms of textual descriptions for each image: (1) Captions associated with images, describing the image using free-form language, and (2) The WikiCategory hierarchy obtained according to the hierarchy of WikiCategories associated with each image (see the examples in the image below). Overall, WikiScenes contains approximately 63K images with textual descriptions.

DeCOCO

DeCOCO is a bilingual (English-German) corpus of image descriptions, where the English part is extracted from the COCO dataset, and the German part are translations by a native German speaker.

DPC-Captions

This is an open-source image captions dataset for the aesthetic evaluation of images. The dataset is called DPC-Captions, which contains comments of up to five aesthetic attributes of one image through knowledge transfer from a full-annotated small-scale dataset. Source: https://github.com/BestiVictory/DPC-Captions

EgoShots

Egoshots is a 2-month Ego-vision Dataset with Autographer Wearable Camera annotated "for free" with transfer learning. Three state of the art pre-trained image captioning models are used. The dataset represents the life of 2 interns while working at Philips Research (Netherlands) (May-July 2015) generously donating their data. Source: https://github.com/NataliaDiaz/Egoshots

VizWiz-Priv

VizWiz-Priv includes 8,862 regions showing private content across 5,537 images taken by blind people. Of these, 1,403 are paired with questions and 62% of those directly ask about the private content. Source: VizWiz-Priv: A Dataset for Recognizing the Presence and Purpose of Private Visual Information in Images Taken by Blind People

VizWiz-QualityIssues

A large-scale dataset that links the assessment of image quality issues to two practical vision tasks: image captioning and visual question answering. Source: Assessing Image Quality Issues for Real-World Problems

3U-VQA

To tackle the challenge of obtaining out-of-distribution (OOD) data for LVQA models, we introduced a novel dataset named 3U-VQA dataset (Usual, Unusual and Unknown object scenarios for LVQA with difficulty scoring dataset). The dataset comprises question and image sets. Each instance in the questions set is associated with a set of features representing the question-related criteria set. The questions and their ground truth answers are written using placeholders for the objects and their features, which can be specified based on the user needs and requirements. When creating the questions, we avoided deliberately binary (Yes/No) questions to prevent potential bias caused by the question's type in the model response.

BNATURE

This is a dataset for Bengali Captioning from Images.

CapGaze

Consists of eye movements and verbal descriptions recorded synchronously over images. Source: Human Attention in Image Captioning: Dataset and Analysis

ESP

ESP dataset (Evaluation for Styled Prompt dataset) is a benchmark for zero-shot domain-conditional caption generation. ESP is a new dataset focusing on providing multiple styled text targets for the same image. It comprises 4.8k captions from 1k images in the COCO Captions test set. We collect five text domains with everyday usage: blog, social media, instruction, story, and news.

GroundCap

GroundCap is a novel grounded image captioning dataset derived from MovieNet, containing 52,350 movie frames with detailed grounded captions. The dataset uniquely features an ID-based system that maintains object identity throughout captions, enables tracking of object interactions, and grounds not only objects but also actions and locations in the scene. ### Data Instances Each sample in the dataset contains: - An image (movie frame) - Object detections with: - Unique object IDs - Class labels - Confidence scores - Bounding box coordinates - A grounded caption with three types of grounding tags: - <gdo> for grounding objects (e.g., "the person", "a car") - <gda> for grounding actions (e.g., "running", "sitting") - <gdl> for grounding locations (e.g., "on the bridge", "in the kitchen") ### Data Fields - id: Unique identifier for each caption - image: The movie frame being captioned - detections: List of detected objects containing: - id: Object's unique identifier (integer starting at 0 for each class) - label: Object class label - score: Detection confidence score - box: Bounding box coordinates (x, y, w, h) - caption: Grounded caption text with HTML tags - <gdo> tags ground object references to detections using {class}-{id} as attribute (e.g., <gdo class="person" person-0>the man</gdo>) - <gda> tags ground actions to objects using {class}-{id} as attribute (e.g., <gda class="run" person-0>running</gda>) - <gdl> tags ground locations to objects using {class}-{id} as attribute (e.g., <gdl class="couch" couch-0>on the couch</gdl>) - human_annotated: Boolean indicating whether the caption was automatically generated (False) or human-refined (True) Multiple objects can be referenced in a single tag. For instance, <gdo class="person" person-0 person-1>the two people</gdo> refers to two detected people, namely to the detections with IDs 0 and 1 and the class label "person".

InFashAI

AI algorithms, and in particular Machine Learning (ML) algorithms, learn from data tasks that have been traditionally done by humans such as: image classification, facial recognition, linguistic translation etc. To have a good generalization capability, AI algorithms must learn from sufficiently representative data, which is unfortunately not often the case. This results in a hyper-specialization of AI and its inability to perform well on new data whose distribution is too far from the one of the training set. It raises ethical questions which will undoubtedly have direct or indirect consequences on society. However, and despite biases they can entail, AI technologies are revolutionizing virtually every industry, and are forcing players in those industries to reinvent their businesses. Like many industries, the fashion industy is being hit hard by exponential advances in AI. AI technologies are now able to describe a dress model, extract its attributes (color, style, type of sleeve, etc.), predict future fashion trends and even offer personalized clothing styles. Several publicly large datasets of fashion images made it possible. However, the lack of diversity in available datasets is palpable. For example, images of African fashion are almost absent from these datasets. It raises problems in terms of ability to generalize algorithms trained on those datasets to African styles for instance, and therefore it limits the adoption of AI technologies within the African fashion industry. Therefore, for an inclusive AI in the field of fashion, and to ensure that African fashion can benefit from the potentials of AI, Ai4Innov initiated the Inclusive Fashion AI project (InFashAI) which aims to create datasets that are much more representative of the diversity that exists in the world of fashion. We will first focus on building up a large volume of data on African fashion. This dataset will be progressively open source and, we hope, will be the backbone for AI tools adapted to African fashion.

MCIC-COCO

A large-scale machine comprehension dataset (based on the COCO images and captions). Source: Understanding Image and Text Simultaneously: a Dual Vision-Language Machine Comprehension Task

MMInstruct-GPT4V

Vision-language supervised fine-tuning effectively enhances VLLM performance, but existing visual instruction tuning datasets have limitations: 1. Instruction Annotation Quality: Despite strong performance, advanced VLLMs may generate instructions with inaccuracies, such as hallucinations. 2. Instruction and Image Diversity: Limited instruction types and lack of diverse image data impact the model's ability to generate varied and realistic outputs. MMInstruct Dataset To address these challenges, we created the MMInstruct dataset, featuring: - 973K instructions from 24 domains - Four instruction types: Judgement, Multiple-Choice, Long Visual Question Answering, and Short Visual Question Answering.

ParsVQA-Caps

Despite recent advances in vision-and-language tasks, most progress is still focused on resource-rich languages such as English. Furthermore, widespread vision-and-language datasets directly adopt images representative of American or European cultures resulting in bias. Hence we introduce ParsVQA-Caps, the first benchmark in Persian for Visual Question Answering and Image Captioning tasks. We utilize two ways to collect datasets for each task, human-based and template-based for VQA and human-based and web-based for image captioning. The image captioning dataset consists of over 7.5k images and about 9k captions. The VQA dataset consists of almost 11k images and 28.5k question and answer pairs with short and long answers usable for both classification and generation VQA. source: ParsVQA-Caps: A Benchmark for Visual Question Answering and Image Captioning in Persian

PatentDesc-355K

Click to add a brief description of the dataset (Markdown and LaTeX enabled). Provide: * a high-level explanation of the dataset characteristics * explain motivations and summary of its content * potential use cases of the dataset

RPCD

The Reddit Photo Critique Dataset (RPCD) contains tuples of image and photo critiques. RPCD consists of 74K images and 220K comments and is collected from a Reddit community used by hobbyists and professional photographers to improve their photography skills by leveraging constructive community feedback. The proposed dataset differs from previous aesthetics datasets mainly in three aspects, namely * the large scale of the dataset and the extension of the comments criticizing different aspects of the image; * it contains mostly UltraHD images; * it can easily be extended to new data as it is collected through an automatic pipeline.

출처: Papers with Code · CC BY-SA 4.0. 아카이브 자료와 새로 수집한 논문은 별도로 관리합니다.