Papers with Code · CC BY-SA 4.0

데이터셋 탐색

데이터셋의 설명·홈페이지를 찾고, 관련 태스크의 평가 기록을 살펴보세요.

관련 논문 14편

Vimeo Creative Commons Collection

V3C
V3C 설명 보기

The Vimeo Creative Commons Collection, in short V3C, is a collection of 28’450 videos (with overall length of about 3’800 h) published under creative commons license on Vimeo. V3C comes with a shot segmentation for each video, together with the resulting keyframes in original as well as reduced resolution and additional metadata. It is intended to be used from 2019 at the International large-scale TREC Video Retrieval Evaluation campaign (TRECVid).

관련 논문 14편

VegFru

VegFru 설명 보기

VegFru is a domain-specific dataset for fine-grained visual categorization. VegFru categorizes vegetables and fruits according to their eating characteristics, and each image contains at least one edible part of vegetables or fruits with the same cooking usage. Particularly, all the images are labelled hierarchically. The current version covers vegetables and fruits of 25 upper-level categories and 292 subordinate classes. And it contains more than 160,000 images in total and at least 200 images for each subordinate class. Source: https://openaccess.thecvf.com/content_ICCV_2017/papers/Hou_VegFru_A_Domain-Specific_ICCV_2017_paper.pdf Image Source: https://openaccess.thecvf.com/content_ICCV_2017/papers/Hou_VegFru_A_Domain-Specific_ICCV_2017_paper.pdf

관련 논문 14편

Video Timeline Tags

ViTT
ViTT 설명 보기

The ViTT dataset consists of human produced segment-level annotations for 8,169 videos. Of these, 5,840 videos have been annotated once, and the rest of the videos have been annotated twice or more. A total of 12,461 sets of annotations are released. The videos in the dataset are from the Youtube-8M dataset. An annotation has the following format: { "id": "FmTp", "annotations": [ { "timestamp": 260, "tag": "Opening" }, { "timestamp": 16000, "tag": "Displaying technique" }, { "timestamp": 23990, "tag": "Showing foot positioning" }, { "timestamp": 55530, "tag": "Demonstrating crossover" }, { "timestamp": 114100, "tag": "Closing" } ] } Source: Video Timeline Tags (ViTT)

관련 논문 14편

VisIT-Bench

VisIT-Bench 설명 보기

VisIT-Bench is a new vision-language instruction following benchmark inspired by real-world use cases. Testing 70 diverse “wish-list” skills with an automated ranking system, it advances the ongoing assessment of multimodal chatbot performance. Why VisIT-Bench 🤔? Though recent VLMs have shown promise in following instructions, their evaluation for real-world human-chatbot instructions is often limited. Typically, VLMs are evaluated through qualitative comparison of outputs, which makes it challenging to quantify progress and potential shortcomings. VisIT-Bench helps address this problem by offering a comprehensive testbed for measuring model performance across a diverse set of instruction-following tasks, inspired by real world scenarios.🌍

관련 논문 14편

WHOOPS!

WHOOPS! 설명 보기

WHOOPS! Is a dataset and benchmark for visual commonsense. The dataset is comprised of purposefully commonsense-defying images created by designers using publicly-available image generation tools like Midjourney. It contains commonsense-defying image from a wide range of reasons, deviations from expected social norms and everyday knowledge.

관련 논문 14편

WHU

WHU 설명 보기

Created for MVS tasks and is a large-scale multi-view aerial dataset generated from a highly accurate 3D digital surface model produced from thousands of real aerial images with precise camera parameters. Source: A Novel Recurrent Encoder-Decoder Structure for Large-Scale Multi-view Stereo Reconstruction from An Open Aerial Dataset

관련 논문 14편

Winograd NLI

WNLI
WNLI 설명 보기

The WNLI dataset is a part of the GLUE benchmark used for Natural Language Inference (NLI). It contains pairs of sentences, and the task is to determine whether the second sentence is an entailment of the first one or not. The dataset is used to train and evaluate models on their ability to understand these relationships between sentences.

관련 논문 14편

Multi-View Operating Room

mvor
mvor 설명 보기

Multi-View Operating Room (MVOR) dataset consists of 732 synchronized multi-view frames recorded by three RGB-D cameras in a hybrid OR during real clinical interventions. Each multi-view frame consists of three color and three depth images. The MVOR dataset was sampled from four days of recording in an interventional room at the University Hospital of Strasbourg during procedures such as vertebroplasty and lung biopsy. There are in total 4699 bounding boxes, 2926 2D keypoint annotations, and 1061 3D keypoint annotations. Source: https://github.com/CAMMA-public/mvor Image Source: https://github.com/CAMMA-public/mvor

관련 논문 14편

speechocean762

speechocean762 설명 보기

speechocean762 is an open-source speech corpus designed for pronunciation assessment use, consisting of 5000 English utterances from 250 non-native speakers, where half of the speakers are children. Five experts annotated each of the utterances at sentence-level, word-level and phoneme-level. This corpus is allowed to be used freely for commercial and non-commercial purposes. To avoid subjective bias, each expert scores independently under the same metric

관련 논문 13편

3D Ken Burns

3D Ken Burns 설명 보기

This dataset accompanies our paper on synthesizing the 3D Ken Burns effect from a single image. It consists of 134041 captures from 32 virtual environments where each capture consists of 4 views. Each view contains color-, depth-, and normal-maps at a resolution of 512x512 pixels.

관련 논문 13편

ADVIO

ADVIO 설명 보기

Provides a wide range of raw sensor data that is accessible on almost any modern-day smartphone together with a high-quality ground-truth track. Source: ADVIO: An authentic dataset for visual-inertial odometry

관련 논문 13편

ANTIQUE

ANTIQUE 설명 보기

ANTIQUE is a collection of 2,626 open-domain non-factoid questions from a diverse set of categories. The dataset contains 34,011 manual relevance annotations. The questions were asked by real users in a community question answering service, i.e., Yahoo! Answers. Relevance judgments for all the answers to each question were collected through crowdsourcing.

관련 논문 13편

Aligned Scores and Performances

ASAP
ASAP 설명 보기

ASAP is a dataset of 222 digital musical scores aligned with 1068 performances (more than 92 hours) of Western classical piano music. The scores are provided as paired MusicXML files and quantized MIDI files, and the performances as paired MIDI files and partially as audio recordings. Scores and performances are aligned with downbeat, beat, time signature, and key signature annotations. Source: ASAP: a dataset of aligned scores and performances for piano transcription

관련 논문 13편

Amazon Baby 5-core

Amazon Baby
Amazon Baby 설명 보기

This dataset includes reviews (ratings, text, helpfulness votes), product metadata (descriptions, category information, price, brand, and image features), and links (also viewed/also bought graphs).

관련 논문 13편

BC2GM

BC2GM 설명 보기

Created by Smith et al. at 2008, the BioCreative II Gene Mention Recognition (BC2GM) Dataset contains data where participants are asked to identify a gene mention in a sentence by giving its start and end characters. The training set consists of a set of sentences, and for each sentence a set of gene mentions (GENE annotations). [registration required for access], in English language. Containing 20 in n/a file format.

관련 논문 13편

BIOMRC

BIOMRC 설명 보기

A large-scale cloze-style biomedical MRC dataset. Care was taken to reduce noise, compared to the previous BIOREAD dataset of Pappas et al. (2018). Source: BIOMRC: A Dataset for Biomedical Machine Reading Comprehension

관련 논문 13편

Badminton

Badminton 설명 보기

This dataset was introduced by the TrackNetV2 work. Following the dataset split defined by the authors, we use all the clips from 26 matches as a training set and the remaining 3 matches as a testing set.

관련 논문 13편

BrixIA Covid-19

BrixIA
BrixIA 설명 보기

BrixIA Covid-19 is a large dataset of CXR images corresponding to the entire amount of images taken for both triage and patient monitoring in sub-intensive and intensive care units during one month (between March 4th and April 4th 2020) of pandemic peak at the ASST Spedali Civili di Brescia, and contains all the variability originating from a real clinical scenario. It includes 4,707 CXR images of COVID-19 subjects, acquired with both CR and DX modalities, in AP or PA projection, and retrieved from the facility RIS-PACS system.

관련 논문 13편

BrnoCompSpeed

BrnoCompSpeed 설명 보기

The dataset contains 21 full-HD videos, each around 1 hr long, captured at six different locations. Vehicles in the videos (20 865 instances in total) are annotated with the precise speed measurements from optical gates using LiDAR and verified with several reference GPS tracks. The dataset is available for download and it contains the videos and metadata (calibration, lengths of features in image, annotations, and so on) for future comparison and evaluation. This dataset was published with paper SOCHOR Jakub et al. Comprehensive Data Set for Automatic Single Camera Visual Speed Measurement, IEEE T-ITS.

관련 논문 13편

Curated Breast Imaging Subset of Digital Database for Screening Mammography

CBIS-DDSM
CBIS-DDSM 설명 보기

This CBIS-DDSM (Curated Breast Imaging Subset of DDSM) is an updated and standardized version of the Digital Database for Screening Mammography (DDSM) . The DDSM is a database of 2,620 scanned film mammography studies. It contains normal, benign, and malignant cases with verified pathology information. The scale of the database along with ground truth validation makes the DDSM a useful tool in the development and testing of decision support systems. The CBIS-DDSM collection includes a subset of the DDSM data selected and curated by a trained mammographer. The images have been decompressed and converted to DICOM format. Updated ROI segmentation and bounding boxes, and pathologic diagnosis for training data are also included. A manuscript describing how to use this dataset in detail is available at https://www.nature.com/articles/sdata2017177. Published research results from work in developing decision support systems in mammography are difficult to replicate due to the lack of a standard evaluation data set; most computer-aided diagnosis (CADx) and detection (CADe) algorithms for breast cancer in mammography are evaluated on private data sets or on unspecified subsets of public databases. Few well-curated public datasets have been provided for the mammography community. These include the DDSM, the Mammographic Imaging Analysis Society (MIAS) database, and the Image Retrieval in Medical Applications (IRMA) project. Although these public data sets are useful, they are limited in terms of data set size and accessibility. For example, most researchers using the DDSM do not leverage all its images for a variety of historical reasons. When the database was released in 1997, computational resources to process hundreds or thousands of images were not widely available. Additionally, the DDSM images are saved in non-standard compression files that require the use of decompression code that has not been updated or maintained for modern computers. Finally, the ROI annotations for the abnormalities in the DDSM were provided to indicate a general position of lesions, but not a precise segmentation for them. Therefore, many researchers must implement segmentation algorithms for accurate feature extraction. This causes an inability to directly compare the performance of methods or to replicate prior results. The CBIS-DDSM collection addresses that challenge by publicly releasing an curated and standardized version of the DDSM for evaluation of future CADx and CADe systems (sometimes referred to generally as CAD) research in mammography. Please note that the image data for this collection is structured such that each participant has multiple patient IDs. For example, participant 00038 has 10 separate patient IDs which provide information about the scans within the IDs (e.g. Calc-Test_P_00038_LEFT_CC, Calc-Test_P_00038_RIGHT_CC_1). This makes it appear as though there are 6,671 patients according to the DICOM metadata, but there are only 1,566 actual participants in the cohort.

관련 논문 13편

CirCor DigiScope

CirCor DigiScope 설명 보기

CirCor DigiScope is currently the largest pediatric heart sound dataset. A total of 5282 recordings have been collected from the four main auscultation locations of 1568 patients, in the process 215780 heart sounds have been manually annotated. Each cardiac murmur has been manually annotated by an expert annotator according to its timing, shape, pitch, grading and quality. Source: https://arxiv.org/pdf/2108.00813v1.pdf

관련 논문 13편

Cityscapes 3D

Cityscapes 3D 설명 보기

Detecting vehicles and representing their position and orientation in the three dimensional space is a key technology for autonomous driving. Recently, methods for 3D vehicle detection solely based on monocular RGB images gained popularity. In order to facilitate this task as well as to compare and drive state-of-the-art methods, several new datasets and benchmarks have been published. Ground truth annotations of vehicles are usually obtained using lidar point clouds, which often induces errors due to imperfect calibration or synchronization between both sensors. To this end, we propose Cityscapes 3D, extending the original Cityscapes dataset with 3D bounding box annotations for all types of vehicles. In contrast to existing datasets, our 3D annotations were labeled using stereo RGB images only and capture all nine degrees of freedom. This leads to a pixel-accurate reprojection in the RGB image and a higher range of annotations compared to lidar-based approaches. In order to ease multitask learning, we provide a pairing of 2D instance segments with 3D bounding boxes. In addition, we complement the Cityscapes benchmark suite with 3D vehicle detection based on the new annotations as well as metrics presented in this work. Dataset and benchmark are available online.

관련 논문 13편

DNA-Rendering

DNA-Rendering 설명 보기

DNA-Rendering is a large-scale, high-fidelity repository of human performance data for neural actor rendering. It contains over 1500 human subjects, 5000 motion sequences, and 67.5M frames' data volume. Upon the massive collections, the authors provide human subjects with grand categories of pose actions, body shapes, clothing, accessories, hairdos, and object intersection, which ranges the geometry and appearance variances from everyday life to professional occasions. Second, they provide rich assets for each subject -- 2D/3D human body keypoints, foreground masks, SMPLX models, cloth/accessory materials, multi-view images, and videos. These assets boost the current method's accuracy on downstream rendering tasks. Third, they construct a professional multi-View system to capture data, which contains 60 synchronous cameras with max 4096×3000 resolution, 15 fps speed, and stern camera calibration steps, ensuring high-quality resources for task training and evaluation.

관련 논문 13편

zaid allal

Darmstadt Noise Dataset
Darmstadt Noise Dataset 설명 보기

the dataset contains data about hydrogen storage in metal hydrides

관련 논문 13편

Dayton

Dayton 설명 보기

The Dayton dataset is a dataset for ground-to-aerial (or aerial-to-ground) image translation, or cross-view image synthesis. It contains images of road views and aerial views of roads. There are 76,048 images in total and the train/test split is 55,000/21,048. The images in the original dataset have 354×354 resolution. Source: Multi-Channel Attention Selection GANs for Guided Image-to-Image Translation Image Source: https://arxiv.org/abs/1912.06112

관련 논문 13편

DialFact

DialFact 설명 보기

DialFact is a testing benchmark dataset of 22,245 annotated conversational claims, paired with pieces of evidence from Wikipedia. There are three sub-tasks in DialFact: 1) Verifiable claim detection task distinguishes whether a response carries verifiable factual information; 2) Evidence retrieval task retrieves the most relevant Wikipedia snippets as evidence; 3) Claim verification task predicts a dialogue response to be supported, refuted, or not enough information. Description from: DialFact: A Benchmark for Fact-Checking in Dialogue

관련 논문 13편

Electric Guitar Effects Dataset, ISMIR 2022 LBD

EGFxSet
EGFxSet 설명 보기

EGFxSet (Electric Guitar Effects dataset) features recordings for all clean tones in a 22-fret Stratocaster, recorded with 5 different pickup configurations, also processed through 12 popular guitar effects. Our dataset was recorded in real hardware, making it relevant for music information retrieval tasks on real music. We also include annotations for parameter settings of the effects we used. EGFxSet is a dataset of 8,970 audio files with a 5-second duration each, summing a total time of 12 hours and 28 minutes. All possible 138 notes of a standard tuning 22 frets guitar were recorded in each one of the 5 pickup configurations, giving a total of 690 clean tone audio files ( 58 min ). The 690 clean audio (58 min) files were processed through 12 different audio effects employing actual guitar gear (no VST emulations were used), summing a total of 8,280 processed audio files (11 hours 30 min). The effects employed were divided into four categories, and each category comprised three different effects. Some gear used supported the generation of more than one effect setting (but only one was recorded at a time). Categories, Models and Effects: Distortion (Blues Driver, Tube Screamer, RAT2 Distortion), Modulation (Chorus, Phaser, Flanger), Delays (Digital Delay, Tape Echo, Sweep Echo), Reverb (Plate Reverb, Hall Reverb, Spring Reverb). Annotations are labeled by a trained electric guitar musician. For each tone, we provide: Guitar string number, Fret number, Guitar pickup configuration, Effect name, Effect type, Hardware modes, Knob names, Knob types, Knob settings). The dataset website is: https://egfxset.github.io/ The data can be accessed here: https://zenodo.org/record/7044411#.YxKdSWzMKEI An ISMIR extended abstract was presented in 2022: https://ismir2022.ismir.net/program/lbd/ This dataset was conceived during Iran Roman's "Deep Learning for Music Information Retrieval" course imparted in the postgraduate studies in music technology at the UNAM (Universidad Nacional Autónoma de México). The result is a combined effort between two UNAM postgraduate students (Hegel Pedroza and Gerardo Meza) and Iran Roman (NYU).

관련 논문 13편

E-commerce Product Review Dataset for Sentiment Analysis

EPRSTMT
EPRSTMT 설명 보기

The EPRSTMT dataset, also known as EPR-sentiment, is a binary sentiment analysis dataset based on product reviews on an e-commerce platform. Each sample in the dataset is labeled as either Positive or Negative. It was collected by the ICIP Lab of Beijing Normal University and has been re-organized to make it suitable for sentiment analysis tasks.

관련 논문 13편

EQ-Bench

EQ-Bench 설명 보기

This dataset contains benchmark scores for EQ-Bench, a novel benchmark designed to evaluate aspects of emotional intelligence in Large Language Models (LLMs). We assess the ability of LLMs to understand complex emotions and social interactions by asking them to predict the intensity of emotional states of characters in a dialogue. The benchmark is able to discriminate effectively between a wide range of models. We find that EQ-Bench correlates strongly with comprehensive multi-domain benchmarks like MMLU (Hendrycks et al., 2020) (r=0.97), indicating that we may be capturing similar aspects of broad intelligence. Our benchmark produces highly repeatable results using a set of 60 English-language questions. We also provide open-source code for an automated benchmarking pipeline at https://github.com/EQ-bench/EQ-Bench and a leaderboard at https://www.eqbench.com.

관련 논문 13편

End-to-end Video-based Eye-tracking

EVE
EVE 설명 보기

EVE (End-to-end Video-based Eye-tracking) is a dataset for eye-tracking. It is collected from 54 participants and consists of 4 camera views, over 12 million frames and 1327 unique visual stimuli (images, video, text), adding up to approximately 105 hours of video data in total. Official competition on Codalab: https://competitions.codalab.org/competitions/28954

관련 논문 13편

EgoDexter

EgoDexter 설명 보기

The EgoDexter dataset provides both 2D and 3D pose annotations for 4 testing video sequences with 3190 frames. The videos are recorded with body-mounted camera from egocentric viewpoints and contain cluttered backgrounds, fast camera motion, and complex interactions with various objects. Fingertip positions were manually annotated for 1485 out of 3190 frames. Source: Hand Pose Estimation via Latent 2.5D Heatmap Regression Image Source: https://handtracker.mpi-inf.mpg.de/projects/OccludedHands/EgoDexter.htm

관련 논문 13편

EuRoC MAV

EuRoC MAV 설명 보기

EuRoC MAV is a visual-inertial datasets collected on-board a Micro Aerial Vehicle (MAV). The dataset contains stereo images, synchronized IMU measurements, and accurate motion and structure ground-truth. The datasets facilitates the design and evaluation of visual-inertial localization algorithms on real flight data Source: https://projects.asl.ethz.ch/datasets/doku.php?id=kmavvisualinertialdatasets Image Source: https://projects.asl.ethz.ch/datasets/doku.php?id=kmavvisualinertialdatasets

관련 논문 13편

FE108

FE108 설명 보기

Large-scale single-object tracking dataset, containing 108 sequences with a total length of 1.5 hours. FE108 provides ground truth annotations on both the frame and event domain. The annotation frequency is up to 40Hz and 240Hz for the frame and event domains, respectively. FE108 is the largest event-frame-based dataset for single object tracking, and also offers the highest annotation frequency in the event domain.

관련 논문 13편

FaceVerse-High Quality 3D Face Dataset

FaceVerse
FaceVerse 설명 보기

FaceVerse-High Quality 3D Face Dataset contains 2,688 high-quality head scans (21 expressions from 128 identities) captured by a dense DLSR rig. For each scan, we provide the 3D model (.obj), the corresponding texture map (.jpeg) and the FaceVerse fitted model (.ply) with the same topology.

관련 논문 13편

Flare7K

Flare7K 설명 보기

Flare7K, the first nighttime flare removal dataset, which is generated based on the observation and statistic of real-world nighttime lens flares. It offers 5,000 scattering flare images and 2,000 reflective flare images, consisting of 25 types of scattering flares and 10 types of reflective flares. The 7,000 flare patterns can be randomly added to the flare-free images, forming the flare-corrupted and flare-free image pairs.

관련 논문 13편

Prophesee GEN1 Automotive Detection Dataset

GEN1 Detection
GEN1 Detection 설명 보기

Prophesee’s GEN1 Automotive Detection Dataset is the largest Event-Based Dataset to date. The dataset was recorded using a PROPHESEE GEN1 sensor with a resolution of 304×240 pixels, mounted on a car dashboard. The labels were obtained using the gray level estimation feature of the ATIS camera by labelling manually. It contains 39 hours of open road and various driving scenarios ranging from urban, highway, suburbs and countryside scenes. Manual bounding box annotations are available for two classes are present: pedestrians and cars. (Truck and buses are not labelled).

관련 논문 13편

Genome Understanding Evaluation

GUE
GUE 설명 보기

A collection of $28$ datasets across $7$ tasks constructed for genome language model evaluation. Contains seven tasks: promoter prediction. core promoter prediction, splice site prediction, covid variant classification, epigenetic marks prediction, and transcription factor binding sites prediction on human and mouse.

관련 논문 13편

Georgetown University Multilayer corpus

GUM
GUM 설명 보기

GUM is an open source multilayer English corpus of richly annotated texts from twelve text types. Annotations include: * Multiple POS tags, morphological features and lemmatization * Sentence segmentation and rough speech act * Document structure in TEI XML (paragraphs, headings, figures, etc.) * ISO date/time annotations * Speaker and addressee information (where relevant) * Constituent and dependency syntax * Information status (given, accessible, new, split antecedent) * Entity and coreference annotation, including bridging anaphora * Entity linking (Wikification) * Discourse parses in Rhetorical Structure Theory and discourse dependencies

관련 논문 13편

GenAI-Bench: Evaluating and Improving Compositional Text-to-Visual Generation

GenAI-Bench
GenAI-Bench 설명 보기

GenAI-Bench is a benchmarking framework designed to evaluate and improve compositional text-to-visual generation models. It was developed by researchers from Carnegie Mellon University and Meta. The key aspects of GenAI-Bench include: - Compositional Text-to-Visual Generation: It focuses on the ability of generative models to handle compositional text prompts that involve attributes, relationships, and higher-order reasoning such as logic and comparison¹. - Diverse Text Prompts: GenAI-Bench uses 1,600 text prompts collected from professional graphic designers to cover a wide range of compositional reasoning skills¹. - Human Studies: The framework includes human annotators who rate the performance of leading generative models like DALL-E 3, Stable Diffusion, and others based on image-text or video-text alignment¹. - Automated Evaluation Metrics: It aims to benchmark automated evaluation metrics that measure the alignment between an image and a text prompt². - Improving Generation: GenAI-Bench also explores how VQAScore, an automated metric, can improve image generation by selecting the highest-scoring images from generated candidates¹. Overall, GenAI-Bench provides a comprehensive and challenging testbed for state-of-the-art text-to-visual generative models, pushing the boundaries of what these models can achieve in terms of understanding and creating complex visual compositions based on textual descriptions. (1) GenAI-Bench: Evaluating and Improving Compositional Text-to-Visual .... https://linzhiqiu.github.io/papers/genai_bench/. (2) GenAI-Bench: A Holistic Benchmark for Compositional Text-to-Visual .... https://openreview.net/pdf?id=hJm7qnW3ym. (3) Evaluating Text-to-Visual Generation with Image-to-Text Generation. https://linzhiqiu.github.io/papers/vqascore/.

관련 논문 13편

Ghostbuster

Ghostbuster 설명 보기

The Ghostbusters dataset leverages the GPT-3.5-turbo model for generating texts in the domains of creative writing, news, and student essays, providing 2,000 texts in the first two domains and 1,994 in the latter.

관련 논문 13편

GooAQ

GooAQ 설명 보기

GooAQ is a large-scale dataset with a variety of answer types. This dataset contains over 5 million questions and 3 million answers collected from Google. GooAQ questions are collected semi-automatically from the Google search engine using its autocomplete feature. This results in naturalistic questions of practical interest that are nonetheless short and expressed using simple language. GooAQ answers are mined from Google's responses to the collected questions, specifically from the answer boxes in the search results. This yields a rich space of answer types, containing both textual answers (short and long) as well as more structured ones such as collections.

관련 논문 13편

Holistic Evaluation of Text-to-Image Models

HEIM
HEIM 설명 보기

HEIM stands for Holistic Evaluation of Text-To-Image Models. It is a comprehensive benchmark designed to assess the capabilities and risks of text-to-image generation models. Unlike previous evaluations that primarily focused on image-text alignment and image quality, HEIM considers 12 different aspects that are crucial for real-world model deployment: 1. Image-Text Alignment 2. Image Quality 3. Aesthetics 4. Originality 5. Reasoning 6. Knowledge 7. Bias 8. Toxicity 9. Fairness 10. Robustness 11. Multilinguality 12. Efficiency By curating scenarios that encompass these aspects, HEIM evaluates state-of-the-art text-to-image models. Interestingly, no single model excels in all aspects; different models demonstrate strengths in different areas. For transparency, all prompts, generated images, and results are available on the HEIM website for exploration and study. Additionally, the GitHub repository provides a collection of models accessible via a unified API, along with metrics beyond accuracy, such as efficiency, bias, and toxicity.

관련 논문 13편

Horizon Lines in the Wild

HLW
HLW 설명 보기

We introduce Horizon Lines in the Wild (HLW), a large dataset of real-world images with labeled horizon lines, captured in a diverse set of environments. The dataset is available for download at our project website [1]. We begin by characterizing limitations in existing datasets for evaluating horizon line detection methods and then describe our approach for leveraging structure from motion to automatically label images with horizon lines.