Papers with Code · CC BY-SA 4.0

데이터셋 탐색

데이터셋의 설명·홈페이지를 찾고, 관련 태스크의 평가 기록을 살펴보세요.

관련 논문 40편

LegalBench

LegalBench 설명 보기

LegalBench is a fascinating project that revolves around legal reasoning and evaluation. Let me break it down for you: 1. LegalBench Website: - The LegalBench website serves as your ultimate legal helper. It's a platform where you can hire expert lawyers for various legal tasks, all online and from anywhere. - Here's how it works: 1. Post a Job: If you have a legal task that needs completion, simply post the job on LegalBench. 2. Receive Bids: Within minutes, you'll start receiving competitive bids from freelancers who can handle your legal project. 3. Connect with Lawyers: Choose the right lawyer for your needs and get your project done. - LegalBench covers a range of legal categories, including patents and legal advice¹. 2. LegalBench Benchmark: - The LegalBench project is an ongoing open science effort to collaboratively curate tasks for evaluating legal reasoning in English large language models (LLMs). - The benchmark currently consists of 162 tasks gathered from 40 contributors. - Each task in the benchmark has an associated dataset, consisting of input-output pairs. - It's a valuable resource for assessing how well language models understand and reason about legal concepts and scenarios. - If you're curious, you can explore the LegalBench GitHub repository for more details²³. Source: Conversation with Bing, 3/16/2024 (1) LegalBench | Your ultimate legal helper. https://legalbench.com/. (2) LegalBench - GitHub. https://github.com/HazyResearch/legalbench. (3) Home | LegalBench. https://hazyresearch.stanford.edu/legalbench/. (4) LegalBench Dataset | Papers With Code. https://paperswithcode.com/dataset/legalbench.

관련 논문 40편

MIMII

MIMII 설명 보기

Sound Dataset for Malfunctioning Industrial Machine Investigation and Inspection (MIMII) is a sound dataset of industrial machine sounds. Source: MIMII Dataset: Sound Dataset for Malfunctioning Industrial Machine Investigation and Inspection

관련 논문 40편

ManiSkill2

ManiSkill2 설명 보기

ManiSkill2 is the next generation of the SAPIEN ManiSkill benchmark, to address critical pain points often encountered by researchers when using benchmarks for generalizable manipulation skills. It includes 20 manipulation task families with 2000+ object models and 4M+ demonstration frames, which cover stationary/mobile-base, single/dual-arm, and rigid/soft-body manipulation tasks with 2D/3D input data simulated by fully dynamic engines. Source: ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skills Image Source: https://arxiv.org/pdf/2302.04659v1.pdf

관련 논문 40편

RedCaps

RedCaps 설명 보기

RedCaps is a large-scale dataset of 12M image-text pairs collected from Reddit. Images and captions from Reddit depict and describe a wide variety of objects and scenes. The data is collected from a manually curated set of subreddits (350 total), which give coarse image labels and allow steering of the dataset composition without labeling individual instances. Terms of use: Uses of RedCaps are subject to Reddit API terms. Users must comply with Reddit User Agreeement, Content Policy, and Privacy Policy. Usage Restrictions: RedCaps should only be used for non-commercial research. RedCaps should not be used for any tasks that involve identifying features related to people (facial recognition, gender, age, ethnicity identification, etc.) or make decisions that impact people (mortgages, job applications, criminal sentences; or moderation decisions about user-uploaded data that could result in bans from a website). Any commercial and for-profit uses of RedCaps are restricted – it should not be used to train models that will be deployed in production systems as part of a product offered by businesses or government agencies Refer to the datasheet in the paper more details. Image source: https://redcaps.xyz/download

관련 논문 40편

Samanantar

Samanantar 설명 보기

Samanantar is the largest publicly available parallel corpora collection for Indic languages: Assamese, Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Oriya, Punjabi, Tamil, Telugu. The corpus has 49.6M sentence pairs between English to Indian Languages.

관련 논문 40편

TheoremQA

TheoremQA 설명 보기

We propose the first question-answering dataset driven by STEM theorems. We annotated 800 QA pairs covering 350+ theorems spanning across Math, EE&CS, Physics and Finance. The dataset is collected by human experts with very high quality. We provide the dataset as a new benchmark to test the limit of large language models to apply theorems to solve challenging university-level questions. We provide a pipeline in the following to prompt LLMs and evaluate their outputs with WolframAlpha.

관련 논문 40편

VeRi-Wild

VeRi-Wild 설명 보기

Veri-Wild is the largest vehicle re-identification dataset (as of CVPR 2019). The dataset is captured from a large CCTV surveillance system consisting of 174 cameras across one month (30× 24h) under unconstrained scenarios. This dataset comprises 416,314 vehicle images of 40,671 identities. Evaluation on this dataset is split across three subsets: small, medium and large; comprising 3000, 5000 and 10,000 identities respectively (in probe and gallery sets). Source: Vehicle Re-Identification: an Efficient Baseline Using Triplet Embedding Image Source: https://github.com/PKU-IMRE/VERI-Wild

관련 논문 39편

Automatic LLMs' Citation Evaluation

ALCE
ALCE 설명 보기

ALCE is a benchmark for Automatic LLMs' Citation Evaluation. ALCE collects a diverse set of questions and retrieval corpora and requires building end-to-end systems to retrieve supporting evidence and generate answers with citations. Source: Enabling Large Language Models to Generate Text with Citations Image Source: Enabling Large Language Models to Generate Text with Citations

관련 논문 39편

AP-10K

AP-10K 설명 보기

AP-10K is the first large-scale benchmark for general animal pose estimation, to facilitate the research in animal pose estimation. AP-10K consists of 10,015 images collected and filtered from 23 animal families and 60 species following the taxonomic rank and high-quality keypoint annotations labeled and checked manually. Source: AP-10K: A Benchmark for Animal Pose Estimation in the Wild Image source: https://arxiv.org/pdf/2108.12617v1.pdf

관련 논문 39편

Argoverse 2 Motion Forecasting

Argoverse 2 Motion Forecasting 설명 보기

The Argoverse 2 Motion Forecasting Dataset is a curated collection of 250,000 scenarios for training and validation. Each scenario is 11 seconds long and contains the 2D, birds-eye-view centroid and heading of each tracked object sampled at 10 Hz. To curate this collection, we sifted through thousands of hours of driving data from our fleet of self-driving test vehicles to find the most challenging segments. We place special emphasis on kinematically and socially unusual behavior, especially when exhibited by actors relevant to the ego-vehicle’s decision-making process. Some examples of interactions captured within our dataset include: buses navigating through multi-lane intersections, vehicles yielding to pedestrians at crosswalks, and cyclists sharing dense city streets. Spanning 2,000+ km over six geographically diverse cities, Argoverse 2 covers a large geographic area. Argoverse 2 also contains a large object taxonomy with 10 non-overlapping classes that encompass a broad range of actors, both static and dynamic. In comparison to the Argoverse 1 Motion Forecasting Dataset, the scenarios in this dataset are approximately twice as long and more diverse. Together, these changes incentivize methods that perform well on extended forecast horizons, handle multiple types of dynamic objects, and ensure safety in long tail scenarios.

관련 논문 39편

BREAK

BREAK 설명 보기

Break is a question understanding dataset, aimed at training models to reason over complex questions. It features 83,978 natural language questions, annotated with a new meaning representation, Question Decomposition Meaning Representation (QDMR). Each example has the natural question along with its QDMR representation. Break contains human composed questions, sampled from 10 leading question-answering benchmarks over text, images and databases. This dataset was created by a team of NLP researchers at Tel Aviv University and Allen Institute for AI. Source: BREAK

관련 논문 39편

BookSum

BookSum 설명 보기

BookSum is a collection of datasets for long-form narrative summarization. This dataset covers source documents from the literature domain, such as novels, plays and stories, and includes highly abstractive, human written summaries on three levels of granularity of increasing difficulty: paragraph-, chapter-, and book-level. The domain and structure of this dataset poses a unique set of challenges for summarization systems, which include: processing very long documents, non-trivial causal and temporal dependencies, and rich discourse structures. BookSum contains summaries for 142,753 paragraphs, 12,293 chapters and 436 books.

관련 논문 39편

Campus & Shelf

Campus & Shelf 설명 보기

The Campus and Shelf datasets were presented in the paper 3D Pictorial Structures for Multiple Human Pose Estimation. The first dataset shows persons walking and talking in front of a building, and the second up to four persons assembling a shelf.

관련 논문 39편

CulturaX

CulturaX 설명 보기

We present CulturaX, a substantial multilingual dataset with 6.3 trillion tokens in 167 languages, tailored for large language model (LLM) development. Our dataset undergoes meticulous cleaning and deduplication through a rigorous pipeline of multiple stages to accomplish the best quality for model training, including language identification, URL-based filtering, metric-based cleaning, document refinement, and data deduplication. We employ MinHash at document level to achieve fuzzy deduplication for the datasets in different languages. Our data cleaning framework includes diverse criteria and threshold selections, guided by extensive data samples, ensuring comprehensive noise filtering in various aspects. CulturaX is fully released to the public in HuggingFace to facilitate research and advancements in multilingual LLMs. Our dataset combines the most recent iteration of mC4 (version 3.1.0) [1] with all accessible OSCAR corpora up to the present year, including 20.19, 21.09, 22.01, and 23.01 [2]. After deep cleaning and deduplication, CulturaX involves 16TB data in the parquet format (expanding to 27TB when unpacked). More than a half of our dataset is dedicated to non-English languages to significantly boost the data size and enhance the feasibility of training models in multilingual scenarios. To obtain perplexity scores for data cleaning, we train a SentencePiece tokenizer and 5-gram Kneser-Ney language models as provided in the KenLM library [3] using the 20230501 dumps of Wikipedia. Our KenLM models are also released in HuggingFace: https://huggingface.co/uonlp/kenlm. Details for the dataset can be found in our technical paper: https://arxiv.org/abs/2309.09400 and https://huggingface.co/datasets/uonlp/CulturaX

관련 논문 39편

Dichotomous Image Segmentation (DIS) Dataset

DIS5K
DIS5K 설명 보기

To build the highly accurate Dichotomous Image Segmentation dataset (DIS5K), we first manually collected over 12,000 images from Flickr1 based on our pre-designed keywords. Then, we obtained 5,470 images of 22 groups and 225 categories from the 12,000 images according to the structural complexities of the objects. Each image is then manually labeled with pixel-wise accuracy using GIMP. The labeled targets in DIS5K mainly focus on the “objects of the images defined by the pre-designed keywords (categories)” regardless of their characteristics e.g., salient, common, camouflaged, meticulous, etc. The average per-image labeling time is ∼30 minutes and some images cost up to 10 hours.

관련 논문 39편

FIN

FIN 설명 보기

A dataset of financial agreements made public through U.S. Security and Exchange Commission (SEC) filings. Eight documents (totalling 54,256 words) were randomly selected for manual annotation, based on the four NE types provided in the CoNLL-2003 dataset: LOCATION (LOC), ORGANISATION (ORG), PERSON (PER), and MISCELLANEOUS (MISC).

관련 논문 39편

Glint360K

Glint360K 설명 보기

The largest and cleanest face recognition dataset Glint360K, which contains 17,091,657 images of 360,232 individuals, baseline models trained on Glint360K can easily achieve state-of-the-art performance.

관련 논문 39편

Honda Research Institute 3D

H3D
H3D 설명 보기

The H3D is a large scale full-surround 3D multi-object detection and tracking dataset. It is gathered from HDD dataset, a large scale naturalistic driving dataset collected in San Francisco Bay Area. H3D consists of following features: * Full 360 degree LiDAR dataset (dense pointcloud from Velodyne-64) * 160 crowded and highly interactive traffic scenes * 1,071,302 3D bounding box labels * 8 common classes of traffic participants (Manually annotated every 2Hz and linearly propagated for 10 Hz data) * Benchmarked on state-of-the art algorithms for 3D only detection and tracking algorithms.

관련 논문 39편

MTG-Jamendo

MTG-Jamendo 설명 보기

The MTG-Jamendo dataset is an open dataset for music auto-tagging. The dataset contains over 55,000 full audio tracks with 195 tags categories (87 genre tags, 40 instrument tags, and 56 mood/theme tags). It is built using music available at Jamendo under Creative Commons licenses and tags provided by content uploaders. All audio is distributed in 320kbps MP3 format. A subset of the dataset is used in the Emotion and Theme Recognition in Music Task within MediaEval 2019. Source: https://mtg.github.io/mtg-jamendo-dataset/ Audio Source: https://essentia.upf.edu/datasets/mtg-jamendo/raw_30s/audio/

관련 논문 39편

MemeTracker

MemeTracker 설명 보기

The Memetracker corpus contains articles from mainstream media and blogs from August 1 to October 31, 2008 with about 1 million documents per day. It has 10,967 hyperlink cascades among 600 media sites. Source: Marked Temporal Dynamics Modeling based on Recurrent Neural Network Image Source: http://blog.fabric.ch/index.php?/archives/292-Memetracker-Tracking-News-Phrases-over-the-Web.html

관련 논문 39편

RoadTracer

RoadTracer 설명 보기

RoadTracer is a dataset for extraction of road networks from aerial images. It consists of a large corpus of high-resolution satellite imagery and ground truth road network graphs covering the urban core of forty cities across six countries. For each city, the dataset covers a region of approximately 24 sq km around the city center. The satellite imagery is obtained from Google at 60 cm/pixel resolution, and the road network from OSM. The dataset is split into a training set with 25 cities and a test set with 15 other cities. Source: RoadTracer: Automatic Extraction of Road Networks from Aerial Images Image Source: Bastani et al

관련 논문 39편

SIXray

SIXray 설명 보기

The SIXray dataset is constructed by the Pattern Recognition and Intelligent System Development Laboratory, University of Chinese Academy of Sciences. It contains 1,059,231 X-ray images which are collected from some several subway stations. There are six common categories of prohibited items, namely, gun, knife, wrench, pliers, scissors and hammer. It has three subsets called SIXray10, SIXray100 and SIXray1000, There are image-level annotations provided by human security inspectors for the whole dataset. In addition the images in the test set are annotated with a bounding-box for each prohibited item to evaluate the performance of object localization. Source: https://github.com/MeioJane/SIXray Image Source: https://github.com/MeioJane/SIXray

관련 논문 39편

Task Directed Image Understanding Challenge

TDIUC
TDIUC 설명 보기

Task Directed Image Understanding Challenge (TDIUC) dataset is a Visual Question Answering dataset which consists of 1.6M questions and 170K images sourced from MS COCO and the Visual Genome Dataset. The image-question pairs are split into 12 categories and 4 additional evaluation matrices which help evaluate models’ robustness against answer imbalance and its ability to answer questions that require higher reasoning capability. The TDIUC dataset divides the VQA paradigm into 12 different task directed question types. These include questions that require a simpler task (e.g., object presence, color attribute) and more complex tasks (e.g., counting, positional reasoning). The dataset includes also an “Absurd” question category in which questions are irrelevant to the image contents to help balance the dataset. Source: Question-Agnostic Attention for Visual Question Answering Image Source: https://kushalkafle.com/projects/tdiuc.html

관련 논문 39편

University

University 설명 보기

Univesity is an indoor localization dataset proposed in "Camera Relocalization by Computing Pairwise Relative Poses Using Convolutional Neural Network". It contains camera images and ground-truth poses, split in 9694 training pairs and 5068 test pairs. It was created to test localization of multiple scenes in one global coordinate frame, and has a bigger indoor environment compared to 7scenes.

관련 논문 39편

Large Scale Vertebrae Segmentation Challenge

VerSe
VerSe 설명 보기

Spine or vertebral segmentation is a crucial step in all applications regarding automated quantification of spinal morphology and pathology. With the advent of deep learning, for such a task on computed tomography (CT) scans, a big and varied data is a primary sought-after resource. However, a large-scale, public dataset is currently unavailable. VerSe is a large scale, multi-detector, multi-site, CT spine dataset consisting of 374 scans from 355 patients. The challenge was held in two iterations in conjunction with MICCAI 2019 and 2020. The tasks evaluated for include: vertebral labelling and segmentation. Image source: https://arxiv.org/pdf/2001.09193v5.pdf

관련 논문 38편

AISHELL-3

AISHELL-3 설명 보기

AISHELL-3 is a large-scale and high-fidelity multi-speaker Mandarin speech corpus which could be used to train multi-speaker Text-to-Speech (TTS) systems. The corpus contains roughly 85 hours of emotion-neutral recordings spoken by 218 native Chinese mandarin speakers and total 88035 utterances. Their auxiliary attributes such as gender, age group and native accents are explicitly marked and provided in the corpus. Accordingly, transcripts in Chinese character-level and pinyin-level are provided along with the recordings. The word & tone transcription accuracy rate is above 98%, through professional speech annotation and strict quality inspection for tone and prosody.

관련 논문 38편

Biomedical Semantic Similarity Estimation System

BIOSSES
BIOSSES 설명 보기

The BIOSSES data set comprises total 100 sentence pairs all of which were selected from the "TAC2 Biomedical Summarization Track Training Data Set" . The sentence pairs were evaluated by five different human experts that judged their similarity and gave scores in a range [0-4]. Our guideline was prepared based on SemEval 2012 Task 6 Guideline. Image source: BIOSSES

관련 논문 38편

BigCodeBench

BigCodeBench 설명 보기

BigCodeBench is an easy-to-use benchmark for code generation with practical and challenging programming tasks¹. It aims to evaluate the true programming capabilities of large language models (LLMs) in a more realistic setting¹. The benchmark is designed for HumanEval-like function-level code generation tasks, but with much more complex instructions and diverse function calls¹. Here are some key features of BigCodeBench: - Precise evaluation & ranking: It provides a leaderboard for latest LLM rankings before & after rigorous evaluation¹. - Pre-generated samples: BigCodeBench accelerates code intelligence research by open-sourcing LLM-generated samples for various models¹. - Execution Environment: The execution environment in BigCodeBench is less bounded than EvalPlus to support tasks with diverse library dependencies¹. - Test Evaluation: BigCodeBench relies on unittest for evaluating the generated code¹. (1) GitHub - bigcode-project/bigcodebench: BigCodeBench: The Next .... https://github.com/bigcode-project/bigcodebench/.

관련 논문 38편

Contract Understanding Atticus Dataset

CUAD
CUAD 설명 보기

Contract Understanding Atticus Dataset (CUAD) is a dataset for legal contract review. CUAD was created with dozens of legal experts from The Atticus Project and consists of over 13,000 annotations. The task is to highlight salient portions of a contract that are important for a human to review.

관련 논문 38편

CVEfixes

CVEfixes 설명 보기

CVEfixes is a comprehensive vulnerability dataset that is automatically collected and curated from Common Vulnerabilities and Exposures (CVE) records in the public U.S. National Vulnerability Database (NVD). The goal is to support data-driven security research based on source code and source code metrics related to fixes for CVEs in the NVD by providing detailed information at different interlinked levels of abstraction, such as the commit-, file-, and method level, as well as the repository- and CVE level. At the initial release, the dataset covers all published CVEs up to 9 June 2021. All open-source projects that were reported in CVE records in the NVD in this time frame and had publicly available git repositories were fetched and considered for the construction of this vulnerability dataset. The dataset is organized as a relational database and covers 5495 vulnerability fixing commits in 1754 open source projects for a total of 5365 CVEs in 180 different Common Weakness Enumeration (CWE) types. The dataset includes the source code before and after fixing of 18249 files, and 50322 functions.

관련 논문 38편

Chinese IDiom dataset

ChID
ChID 설명 보기

ChID is a large-scale Chinese IDiom dataset for cloze test. ChID contains 581K passages and 729K blanks, and covers multiple domains. In ChID, the idioms in a passage were replaced with blank symbols. For each blank, a list of candidate idioms including the golden idiom are provided as choice. Source: ChID: A Large-scale Chinese IDiom Dataset for Cloze Test Image Source: https://arxiv.org/pdf/1906.01265v3.pdf

관련 논문 38편

Conversational Finance Question Answering

ConvFinQA
ConvFinQA 설명 보기

ConvFinQA is a dataset designed to study the chain of numerical reasoning in conversational question answering. The dataset contains 3892 conversations containing 14115 questions where 2715 of the conversations are simple conversations, and the rest 1,177 are hybrid conversations. Source: ConvFinQA: Exploring the Chain of Numerical Reasoning in Conversational Finance Question Answering Image Source: https://arxiv.org/pdf/2210.03849.pdf

관련 논문 38편

DAIR-V2X

DAIR-V2X 설명 보기

DAIR-V2X is a large-scale, multi-modality, multi-view dataset from real scenarios for VICAD. DAIR-V2X comprises 71254 LiDAR frames and 71254 Camera frames, and all frames are captured from real scenes with 3D annotations.

관련 논문 38편

ECE

ECE 설명 보기

The ECE dataset (Gui et al., 2016a) is collected from SINA city news and contains 2105 instances. Its document has only one emotion word and one or more emotion causes.

관련 논문 38편

EMOTIons in Context

EMOTIC
EMOTIC 설명 보기

The EMOTIC dataset, named after EMOTions In Context, is a database of images with people in real environments, annotated with their apparent emotions. The images are annotated with an extended list of 26 emotion categories combined with the three common continuous dimensions Valence, Arousal and Dominance. Source: Context Based Emotion Recognition using EMOTIC Dataset

관련 논문 38편

FewRel 2.0

FewRel 2.0 설명 보기

A more challenging task to investigate two aspects of few-shot relation classification models: (1) Can they adapt to a new domain with only a handful of instances? (2) Can they detect none-of-the-above (NOTA) relations? Source: FewRel 2.0: Towards More Challenging Few-Shot Relation Classification

관련 논문 38편

GSM-Hard

GSM-Hard 설명 보기

Click to add a brief description of the dataset (Markdown and LaTeX enabled). Provide: * a high-level explanation of the dataset characteristics * explain motivations and summary of its content * potential use cases of the dataset

관련 논문 38편

GazeFollow

GazeFollow 설명 보기

GazeFollow is a large-scale dataset annotated with the location of where people in images are looking. It uses several major datasets that contain people as a source of images: 1, 548 images from SUN, 33, 790 images from MS COCO, 9, 135 images from Actions 40, 7, 791 images from PASCAL, 508 images from the ImageNet detection challenge and 198, 097 images from the Places dataset. This concatenation results in a challenging and large image collection of people performing diverse activities in many everyday scenarios. Source: GazeFollow

관련 논문 38편

Honda Research Institute Driving Dataset

HDD
HDD 설명 보기

Honda Research Institute Driving Dataset (HDD) is a dataset to enable research on learning driver behavior in real-life environments. The dataset includes 104 hours of real human driving in the San Francisco Bay Area collected using an instrumented vehicle equipped with different sensors. Source: Toward Driving Scene Understanding: A Dataset for Learning Driver Behavior and Causal Reasoning Image Source: https://usa.honda-ri.com/hdd

관련 논문 38편

InsuranceQA

InsuranceQA 설명 보기

InsuranceQA is a question answering dataset for the insurance domain, the data stemming from the website Insurance Library. There are 12,889 questions and 21,325 answers in the training set. There are 2,000 questions and 3,354 answers in the validation set. There are 2,000 questions and 3,308 answers in the test set. Source: APPLYING DEEP LEARNING TO ANSWER SELECTION: A STUDY AND AN OPEN TASK

관련 논문 38편

LTCC

LTCC 설명 보기

LTCC contains 17,119 person images of 152 identities, and each identity is captured by at least two cameras. The dataset can be divided into two subsets: one cloth-change set where 91 persons appear with 416 different sets of outfits in 14,783 images, and one cloth-consistent subset containing the remaining 61 identities with 2,336 images without outfit changes. On average, there are 5 different clothes for each cloth-changing person, with the number of outfit changes ranging from 2 to 14. Source: LTCC dataset homepage

관련 논문 38편

ManiSkill

ManiSkill 설명 보기

ManiSkill is a large-scale learning-from-demonstrations benchmark for articulated object manipulation with visual input (point cloud and image). ManiSkill supports object-level variations by utilizing a rich and diverse set of articulated objects, and each task is carefully designed for learning manipulations on a single category of objects. ManiSkill is equipped with high-quality demonstrations to facilitate learning-from-demonstrations approaches and perform evaluations on common baseline algorithms. ManiSkill can encourage the robot learning community to explore more on learning generalizable object manipulation skills. Source: ManiSkill: Learning-from-Demonstrations Benchmark for Generalizable Manipulation Skills Image source: ManiSkill: Learning-from-Demonstrations Benchmark for Generalizable Manipulation Skills

관련 논문 38편

OpenML-CC18

OpenML-CC18 설명 보기

We advocate the use of curated, comprehensive benchmark suites of machine learning datasets, backed by standardized OpenML-based interfaces and complementary software toolkits written in Python, Java and R. We demonstrate how to easily execute comprehensive benchmarking studies using standardized OpenML-based benchmarking suites and complementary software toolkits written in Python, Java and R. Major distinguishing features of OpenML benchmark suites are (i) ease of use through standardized data formats, APIs, and existing client libraries; (ii) machine-readable meta-information regarding the contents of the suite; and (iii) online sharing of results, enabling large scale comparisons. As a first such suite, we propose the OpenML-CC18, a machine learning benchmark suite of 72 classification datasets carefully curated from the thousands of datasets on OpenML. The inclusion criteria are: * classification tasks on dense data set independent observations * number of classes >= 2, each class with at least 20 observations and ratio of minority to majority class must exceed 5% * 500 <= number of observations <= 100000 * number of features after one-hot-encoding < 5000 * no artificial data sets * no subsets of larger data sets nor binarizations of other data sets * no data sets which are perfectly predictable by using a single feature or by using a simple decision tree * source or reference available If you use this benchmarking suite, please cite: Bernd Bischl, Giuseppe Casalicchio, Matthias Feurer, Frank Hutter, Michel Lang, Rafael G. Mantovani, Jan N. van Rijn and Joaquin Vanschoren. “OpenML Benchmarking Suites” arXiv:1708.03731v2 [stats.ML] (2019). @article{oml-benchmarking-suites, title={OpenML Benchmarking Suites}, author={Bernd Bischl and Giuseppe Casalicchio and Matthias Feurer and Frank Hutter and Michel Lang and Rafael G. Mantovani and Jan N. van Rijn and Joaquin Vanschoren}, year={2019}, journal={arXiv:1708.03731v2 [stat.ML]} }