Papers with Code · CC BY-SA 4.0

데이터셋 탐색

데이터셋의 설명·홈페이지를 찾고, 관련 태스크의 평가 기록을 살펴보세요.

관련 논문 28편

EyeQ

EyeQ 설명 보기

Dataset with 28,792 retinal images from the EyePACS dataset, based on a three-level quality grading system (i.e., Good', Usable' and `Reject') for evaluating RIQA methods. Source: Evaluation of Retinal Image Quality Assessment Networks in Different Color-spaces

관련 논문 28편

Gaming 3D Dataset

G3D
G3D 설명 보기

The Gaming 3D Dataset (G3D) focuses on real-time action recognition in a gaming scenario. It contains 10 subjects performing 20 gaming actions: “punch right”, “punch left”, “kick right”, “kick left”, “defend”, “golf swing”, “tennis swing forehand”, “tennis swing backhand”, “tennis serve”, “throw bowling ball”, “aim and fire gun”, “walk”, “run”, “jump”, “climb”, “crouch”, “steer a car”, “wave”, “flap” and “clap”. Source: Skeleton Based Action Recognition Using Translation-Scale Invariant Image Mapping And Multi-Scale Deep CNN Image Source: G3D: A gaming action dataset and real time action recognition evaluation framework

관련 논문 28편

How2QA

How2QA 설명 보기

To collect How2QA for video QA task, the same set of selected video clips are presented to another group of AMT workers for multichoice QA annotation. Each worker is assigned with one video segment and asked to write one question with four answer candidates (one correctand three distractors). Similarly, narrations are hidden from the workers to ensure the collected QA pairs are not biased by subtitles. Similar to TVQA, the start and end points are provided for the relevant moment for each question. After filtering low-quality annotations, the final dataset contains 44,007 QA pairs for 22k 60-second clips selected from 9035 videos. Source: HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training

관련 논문 28편

IndicCorp

IndicCorp 설명 보기

IndicCorp is a large monolingual corpora with around 9 billion tokens covering 12 of the major Indian languages. It has been developed by discovering and scraping thousands of web sources - primarily news, magazines and books, over a duration of several months. Languages covered: Assamese, Bengali, English, Gujarati, Hindi, Kannada, Malayalam, Marathi, Oriya, Punjabi, Tamil, Telugu Corpus Format: The corpus is a single large text file containing one sentence per line. The publicly released version is randomly shuffled, untokenized and deduplicated. Downloads | Language | # News Articles* | Sentences | Tokens | Link | | -------- | ----------------- | ------------- | ------------- | -------- | | as | 0.60M | 1.39M | 32.6M | link | | bn | 3.83M | 39.9M | 836M | link | | en | 3.49M | 54.3M | 1.22B | link | | gu | 2.63M | 41.1M | 719M | link | | hi | 4.95M | 63.1M | 1.86B | link | | kn | 3.76M | 53.3M | 713M | link | | ml | 4.75M | 50.2M | 721M | link | | mr | 2.31M | 34.0M | 551M | link | | or | 0.69M | 6.94M | 107M | link | | pa | 2.64M | 29.2M | 773M | link | | ta | 4.41M | 31.5M | 582M | link | | te | 3.98M | 47.9M | 674M | link | * Excluding articles obtained from the OSCAR corpus

관련 논문 28편

KITTI Multi-Object Tracking and Segmentation (MOTS) Evaluation

KITTI MOTS
KITTI MOTS 설명 보기

The Multi-Object and Segmentation (MOTS) benchmark [2] consists of 21 training sequences and 29 test sequences. It is based on the KITTI Tracking Evaluation 2012 and extends the annotations to the Multi-Object and Segmentation (MOTS) task. To this end, we added dense pixel-wise segmentation labels for every object. We evaluate submitted results using the metrics HOTA, CLEAR MOT, and MT/PT/ML. We rank methods by HOTA [1]. Our development kit and GitHub evaluation code provide details about the data format as well as utility functions for reading and writing the label files. (adapted for the segmentation case). Evaluation is performed using the code from the TrackEval repository. [1] J. Luiten, A. Os̆ep, P. Dendorfer, P. Torr, A. Geiger, L. Leal-Taixé, B. Leibe: HOTA: A Higher Order Metric for Evaluating Multi-object Tracking. IJCV 2020. [2] P. Voigtlaender, M. Krause, A. Os̆ep, J. Luiten, B. Sekar, A. Geiger, B. Leibe: MOTS: Multi-Object Tracking and Segmentation. CVPR 2019.

관련 논문 28편

KaggleDBQA: Realistic Text-to-SQL dataset

KaggleDBQA
KaggleDBQA 설명 보기

KaggleDBQA is a challenging cross-domain and complex evaluation dataset of real Web databases, with domain-specific data types, original formatting, and unrestricted questions. It expands upon contemporary cross-domain text-to-SQL datasets in three key aspects: (1) Its databases are pulled from real-world data sources and not normalized. (2) Its questions are authored in environments that mimic natural question answering. (3) It also provides database documentation that contains rich in-domain knowledge.

관련 논문 28편

Minimalist Histopathology image analysis dataset

MHIST
MHIST 설명 보기

The minimalist histopathology image analysis dataset (MHIST) is a binary classification dataset of 3,152 fixed-size images of colorectal polyps, each with a gold-standard label determined by the majority vote of seven board-certified gastrointestinal pathologists. MHIST also includes each image’s annotator agreement level. As a minimalist dataset, MHIST occupies less than 400 MB of disk space, and a ResNet-18 baseline can be trained to convergence on MHIST in just 6 minutes using approximately 3.5 GB of memory on a NVIDIA RTX 3090. As example use cases, the authors use MHIST to study natural questions that arise in histopathology image classification such as how dataset size, network depth, transfer learning, and high-disagreement examples affect model performance. Source: Wei et al. Image source: Wei et al.

관련 논문 28편

MIT-Adobe FiveK

MIT-Adobe FiveK 설명 보기

The MIT-Adobe FiveK dataset consists of 5,000 photographs taken with SLR cameras by a set of different photographers. They are all in RAW format; that is, all the information recorded by the camera sensor is preserved. We made sure that these photographs cover a broad range of scenes, subjects, and lighting conditions. We then hired five photography students in an art school to adjust the tone of the photos. Each of them retouched all the 5,000 photos using a software dedicated to photo adjustment (Adobe Lightroom) on which they were extensively trained. We asked the retouchers to achieve visually pleasing renditions, akin to a postcard. The retouchers were compensated for their work. This dataset was collected for our project on learning photographic adjustments. When using images from this dataset, please cite this dataset using the following BibTeX: @inproceedings{fivek, author = "Vladimir Bychkovsky and Sylvain Paris and Eric Chan and Fr{\'e}do Durand", title = "Learning Photographic Global Tonal Adjustment with a Database of Input / Output Image Pairs", booktitle = "The Twenty-Fourth IEEE Conference on Computer Vision and Pattern Recognition", year = "2011" } Source: https://data.csail.mit.edu/graphics/fivek/ Image source: https://data.csail.mit.edu/graphics/fivek/

관련 논문 28편

Metastable crystal structures from Materials Project

MP20
MP20 설명 보기

MP20 (Xie et al., 2022) contains 45,231 metastable crystal structures from the Materials Project (Jain et al., 2013), each with up to 20 atoms and spanning 89 different element types.

관련 논문 28편

MR Movie Reviews

MR
MR 설명 보기

MR Movie Reviews is a dataset for use in sentiment-analysis experiments. Available are collections of movie-review documents labeled with respect to their overall sentiment polarity (positive or negative) or subjective rating (e.g., "two and a half stars") and sentences labeled with respect to their subjectivity status (subjective or objective) or polarity. Source: http://www.cs.cornell.edu/people/pabo/movie-review-data/ Image Source: https://storage.googleapis.com/kaggle-competitions/kaggle/3810/media/treebank.png

관련 논문 28편

Multi Vehicle Stereo Event Camera

MVSEC
MVSEC 설명 보기

The Multi Vehicle Stereo Event Camera (MVSEC) dataset is a collection of data designed for the development of novel 3D perception algorithms for event based cameras. Stereo event data is collected from car, motorbike, hexacopter and handheld data, and fused with lidar, IMU, motion capture and GPS to provide ground truth pose and depth images. Source: EV-FlowNet: Self-Supervised Optical Flow Estimation for Event-based Cameras

관련 논문 28편

MannequinChallenge

MannequinChallenge 설명 보기

The MannequinChallenge Dataset (MQC) provides in-the-wild videos of people in static poses while a hand-held camera pans around the scene. The dataset consists of three splits for training, validation and testing. Source: Weakly-Supervised 3D Human Pose Learning via Multi-view Images in the Wild Image Source: https://google.github.io/mannequinchallenge/www/index.html

관련 논문 28편

MedICaT

MedICaT 설명 보기

MedICaT is a dataset of medical images, captions, subfigure-subcaption annotations, and inline textual references. Figures and captions are extracted from open access articles in PubMed Central and corresponding reference text is derived from S2ORC. The dataset consists of: 217,060 figures from 131,410 open access papers 7507 subcaption and subfigure annotations for 2069 compound figures Inline references for ~25K figures in the ROCO dataset Source: https://github.com/allenai/medicat

관련 논문 28편

Open Annotations of Single Image Surfaces

OASIS
OASIS 설명 보기

A dataset for single-image 3D in the wild consisting of annotations of detailed 3D geometry for 140,000 images. Source: OASIS: A Large-Scale Dataset for Single Image 3D in the Wild

관련 논문 28편

Oxford Radar RobotCar Dataset

Oxford Radar RobotCar Dataset 설명 보기

The Oxford Radar RobotCar Dataset is a radar extension to The Oxford RobotCar Dataset. It has been extended with data from a Navtech CTS350-X Millimetre-Wave FMCW radar and Dual Velodyne HDL-32E LIDARs with optimised ground truth radar odometry for 280 km of driving around Oxford, UK (in addition to all sensors in the original Oxford RobotCar Dataset). Source: The Oxford Radar RobotCar Dataset: A Radar Extension to the Oxford RobotCar Dataset

관련 논문 28편

PMIndia

PMIndia 설명 보기

Consists of parallel sentences which pair 13 major languages of India with English. The corpus includes up to 56000 sentences for each language pair. Source: PMIndia -- A Collection of Parallel Corpora of Languages of India

관련 논문 28편

RoboNet

RoboNet 설명 보기

An open database for sharing robotic experience, which provides an initial pool of 15 million video frames, from 7 different robot platforms, and study how it can be used to learn generalizable models for vision-based robotic manipulation. Source: RoboNet: Large-Scale Multi-Robot Learning

관련 논문 28편

SBU-Kinect-Interaction dataset v2.0

SBU / SBU-Refine
SBU / SBU-Refine 설명 보기

SBU-Kinect-Interaction dataset version 2.0 comprises of RGB-D video sequences of humans performing interaction activities that are recording using the Microsoft Kinect sensor. This dataset was originally recorded for a class project, and it must be used only for the purposes of research. If you use this dataset in your work, please cite the following paper. Kiwon Yun, Jean Honorio, Debaleena Chattopadhyay, Tamara L. Berg, and Dimitris Samaras, The 2nd International Workshop on Human Activity Understanding from 3D Data at Conference on Computer Vision and Pattern Recognition (HAU3D-CVPRW), CVPR 2012 SBU-Refine: SBU-Refine relabels the test set manually and refines the noise labels in training set by algorithm. H. Yang, T. Wang, X. Hu, and C.-W. Fu, “SILT: Shadow-aware iterative label tuning for learning to detect shadows from noisy labels,” in ICCV, 2023, pp. 12 687–12 698.

관련 논문 28편

SafetyBench

SafetyBench 설명 보기

SafetyBench is a comprehensive benchmark designed to evaluate the safety of large language models (LLMs) using multiple-choice questions. As LLMs become increasingly prevalent, concerns about their safety have grown. SafetyBench addresses this by providing a reliable evaluation framework for researchers and developers. Here are the key points about SafetyBench: 1. Purpose: SafetyBench aims to help researchers and developers better understand and assess the safety of LLMs. It serves as a reference for model selection and optimization, promoting the development of safe, responsible, and ethical large models that align with legislative norms, social standards, and human values¹². 2. Comprehensive Benchmark: SafetyBench comprises 11,435 diverse multiple-choice questions across 7 distinct categories related to safety concerns. These questions cover a wide range of topics, allowing for thorough evaluation of LLM safety¹. 3. Multilingual Evaluation: SafetyBench includes both Chinese and English data, making it suitable for evaluating LLMs in both languages. Researchers can assess model safety across different linguistic contexts¹. 4. Performance Insights: Extensive tests using SafetyBench on 25 popular Chinese and English LLMs (including zero-shot and few-shot settings) revealed that GPT-4 outperformed its counterparts. However, there is still room for improvement in enhancing the safety of existing LLMs¹. 5. Availability: Data and evaluation guidelines for SafetyBench are accessible through the following URLs: - SafetyBench Data and Guidelines - SafetyBench Submission Entrance and Leaderboard¹⁴. In summary, SafetyBench provides a valuable resource for assessing and advancing the safety of large language models, contributing to their responsible deployment and alignment with societal norms and values. (1) [2309.07045] SafetyBench: Evaluating the Safety of Large Language .... https://arxiv.org/abs/2309.07045. (2) SafetyBench:通过单选题评估大型语言模型安全性. https://posts.careerengine.us/p/6511afb61a8da974e9d62f40. (3) thu-coai/SafetyBench · Datasets at Hugging Face. https://huggingface.co/datasets/thu-coai/SafetyBench. (4) SafetyBench:通过单选题评估大型语言模型安全性_鲟曦研习社. https://www.kuxai.com/article/1505. (5) GitHub - thu-coai/SafetyBench: Official github repo for SafetyBench, a .... https://github.com/thu-coai/SafetyBench. (6) undefined. https://doi.org/10.48550/arXiv.2309.07045.

관련 논문 28편

Segmentation of THoracic Organs at Risk

SegTHOR
SegTHOR 설명 보기

SegTHOR (Segmentation of THoracic Organs at Risk) is a dataset dedicated to the segmentation of organs at risk (OARs) in the thorax, i.e. the organs surrounding the tumour that must be preserved from irradiations during radiotherapy. In this dataset, the OARs are the heart, the trachea, the aorta and the esophagus, which have varying spatial and appearance characteristics. The dataset includes 60 3D CT scans, divided into a training set of 40 and a test set of 20 patients, where the OARs have been contoured manually by an experienced radiotherapist. Source: SegTHOR: Segmentation of Thoracic Organs at Risk in CT images Image Source: Lambert et al

관련 논문 28편

SensatUrban

SensatUrban 설명 보기

The SensatUrbat dataset is an urban-scale photogrammetric point cloud dataset with nearly three billion richly annotated points, which is five times the number of labeled points than the existing largest point cloud dataset. The dataset consists of large areas from two UK cities, covering about 6 km^2 of the city landscape. In the dataset, each 3D point is labeled as one of 13 semantic classes, such as ground, vegetation, car, etc.. Source: https://github.com/QingyongHu/SensatUrban Image Source: https://github.com/QingyongHu/SensatUrban

관련 논문 28편

ShareGPT4Video

ShareGPT4Video 설명 보기

The ShareGPT4Video dataset is a large-scale resource designed to improve video understanding and generation¹. It features 1.2 million highly descriptive captions⁴ for video clips, surpassing existing datasets in diversity and information content⁴. The captions cover a wide range of aspects, including world knowledge, object properties, spatial relationships, and aesthetic evaluations⁴. The dataset includes detailed captions of 40K videos generated by GPT-4V¹ and 4.8M videos generated by ShareCaptioner-Video¹. The videos are sourced from YouTube and other user-uploaded video websites, and they cover a variety of scenarios, such as human activities and auto-driving¹. The ShareGPT4Video dataset also provides a basis for the ShareCaptioner-Video, an exceptional video captioner capable of efficiently generating high-quality captions for videos of a wide range of resolution, aspect ratio, and duration¹. For example, the dataset includes a detailed caption of a video documenting a meticulous meal preparation by an individual with tattooed forearms¹. The caption describes the individual's actions in detail, from slicing a cucumber to mixing the dressing and adding croutons to the salad¹. In addition to its use in research, the ShareGPT4Video dataset has been used to train the sharegpt4video-8b model, an open-source video chatbot². This model was trained on open-source video instruction data and is primarily intended for researchers and hobbyists in computer vision, natural language processing, machine learning, and artificial intelligence². (1) arXiv:2406.04325v1 [cs.CV] 6 Jun 2024. https://arxiv.org/pdf/2406.04325. (2) ShareGPT4V: Improving Large Multi-Modal Models with Better Captions. https://arxiv.org/abs/2311.12793. (3) Lin-Chen/sharegpt4video-8b · Hugging Face. https://huggingface.co/Lin-Chen/sharegpt4video-8b. (4) ShareGPT4Video: Improving Video Understanding and Generation with .... https://www.aimodels.fyi/papers/arxiv/sharegpt4video-improving-video-understanding-generation-better-captions. (5) GitHub - ShareGPT4Omni/ShareGPT4Video: An official implementation of .... https://github.com/ShareGPT4Omni/ShareGPT4Video. (6) undefined. https://sharegpt4video.github.io/.

관련 논문 28편

TVSum: Summarizing Web Videos Using Titles

TvSum
TvSum 설명 보기

Introduced by Song et al. in TVSum: Summarizing web videos using titles. The TVSum dataset comprises 50 videos, with durations ranging from 1 to 11 minutes. These videos belong to 10 different categories associated with the TRECVid MED task, with 5 videos in each category, and were collected from YouTube. The video categories include various activities like changing a vehicle tire, making a sandwich, and flash mob gatherings. For annotation, each video was reviewed and rated by 20 users, who assigned frame-level importance scores on a scale from 1 (not important) to 5 (very important).

관련 논문 28편

WNUT-2020 Task 2: Identification of Informative COVID-19 English Tweets

WNUT-2020 Task 2
WNUT-2020 Task 2 설명 보기

Briefly describe the dataset. Provide: * a high-level explanation of the dataset characteristics * explain motivations and summary of its content * potential use cases of the dataset If the description or image is from a different paper, please refer to it as follows: Source: title Image Source: title

관련 논문 28편

Crossmodal 3600

XM 3600
XM 3600 설명 보기

Research in massively multilingual image captioning has been severely hampered by a lack of high-quality evaluation datasets. In this paper we present the Crossmodal-3600 dataset (XM3600 in short), a geographically-diverse set of 3600 images annotated with human-generated reference captions in 36 languages. The images were selected from across the world, covering regions where the 36 languages are spoken, and annotated with captions that achieve consistency in terms of style across all languages, while avoiding annotation artifacts due to direct translation. We apply this benchmark to model selection for massively multilingual image captioning models, and show strong correlation results with human evaluations when using XM3600 as golden references for automatic metrics.

관련 논문 28편

Natural Language Decathlon Benchmark

decaNLP
decaNLP 설명 보기

Natural Language Decathlon Benchmark (decaNLP) is a challenge that spans ten tasks: question answering, machine translation, summarization, natural language inference, sentiment analysis, semantic role labeling, zero-shot relation extraction, goal-oriented dialogue, semantic parsing, and commonsense pronoun resolution. The tasks as cast as question answering over a context. Source: The Natural Language Decathlon: Multitask Learning as Question Answering Image Source: http://decanlp.com/

관련 논문 28편

eSCAPE

eSCAPE 설명 보기

Consists of millions of entries in which the MT element of the training triplets has been obtained by translating the source side of publicly-available parallel corpora, and using the target side as an artificial human post-edit. Translations are obtained both with phrase-based and neural models. Source: eSCAPE: a Large-scale Synthetic Corpus for Automatic Post-Editing

관련 논문 27편

2D-3D Match Dataset

2D-3D Match Dataset 설명 보기

2D-3D Match Dataset is a new dataset of 2D-3D correspondences by leveraging the availability of several 3D datasets from RGB-D scans. Specifically, the data from SceneNN and 3DMatch are used. The training dataset consists of 110 RGB-D scans, of which 56 scenes are from SceneNN and 54 scenes are from 3DMatch. The 2D-3D correspondence data is generated as follows. Given a 3D point which is randomly sampled from a 3D point cloud, a set of 3D patches from different scanning views are extracted. To find a 2D-3D correspondence, for each 3D patch, its 3D position is re-projected into all RGB-D frames for which the point lies in the camera frustum, taking occlusion into account. The corresponding local 2D patches around the re-projected point are extracted. In total, around 1.4 millions 2D-3D correspondences are collected. Source: 2D-3D Match Dataset

관련 논문 27편

Asian Face Age Dataset

AFAD
AFAD 설명 보기

The Asian Face Age Dataset (AFAD) is a new dataset proposed for evaluating the performance of age estimation, which contains more than 160K facial images and the corresponding age and gender labels. This dataset is oriented to age estimation on Asian faces, so all the facial images are for Asian faces. It is noted that the AFAD is the biggest dataset for age estimation to date. It is well suited to evaluate how deep learning methods can be adopted for age estimation.

관련 논문 27편

Android in the Wild

AitW
AitW 설명 보기

Android in the Wild (AitW) is a dataset for device-control research which is orders of magnitude larger than current datasets. The dataset contains human demonstrations of device interactions, including the screens and actions, and corresponding natural language instructions. It consists of 715k episodes spanning 30k unique instructions, four versions of Android (v10–13), and eight device types (Pixel 2 XL to Pixel 6) with varying screen resolutions. It contains multi-step tasks that require semantic understanding of language and visual context.

관련 논문 27편

AndroidWorld

AndroidWorld 설명 보기

AndroidWorld is an environment for building and benchmarking autonomous computer control agents. It runs on a live Android emulator and contains a highly reproducible benchmark of 116 hand-crafted tasks across 20 apps, which are dynamically instantiated with randomly-generated parameters to create millions of unique task variations. In addition to the built-in tasks, AndroidWorld also supports the popular web benchmark, MiniWoB++ from Liu et al.. Key features of AndroidWorld include: 📝 116 diverse tasks across 20 real-world apps 🎲 Dynamic task instantiation for millions of unique variations 🏆 Durable reward signals for reliable evaluation 🌐 Open environment with access to millions of Android apps and websites 💾 Lightweight footprint (2 GB memory, 8 GB disk) 🔧 Extensible design to easily add new tasks and benchmarks 🖥️ Integration with MiniWoB++ web-based tasks

관련 논문 27편

Appliances Energy Prediction

Appliances Energy
Appliances Energy 설명 보기

This dataset details the energy consumption of appliances in a low-energy building over 4.5 months. Data was collected at 10-minute intervals. For more information about the house, data collection, R scripts and figures, please refer to the paper and the following GitHub repository: https://github.com/LuisM78/Appliances-energy-prediction-data

관련 논문 27편

BASIL

BASIL 설명 보기

300 news articles annotated with 1,727 bias spans and find evidence that informational bias appears in news articles more frequently than lexical bias. Source: In Plain Sight: Media Bias Through the Lens of Factual Reporting

관련 논문 27편

COunter NArratives through Nichesourcing

CONAN
CONAN 설명 보기

COunter NArratives through Nichesourcing (CONAN) is a dataset that consists of 4,078 pairs over the 3 languages. Additionally, 3 types of metadata are provided: expert demographics, hate speech sub-topic and counter-narrative type. The dataset is augmented through translation (from Italian/French to English) and paraphrasing, which brought the total number of pairs to 14.988. Source: CONAN Image Source: https://www.aclweb.org/anthology/P19-1271

관련 논문 27편

Case Holdings On Legal Decisions

CaseHOLD
CaseHOLD 설명 보기

CaseHOLD (Case Holdings On Legal Decisions) is a law dataset comprised of over 53,000+ multiple choice questions to identify the relevant holding of a cited case. This dataset presents a fundamental task to lawyers and is both legally meaningful and difficult from an NLP perspective (F1 of 0.4 with a BiLSTM baseline). The citing context from the judicial decision serves as the prompt for the question. The answer choices are holding statements derived from citations following text in a legal decision. There are five answer choices for each citing text. The correct answer is the holding statement that corresponds to the citing text. The four incorrect answers are other holding statements. To read more about the dataset, please see our paper or our blogpost.

관련 논문 27편

ChaosNLI

ChaosNLI 설명 보기

Chaos NLI is a Natural Language Inference (NLI) dataset with 100 annotations per example (for a total of 464,500 annotations) for some existing data points in the development sets of SNLI, MNLI, and Abductive NLI. The dataset provides additional labels for NLI annotations that reflect the distribution of human annotators, instead of picking the majority label as the gold standard label. Source: ChaosNLI Github Repository

관련 논문 27편

ClassEval

ClassEval 설명 보기

In this work, we make the first attempt to evaluate LLMs in a more challenging code generation scenario, i.e. class-level code generation. We first manually construct the first class-level code generation benchmark ClassEval of 100 class-level Python code generation tasks with approximately 500 person-hours. Based on it, we then perform the first study of 11 state-of-the-art LLMs on class-level code generation. Based on our results, we have the following main findings. First, we find that all existing LLMs show much worse performance on class-level code generation compared to on standalone method-level code generation benchmarks like HumanEval; and the method-level coding ability cannot equivalently reflect the class-level coding ability among LLMs. Second, we find that GPT-4 and GPT-3.5 still exhibit dominate superior than other LLMs on class-level code generation, and the second-tier models includes Instruct-Starcoder, Instruct-Codegen, and Wizardcoder with very similar performance. Third, we find that generating the entire class all at once (i.e. holistic generation strategy) is the best generation strategy only for GPT-4 and GPT-3.5, while method-by-method generation (i.e. incremental and compositional) is better strategies for the other models with limited ability of understanding long instructions and utilizing the middle information. Lastly, we find the limited model ability of generating method-dependent code and discuss the frequent error types in generated classes.