Papers with Code · CC BY-SA 4.0

데이터셋 탐색

데이터셋의 설명·홈페이지를 찾고, 관련 태스크의 평가 기록을 살펴보세요.

관련 논문 17편

Multiview Extended Video with Activities

MEVA
MEVA 설명 보기

Large-scale dataset for human activity recognition. Existing security datasets either focus on activity counts by aggregating public video disseminated due to its content, which typically excludes same-scene background video, or they achieve persistence by observing public areas and thus cannot control for activity content. The dataset is over 9300 hours of untrimmed, continuous video, scripted to include diverse, simultaneous activities, along with spontaneous background activity. Source: MEVA: A Large-Scale Multiview, Multimodal Video Dataset for Activity Detection

관련 논문 141편

SummEval

SummEval 설명 보기

The SummEval dataset is a resource developed by the Yale LILY Lab and Salesforce Research for evaluating text summarization models. It was created as part of a project to address shortcomings in summarization evaluation methods. The dataset includes summaries generated by various recent summarization models trained on the CNN/DailyMail dataset. It also contains human annotations, collected from both crowdsource workers and experts. However, the source articles used to generate the summaries are not included. The SummEval project also provides a toolkit for summarization evaluation. This toolkit unifies metrics and promotes robust comparison of summarization systems. It contains popular and recent metrics for summarization as well as several machine translation metrics. The goal of the SummEval project is to promote a more complete evaluation protocol for text summarization and advance research in developing evaluation metrics that better correlate with human judgments.

관련 논문 107편

SemEval-2014 Task-4

SemEval-2014 Task-4 설명 보기

Sentiment analysis is increasingly viewed as a vital task both from an academic and a commercial standpoint. The majority of current approaches, however, attempt to detect the overall polarity of a sentence, paragraph, or text span, regardless of the entities mentioned (e.g., laptops, restaurants) and their aspects (e.g., battery, screen; food, service). By contrast, this task is concerned with aspect based sentiment analysis (ABSA), where the goal is to identify the aspects of given target entities and the sentiment expressed towards each aspect. Datasets consisting of customer reviews with human-authored annotations identifying the mentioned aspects of the target entities and the sentiment polarity of each aspect will be provided. Subtask 2: Aspect term polarity For a given set of aspect terms within a sentence, determine whether the polarity of each aspect term is positive, negative, neutral or conflict (i.e., both positive and negative). For example: “I loved their fajitas” → {fajitas: positive} “I hated their fajitas, but their salads were great” → {fajitas: negative, salads: positive} “The fajitas are their first plate” → {fajitas: neutral} “The fajitas were great to taste, but not to see” → {fajitas: conflict}

관련 논문 67편

SemEval 2019 Task 5: Multilingual Detection of Hate Speech Against Immigrants and Women in Twitter

HatEval
HatEval 설명 보기

Hate Speech is commonly defined as any communication that disparages a person or a group on the basis of some characteristic such as race, color, ethnicity, gender, sexual orientation, nationality, religion, or other characteristics. Given the huge amount of user-generated contents on the Web, and in particular on social media, the problem of detecting, and therefore possibly limit the Hate Speech diffusion, is becoming fundamental, for instance for fighting against misogyny and xenophobia. The proposed task consists in Hate Speech detection in Twitter but featured by two specific different targets, immigrants and women, in a multilingual perspective, for Spanish and English. The task will be articulated around two related subtasks for each of the involved languages: a basic task about Hate Speech, and another one where fine-grained features of hateful contents will be investigated in order to understand how existing approaches may deal with the identification of especially dangerous forms of hate, i.e. those where the incitement is against an individual rather than against a group of people, and where an aggressive behavior of the author can be identified as a prominent feature of the expression of hate. Participants will be asked to identify, on the one hand, if the target of hate is a single human or a group of persons, on the other hand, if the message author intends to be aggressive, harmful, or even to incite, in various forms, to violent acts against the target. - TASK A - Hate Speech Detection against Immigrants and Women: a two-class (or binary) classification where systems have to predict whether a tweet in English or in Spanish with a given target (women or immigrants) is hateful or not hateful. - TASK B - Aggressive behavior and Target Classification: where systems are asked first to classify hateful tweets for English and Spanish (e.g., tweets where Hate Speech against women or immigrants has been identified) as aggressive or not aggressive, and second to identify the target harassed as individual or generic (i.e. single human or group).

관련 논문 37편

SemEval-2018 Task-9

SemEval-2018 Task-9 설명 보기

The SemEval-2018 hypernym discovery evaluation benchmark (Camacho-Collados et al. 2018) contains three domains (general, medical and music) and is also available in Italian and Spanish (not in this repository). For each domain a target corpus and vocabulary (i.e. hypernym search space) are provided. The dataset contains both concepts (e.g. dog) and entities (e.g. Manchester United) up to trigrams. Source: NLP Progress Image source: SemEval-2018 Task 9: Hypernym Discovery

관련 논문 33편

SemEval-2017 Task-10

SemEval-2017 Task-10 설명 보기

We describe the SemEval task of extracting keyphrases and relations between them from scientific documents, which is crucial for understanding which publications describe which processes, tasks and materials. Although this was a new task, we had a total of 26 submissions across 3 evaluation scenarios. We expect the task and the findings reported in this paper to be relevant for researchers working on understanding scientific content, as well as the broader knowledge base population and information extraction communities.

관련 논문 31편

SemEval-2013 Task-2

SemEval-2013 Task-2 설명 보기

The SemEval-2013 Task 2 dataset contains data for two subtasks: A, an expression-level subtask, and B, a message-level subtask. Crowdsourcing was used to label a large Twitter training dataset along with additional test sets of Twitter and SMS messages for both subtasks.

관련 논문 22편

iSarcasmEval

iSarcasmEval 설명 보기

iSarcasmEval is the first shared task to target intended sarcasm detection: the data for this task was provided and labelled by the authors of the texts themselves. Such an approach minimises the downfalls of other methods to collect sarcasm data, which rely on distant supervision or third-party annotations. The shared task contains two languages, English and Arabic, and three subtasks: sarcasm detection, sarcasm category classification, and pairwise sarcasm identification given a sarcastic sentence and its non-sarcastic rephrase. The task received submissions from 60 different teams, with the sarcasm detection task being the most popular. Most of the participating teams utilised pre-trained language models. In this paper, we provide an overview of the task, data, and participating teams.

관련 논문 11편

SemEval-2021 Task 4: Reading Comprehension of Abstract Meaning

ReCAM
ReCAM 설명 보기

Tasks Our shared task has three subtasks. Subtask 1 and 2 focus on evaluating machine learning models' performance with regard to two definitions of abstractness (Spreen and Schulz, 1966; Changizi, 2008), which we call imperceptibility and nonspecificity, respectively. Subtask 3 aims to provide some insights to their relationships. • Subtask 1: ReCAM-Imperceptibility Concrete words refer to things, events, and properties that we can perceive directly with our senses (Spreen and Schulz, 1966; Coltheart 1981; Turney et al., 2011), e.g., donut, trees, and red. In contrast, abstract words refer to ideas and concepts that are distant from immediate perception. Examples include objective, culture, and economy. In subtask 1, the participanting systems are required to perform reading comprehension of abstract meaning for imperceptible concepts. Below is an example. Given a passage and a question, your model needs to choose from the five candidates the best one for replacing @placeholder. • Subtask 2: ReCAM-Nonspecificity Subtask 2 focuses on a different type of definition. Compared to concrete concepts like groundhog and whale, hypernyms such as vertebrate are regarded as more abstract (Changizi, 2008). • Subtask 3: ReCAM-Intersection Subtask 3 aims to provide more insights to the relationship of the two views on abstractness, In this subtask, we test the performance of a system that is trained on one definition and evaluted on the other.

관련 논문 10편

OpenMEVA

OpenMEVA 설명 보기

OpenMEVA is a benchmark for evaluating open-ended story generation metrics. OpenMEVA provides a comprehensive test suite to assess the capabilities of metrics, including (a) the correlation with human judgments, (b) the generalization to different model outputs and datasets, (c) the ability to judge story coherence, and (d) the robustness to perturbations. To this end, OpenMEVA includes both manually annotated stories and auto-constructed test examples. Source: OpenMEVA: A Benchmark for Evaluating Open-ended Story Generation Metrics Image source: OpenMEVA: A Benchmark for Evaluating Open-ended Story Generation Metrics

관련 논문 8편

SemEval-2021 Task-11

SemEval-2021 Task-11 설명 보기

NLPContributionGraph was introduced as Task 11 at SemEval 2021 for the first time. The task is defined on a dataset of Natural Language Processing (NLP) scholarly articles with their contributions structured to be integrable within Knowledge Graph infrastructures such as the Open Research Knowledge Graph. The structured contribution annotations are provided as (1) Contribution sentences : a set of sentences about the contribution in the article; (2) Scientific terms and relations: a set of scientific terms and relational cue phrases extracted from the contribution sentences; and (3) Triples: semantic statements that pair scientific terms with a relation, modeled toward subject-predicate-object RDF statements for KG building. The Triples are organized under three (mandatory) or more of twelve total information units (viz., ResearchProblem, Approach, Model, Code, Dataset, ExperimentalSetup, Hyperparameters, Baselines, Results, Tasks, Experiments, and AblationAnalysis).

관련 논문 6편

SemEval-2014 Task-10

SemEval-2014 Task-10 설명 보기

SemEval 2014 is a collection of datasets used for the Semantic Evaluation (SemEval) workshop, an annual event that focuses on the evaluation and comparison of systems that can analyze diverse semantic phenomena in text. The datasets from SemEval 2014 are used for various tasks, including but not limited to: - Aspect-Based Sentiment Analysis (ABSA): This task is based on laptop and restaurant reviews. It involves identifying the aspects or features mentioned in a review and determining the sentiment expressed towards each aspect. - Text Classification: This task involves classifying text into predefined categories. Sub-tasks include text-scoring, natural language inference, and semantic-similarity-scoring.

관련 논문 4편

SemEval-2022 Task-12

SemEval-2022 Task-12 설명 보기

Symlink is a SemEval shared task of extracting mathematical symbols and their descriptions from LaTeX source of scientific documents. This is a new task in SemEval 2022, which attracted 180 individual registrations and 59 final submissions from 7 participant teams.

관련 논문 3편

DialSummEval

DialSummEval 설명 보기

The DialSummEval is a multi-faceted dataset of human judgments. It was created to revisit the evaluation of dialogue summarization models. The dataset contains the outputs of 14 models on SAMSum, a dialogue summary dataset. The creators of DialSummEval observed that current dialogue summarization models have flaws that may not be well exposed by frequently used metrics such as ROUGE. Therefore, they re-evaluated 18 categories of metrics in terms of four dimensions: coherence, consistency, fluency, and relevance. They also conducted a unified human evaluation of various models in dialogue summarization for the first time.

관련 논문 2편

GermEval

GermEval 설명 보기

The GermEval dataset is a valuable resource for natural language processing (NLP) tasks, specifically named entity recognition (NER), conducted in the German language. Here are some key details about this dataset: - Task: Token Classification (specifically, named entity recognition) - Language: German - Size: The dataset falls within the category of 100K < n < 1M tokens. - Source: The data was sampled from German Wikipedia and News Corpora, comprising a collection of citations. - Annotations: The annotations were created through crowdsourcing efforts. - License: The dataset is available under the cc-by-4.0 license. - Content: It covers over 31,000 sentences, corresponding to more than 590,000 tokens. - Purpose: Researchers and practitioners can use this dataset to train and evaluate NER models for German text. You can find more information and explore the dataset on the Hugging Face Datasets page ¹. (1) germeval_14 · Datasets at Hugging Face. https://huggingface.co/datasets/germeval_14. (2) GermEval-2018 Corpus (DE) - Empirical Linguistics and ... - heiDATA. https://heidata.uni-heidelberg.de/dataset.xhtml?persistentId=doi:10.11588/data/0B5VML. (3) GermEval 2014 Named Entity Recognition Shared Task - Data and Task Setup. https://sites.google.com/site/germeval2014ner/data. (4) 6 Best German Language Datasets of 2022 | Twine - Twine Blog. https://www.twine.net/blog/best-german-language-datasets/. (5) germeval_14 | TensorFlow Datasets. https://www.tensorflow.org/datasets/community_catalog/huggingface/germeval_14. (6) undefined. http://www.stern.de/sport/fussball/krawalle-in-der-fussball-bundesliga-dfb-setzt-auf-falsche-konzepte-1553657.html. (7) undefined. http://www.fr-online.de/in_und_ausland/sport/aktuell/1618625_Frings-schaut-finster-in-die-Zukunft.html.

관련 논문 1편

CL-RuTerm3

RuTermEval (Track 1)
RuTermEval (Track 1) 설명 보기

CL-RuTerm3 dataset is a novel resource featuring nested term annotations across six domains (the main one is computational linguistics, also mathematics, medicine, economics, literature studies, and agrochemistry), and the RuTermEval-2024 competition, designed to evaluate term extraction systems on this data. The CL-RuTerm3 dataset, comprising 1270 abstracts and 15 full-text articles (over 165k tokens with over 37k annotated entities), is the largest of its kind for Russian scientific texts. Terms are classified into three categories based on lexical and domain specificity: specific terms, common terms, and nomens. The dataset’s unique features, such as nested term markup and cross-domain coverage, enable more realistic evaluation of ATE systems. First track is devoted to Nested term extraction (in sequence labeling format) (also called Nested Term Identification).

관련 논문 1편

CL-RuTerm3

RuTermEval (Track 2)
RuTermEval (Track 2) 설명 보기

CL-RuTerm3 dataset is a novel resource featuring nested term annotations across six domains (the main one is computational linguistics, also mathematics, medicine, economics, literature studies, and agrochemistry), and the RuTermEval-2024 competition, designed to evaluate term extraction systems on this data. The CL-RuTerm3 dataset, comprising 1270 abstracts and 15 full-text articles (over 165k tokens with over 37k annotated entities), is the largest of its kind for Russian scientific texts. Terms are classified into three categories based on lexical and domain specificity: specific terms, common terms, and nomens. The dataset’s unique features, such as nested term markup and cross-domain coverage, enable more realistic evaluation of ATE systems. Second track is devoted to Nested term extraction (in sequence labeling format) and classification (labels are specific, common, nomen).

관련 논문 1편

CL-RuTerm3

RuTermEval (Track 3)
RuTermEval (Track 3) 설명 보기

CL-RuTerm3 dataset is a novel resource featuring nested term annotations across six domains (the main one is computational linguistics, also mathematics, medicine, economics, literature studies, and agrochemistry), and the RuTermEval-2024 competition, designed to evaluate term extraction systems on this data. The CL-RuTerm3 dataset, comprising 1270 abstracts and 15 full-text articles (over 165k tokens with over 37k annotated entities), is the largest of its kind for Russian scientific texts. Terms are classified into three categories based on lexical and domain specificity: specific terms, common terms, and nomens. The dataset’s unique features, such as nested term markup and cross-domain coverage, enable more realistic evaluation of ATE systems. Third track is devoted to Nested term extraction (in sequence labeling format) and classification (labels are specific, common, nomen) in cross-domain task.

관련 논문 1편

SemEval-2016 Task-6

SemEval-2016 Task-6 설명 보기

SemEval-2016 Task 6, titled "Stance Detection in Tweets," provides a specialized dataset for the computational linguistics and natural language processing (NLP) communities to explore and analyze users' positions towards certain targets, based solely on the content of their tweets. Stance detection aims to determine whether the author of a piece of text is in favor of, against, or neutral towards a specified target, such as a political figure, policy, or product. ### Dataset Characteristics - Content: The dataset consists of tweets, each labeled with the stance of the tweet towards a target. The targets are predefined topics or entities relevant to public debate or interest at the time of the dataset's creation. - Labels: Each tweet is annotated with one of three possible stance labels: FAVOR, AGAINST, or NEUTRAL. These labels represent the author's stance towards the target mentioned in the tweet. - Targets: A limited set of targets is included, each representing a contentious topic of discussion. Examples of targets might include political figures, legislation, or social issues. - Volume: The dataset includes thousands of tweets, providing a substantial volume for training and evaluating machine learning models. - Language: The primary language of the dataset is English, encompassing a wide range of linguistic expressions, idioms, and slang, reflective of the Twitter platform's diverse user base. ### Motivations and Summary The primary motivation behind the creation of this dataset was to foster advancements in stance detection, an area of sentiment analysis that goes beyond simple positive, negative, or neutral sentiment classification. Stance detection requires understanding not just the sentiment but the specific position taken towards a target, which can be challenging due to the subtlety and complexity of language used in social media. The dataset serves as a benchmark for developing and evaluating NLP models capable of automatically detecting stance in text, an essential task for applications such as opinion mining, political polling, and market research. By providing real-world, annotated data, the dataset allows for the exploration of how language is used to express support, opposition, or neutrality towards specific entities or topics. ### Potential Use Cases - Opinion Mining and Sentiment Analysis: Researchers and practitioners can use the dataset to improve algorithms that automatically identify public opinion trends on various issues, enabling more nuanced analysis than simple positive/negative sentiment classification. - Social Media Monitoring: Companies and organizations can leverage stance detection models trained on this dataset to monitor social media discourse around their products, policies, or brand to better understand public perception. - Political Analysis: Analysts interested in the public opinion dynamics of election campaigns or policy debates can use stance detection to gauge support or opposition levels among social media users. - Content Filtering and Recommendation Systems: By understanding the stance of users towards different topics, platforms can tailor content recommendations more effectively, promoting content that aligns with users' views or introducing them to counterpoints to encourage diverse exposure. Overall, the SemEval-2016 Task 6 dataset provides valuable resources for advancing the field of stance detection, with broad applications in analyzing and understanding public opinions expressed on social media platforms.

관련 논문 0편

Paraphrase and Semantic Similarity in Twitter

SemEval-2015 Task-1
SemEval-2015 Task-1 설명 보기

Given two sentences, the participants are asked to determine whether they express the same or very similar meaning and optionally a degree score between 0 and 1. Following the literature on paraphrase identification, we evaluate system performance primarily by the F-1 score and Accuracy against human judgments. We also provide additional evaluations by Pearson correlation and PINC (Chen and Dolan, 2011), which measure lexical dissimilarity between sentence pairs.

관련 논문 0편

Shared Task on Hierarchical Classification of Blurbs - GermEval 2019

Shared Task on Hierarchical Classification of Blurbs - GermEval 2019 설명 보기

This dataset can be used as a benchmark for clustering word embeddings for German. It contains 18'084 unique samples, 28 splits with 177 to 16'425 samples, and 4 to 93 unique classes.