Papers with Code 아카이브

Text Classification

태스크의 하위 분류와 논문·코드·평가·기법·데이터셋을 탐색하세요.

하위 태스크
16
평가표
101
보존된 평가 기록
341

Text Classification is the task of assigning a sentence or document an appropriate category. The categories depend on the chosen dataset and can range from topics.

Text Classification problems include emotion classification, news classification, citation intent classification, among others. Benchmark datasets for evaluating text classification capabilities include GLUE, AGNews, among others.

In recent years, deep learning techniques like XLNet and RoBERTa have attained some of the biggest performance jumps for text classification problems.

<span style="color:grey; opacity: 0.6">( Image credit: Text Classification Algorithms: A Survey )</span>

데이터셋

출처에 같은 태스크 이름으로 연결된 데이터셋입니다. 동명 태스크가 포함될 수 있으므로, 비교할 때는 평가 결과의 조건도 확인하세요.

DEFT Corpus

A SemEval shared task in which participants must extract definitions from free text using a term-definition pair corpus that reflects the complex reality of definitions in natural language. Source: SemEval-2020 Task 6: Definition extraction from free text with the DEFT corpus

IndoNLU Benchmark

The IndoNLU benchmark is a collection of resources for training, evaluating, and analyzing natural language understanding systems for Bahasa Indonesia. It is a joint venture from many Indonesia NLP enthusiasts from different institutions such as Gojek, Institut Teknologi Bandung, HKUST, Universitas Multimedia Nusantara, Prosa.ai, and Universitas Indonesia. Source: IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understanding

GCDC

A corpus of real-world texts. Source: Discourse Coherence in the Wild: A Dataset, Evaluation and Methods

Ghostbuster

The Ghostbusters dataset leverages the GPT-3.5-turbo model for generating texts in the domains of creative writing, news, and student essays, providing 2,000 texts in the first two domains and 1,994 in the latter.

HopeEDI

Over the past few years, systems have been developed to control online content and eliminate abusive, offensive or hate speech content. However, people in power sometimes misuse this form of censorship to obstruct the democratic right of freedom of speech. Therefore, it is imperative that research should take a positive reinforcement approach towards online content that is encouraging, positive and supportive contents. Until now, most studies have focused on solving this problem of negativity in the English language, though the problem is much more than just harmful content. Furthermore, it is multilingual as well. Thus, we have constructed a Hope Speech dataset for Equality, Diversity and Inclusion (HopeEDI) containing user-generated comments from the social media platform YouTube with 28,451, 20,198 and 10,705 comments in English, Tamil and Malayalam, respectively, manually labelled as containing hope speech or not. To our knowledge, this is the first research of its kind to annotate hope speech for equality, diversity and inclusion in a multilingual setting. We determined that the inter-annotator agreement of our dataset using Krippendorff’s alpha. Further, we created several baselines to benchmark the resulting dataset and the results have been expressed using precision, recall and F1-score. The dataset is publicly available for the research community. We hope that this resource will spur further research on encouraging inclusive and responsive speech that reinforces positiveness.

OUTFOX

It contains 15K triplets of essay problem statements, student-written, and LLM-generated essays. LLMs include ChatGPT, GPT-3.5, and FLAN-T5-XXL. This dataset was created mainly for LLM-generated text detection but can be useful for other tasks as well.

Story Commonsense

Story Commonsense is a new large-scale dataset with rich low-level annotations and establishes baseline performance on several new tasks, suggesting avenues for future research. Source: Modeling Naive Psychology of Characters in Simple Commonsense Stories

FLUE

FLUE is a French Language Understanding Evaluation benchmark. It consists of 5 tasks: Text Classification, Paraphrasing, Natural Language Inference, Constituency Parsing and Part-of-Speech Tagging, and Word Sense Disambiguation.

Overruling

The Overruling dataset is a law dataset corresponding to the task of determining when a sentence is overruling a prior decision. This is a binary classification task, where positive examples are overruling sentences and negative examples are non-overruling sentences extracted from legal opinions. In law, an overruling sentence is a statement that nullifies a previous case decision as a precedent, by a constitutionally valid statute or a decision by the same or higher ranking court which establishes a different rule on the point of law involved. The Overruling dataset consists of 2,400 sentences. To read more about the dataset, please see our paper or our blogpost.

DEAP

The DEAP dataset consists of two parts: - The ratings from an online self-assessment where 120 one-minute extracts of music videos were each rated by 14-16 volunteers based on arousal, valence and dominance. - The participant ratings, physiological recordings and face video of an experiment where 32 volunteers watched a subset of 40 of the above music videos. EEG and physiological signals were recorded and each participant also rated the videos as above. For 22 participants frontal face video was also recorded.

Fashion 144K

Fashion 144K is a novel heterogeneous dataset with 144,169 user posts containing diverse image, textual and meta information. Source: Neuroaesthetics in Fashion: Modeling the Perception of Fashionability

MAGE

The MAGE dataset provides a large set of generated texts using 27 LLMs from seven different groups: OpenAI GPT, LLaMA, GLM130B, FLAN-T5, OPT, BigScience, and EleutherAI. In total, the dataset contains 432,682 texts, along with two additional sets. The first is an additional test set with texts from unseen domains generated by an unseen model, namely GPT-4. The second set is designed to evaluate the robustness of detectors against paraphrasing attacks. To achieve this, the GPT-3.5-turbo model was employed to paraphrase the sentences from the first set, with all paraphrased texts treated as machine-generated.

MultiEURLEX

MultiEURLEX is a multilingual dataset for topic classification of legal documents. The dataset comprises 65k European Union (EU) laws, officially translated in 23 languages, annotated with multiple labels from the EUROVOC taxonomy. The dataset covers 23 official EU languages from 7 language families.

Ohsumed

Ohsumed includes medical abstracts from the MeSH categories of the year 1991. In [Joachims, 1997] were used the first 20,000 documents divided in 10,000 for training and 10,000 for testing. The specific task was to categorize the 23 cardiovascular diseases categories. After selecting the such category subset, the unique abstract number becomes 13,929 (6,286 for training and 7,643 for testing). As current computers can easily manage larger number of documents we make available all 34,389 cardiovascular diseases abstracts out of 50,216 medical abstracts contained in the year 1991. Source: http://disi.unitn.it/moschitti/corpora.htm

CARD-660

An expert-annotated word similarity dataset which provides a highly reliable, yet challenging, benchmark for rare word representation techniques. Source: Card-660: Cambridge Rare Word Dataset - a Reliable Benchmark for Infrequent Word Representation Models

COVID-19 Twitter Chatter Dataset

A large-scale curated dataset of over 152 million tweets, growing daily, related to COVID-19 chatter generated from January 1st to April 4th at the time of writing. Source: A large-scale COVID-19 Twitter chatter dataset for open scientific research -- an international collaboration

DemogPairs

Although deep face recognition has achieved impressive results in recent years, there is increasing controversy regarding racial and gender bias of the models, questioning their trustworthiness and deployment into sensitive scenarios. DemogPairs is a validation set with 10.8K facial images and 58.3M identity verification pairs, distributed in demographically-balanced folds of Asian, Black and White females and males. We also propose a benchmark of experiments using DemogPairs over state-of-the-art deep face recognition models in order to analyze their cross-demographic behavior and potential demographic biases (see figure below).

unarXive

A scholarly data set with publications’ full-text, annotated in-text citations, and links to metadata. The unarXive data set contains * One million papers in plain text * 63 million citation contexts * 39 million reference strings * A citation network of 16 million connections The data is generated from all LaTeX sources on arXiv from 1991–2020/07 and therefore of higher quality than data generated from PDF files. Furthermore, as all citing papers are available in full text, citation contexts of arbitrary size can be extracted. Typical uses of the data set are approaches in * Citation recommendation * Citation context analysis * Reference string parsing The code for generating the data set is publicly available.

CAD

Dataset of primarily English Reddit entries which addresses several limitations of prior work. It (1) contains six conceptually distinct primary categories as well as secondary categories, (2) has labels annotated in the context of the conversation thread, (3) contains rationales and (4) uses an expert-driven group-adjudication process for high quality annotations.

LUN

LUN is used for unreliable news source classification, this dataset includes 17,250 articles from satire, propaganda, and hoaxe.

JGLUE

JGLUE, Japanese General Language Understanding Evaluation, is built to measure the general NLU ability in Japanese.

OpoSum

OPOSUM is a dataset for the training and evaluation of Opinion Summarization models which contains Amazon reviews from six product domains: Laptop Bags, Bluetooth Headsets, Boots, Keyboards, Televisions, and Vacuums. The six training collections were created by downsampling from the Amazon Product Dataset introduced in McAuley et al. (2015) and contain reviews and their respective ratings. A subset of the dataset has been manually annotated, specifically, for each domain, 10 different products were uniformly sampled (across ratings) with 10 reviews each, amounting to a total of 600 reviews, to be used only for development (300) and testing (300). Source: Summarizing Opinions: Aspect Extraction Meets Sentiment Prediction and They Are Both Weakly Supervised

TUNIZI

A sentiment analysis Tunisian Arabizi Dataset, collected from social networks, preprocessed for analytical studies and annotated manually by Tunisian native speakers. Source: TUNIZI: a Tunisian Arabizi sentiment analysis Dataset

UIT-ViCTSD

UIT-ViCTSD (Vietnamese Constructive and Toxic Speech Detection) is a dataset for constructive and toxic speech detection in Vietnamese. It consists of 10,000 human-annotated comments.

WikiDetox

An annotated dataset of 1m crowd-sourced annotations that cover 100k talk page diffs (with 10 judgements per diff) for personal attacks, aggression, and toxicity.

ArmanEmo

ArmanEmo is a human-labeled emotion dataset of more than 7000 Persian sentences labeled for seven categories. The dataset has been collected from different resources, including Twitter, Instagram, and Digikala (an Iranian e-commerce company) comments. Labels are based on Ekman's six basic emotions (Anger, Fear, Happiness, Hatred, Sadness, Wonder) and another category (Other) to consider any other emotion not included in Ekman's model. Source: ArmanEmo: A Persian Dataset for Text-based Emotion Detection Image Source: https://arxiv.org/pdf/2207.11808v1.pdf

arXiv-10

Benchmark dataset for abstracts and titles of 100,000 ArXiv scientific papers. This dataset contains 10 classes and is balanced (exactly 10,000 per class). The classes include subcategories of computer science, physics, and math. • Direct link: Download • Citation: @inproceedings{farhangi2022protoformer, title={Protoformer: Embedding Prototypes for Transformers}, author={Farhangi, Ashkan and Sui, Ning and Hua, Nan and Bai, Haiyan and Huang, Arthur and Guo, Zhishan}, booktitle={Advances in Knowledge Discovery and Data Mining: 26th Pacific-Asia Conference, PAKDD 2022, Chengdu, China, May 16--19, 2022, Proceedings, Part I}, pages={447--458}, year={2022} }

Bengali Hate Speech

Introduces three datasets of expressing hate, commonly used topics, and opinions for hate speech detection, document classification, and sentiment analysis, respectively. Source: Classification Benchmarks for Under-resourced Bengali Language based on Multichannel Convolutional-LSTM Network

OMICS

OMICS is an extensive collection of knowledge for indoor service robots gathered from internet users. Currently, it contains 48 tables capturing different sorts of knowledge. Each tuple of the Help table maps a user desire to a task that may meet the desire (e.g., ⟨ “feel thirsty”, “by offering drink” ⟩). Each tuple of the Tasks/Steps table decomposes a task into several steps (e.g., ⟨ “serve a drink”, 0. “get a glass”, 1. “get a bottle”, 2. “fill class from bottle”, 3. “give class to person” ⟩). Given this, OMICS offers useful knowledge about hierarchism of naturalistic instructions, where a high-level user request (e.g., “serve a drink”) can be reduced to lower-level tasks (e.g., “get a glass”, ⋯). Another feature of OMICS is that elements of any tuple in an OMICS table are semantically related according to a predefined template. This facilitates the semantic interpretation of the OMICS tuples. Source: Understanding User Instructions by Utilizing Open Knowledge for Service Robots Image Source: https://www.aaai.org/Papers/AAAI/2004/AAAI04-096.pdf

TREC-10

A question type classification dataset with 6 classes for questions about a person, location, numeric information, etc. The test split has 500 questions, and the training split has 5452 questions. Paper: Learning Question Classifiers

CC3M-TagMask

The dataset offers tag and mask annotations for image-text pairs from the CC3M validation set. Tag annotations denote words that aptly describe the relationship between the image and the corresponding text. These annotations provide valuable insights into the semantic connection between each pair's visual and textual elements.

Illness-dataset

A dataset for evaluating text classification, domain adaptation, and active learning models. The dataset consists of 22,660 documents (tweets) collected in 2018 and 2019. It spans across four domains: Alzheimer's, Parkinson's, Cancer, and Diabetes.

MuMiN

MuMiN is a misinformation graph dataset containing rich social media data (tweets, replies, users, images, articles, hashtags), spanning 21 million tweets belonging to 26 thousand Twitter threads, each of which have been semantically linked to 13 thousand fact-checked claims across dozens of topics, events and domains, in 41 different languages, spanning more than a decade. MuMiN fills a gap in the existing misinformation datasets in multiple ways: - By having a large amount of social media information which have been semantically linked to fact-checked claims on an individual basis. - By featuring 41 languages, enabling evaluation of multilingual misinformation detection models. - By featuring both tweets, articles, images, social connections and hashtags, enabling multimodal approaches to misinformation detection. MuMiN features two node classification tasks, related to the veracity of a claim: - Claim classification: Determine the veracity of a claim, given its social network context. - Tweet classification: Determine the likelihood that a social media post to be fact-checked is discussing a misleading claim, given its social network context. To use the dataset, see the "Getting Started" guide and tutorial at the MuMiN website.

Taiga Corpus

Taiga is a corpus, where text sources and their meta-information are collected according to popular ML tasks. Each text in corpus is represented in plain text and with morphological and syntactic annotation (UDPipe, homonymy resolved automatically) + has metainformation - date, theme, authorship, text difficulcy…etc (depending on source) By now, about 5 billions of words are 77% literary texts (33 literary magazines), 19% of naive poetry, 2% of news (4 popular sites) and 2% of other (popular science, culture mags, social networks, amateur poems and prose), with documentation available. Segments Info | Genres | Tokens, millions | % | |------------------|------------------|-----| | News | 92 | 1.5 | | Literary Texts | 4605 | 76 | | Special datasets | 2.5 | 0.5 | | Social media | 80 | 1.5 | | Subtitles | 101 | 1.5 | | Poems | 1130 | 19 | **Annotation Example ** (CONLL-u): # newdoc # newpar # sent_id = 1 # 2003Armeniya.xml 1 # text = В советский период времени число ИТ- специалистов в Армении составляло около десяти тысяч. # sent_id = 1 1 В в ADP _ _ 3 case 3:case _ 2 советский советский ADJ _ Animacy=Inan|Case=Acc|Degree=Pos|Gender=Masc|Number=Sing 3 amod 3:amod _ 3 период период NOUN _ Animacy=Inan|Case=Acc|Gender=Masc|Number=Sing 11 obl 11:obl _ 4 времени время NOUN _ Animacy=Inan|Case=Gen|Gender=Neut|Number=Sing 3 nmod 3:nmod _ 5 число число NOUN _ Animacy=Inan|Case=Acc|Gender=Neut|Number=Sing 11 obj 11:obj _ 6 ИТ ит PROPN _ Animacy=Inan|Case=Nom|Gender=Neut|Number=Sing 8 compound 8:compound SpaceAfter=No 7 - - PUNCT _ _ 6 punct 6:punct _ 8 специалистов специалист NOUN _ Animacy=Anim|Case=Gen|Gender=Masc|Number=Plur 5 nmod 5:nmod _ 9 в в ADP _ _ 10 case 10:case _ 10 Армении армения PROPN _ Animacy=Inan|Case=Loc|Gender=Fem|Number=Sing 5 nmod 5:nmod _ 11 составляло составлять VERB _ Aspect=Imp|Gender=Neut|Mood=Ind|Number=Sing|Tense=Past|VerbForm=Fin|Voice=Act 0 root 0:root _ 12 около около ADP _ _ 14 case 14:case _ 13 десяти десять NUM _ Case=Gen 14 nummod 14:nummod _ 14 тысяч тысяча NOUN _ Animacy=Inan|Case=Gen|Gender=Fem|Number=Plur 11 nsubj 11:nsubj SpaceAfter=No 15 . . PUNCT _ _ 14 punct 14:punct _

Climabench

The topic of Climate Change (CC) has received limited attention in NLP despite its real world urgency. Activists and policy-makers need NLP tools in order to effectively process the vast and rapidly growing textual data produced on CC. Their utility, however, primarily depends on whether the current state-of-the-art models can generalize across various tasks in the CC domain. In order to address this gap, we introduce Climate Change Benchmark (ClimaBench), a benchmark collection of existing disparate datasets for evaluating model performance across a diverse set of CC NLU tasks systematically. Further, we enhance the benchmark by releasing two large-scale labelled text classification and question-answering datasets curated from publicly available environmental disclosures. Lastly, we provide an analysis of several generic and CC-oriented models answering whether fine-tuning on domain text offers any improvements across these tasks. We hope this work provides a standard assessment tool for research on CC text data.

출처: Papers with Code · CC BY-SA 4.0. 아카이브 자료와 새로 수집한 논문은 별도로 관리합니다.