Papers with Code 아카이브

Text Classification

태스크의 하위 분류와 논문·코드·평가·기법·데이터셋을 탐색하세요.

하위 태스크
16
평가표
101
보존된 평가 기록
341

Text Classification is the task of assigning a sentence or document an appropriate category. The categories depend on the chosen dataset and can range from topics.

Text Classification problems include emotion classification, news classification, citation intent classification, among others. Benchmark datasets for evaluating text classification capabilities include GLUE, AGNews, among others.

In recent years, deep learning techniques like XLNet and RoBERTa have attained some of the biggest performance jumps for text classification problems.

<span style="color:grey; opacity: 0.6">( Image credit: Text Classification Algorithms: A Survey )</span>

데이터셋

출처에 같은 태스크 이름으로 연결된 데이터셋입니다. 동명 태스크가 포함될 수 있으므로, 비교할 때는 평가 결과의 조건도 확인하세요.

CSPubSum

CSPubSum is a dataset for summarisation of computer science publications, created by exploiting a large resource of author provided summaries and show straightforward ways of extending it further. Source: A Supervised Approach to Extractive Summarisation of Scientific Papers

Jigsaw Toxic Comment Classification Dataset

You are provided with a large number of Wikipedia comments which have been labeled by human raters for toxic behavior. The types of toxicity are: toxic severe_toxic obscene threat insult identity_hate You must create a model which predicts a probability of each type of toxicity for each comment. File descriptions train.csv - the training set, contains comments with their binary labels test.csv - the test set, you must predict the toxicity probabilities for these comments. To deter hand labeling, the test set contains some comments which are not included in scoring. sample_submission.csv - a sample submission file in the correct format test_labels.csv - labels for the test data; value of -1 indicates it was not used for scoring; (Note: file added after competition close!) Usage The dataset under CC0, with the underlying comment text being governed by Wikipedia's CC-SA-3.0

LIAR2

The LIAR dataset has been widely followed by fake news detection researchers since its release, and along with a great deal of research, the community has provided a variety of feedback on the dataset to improve it. We adopted these feedbacks and released the LIAR2 dataset, a new benchmark dataset of ~23k manually labeled by professional fact-checkers for fake news detection tasks. We have used a split ratio of 8:1:1 to distinguish between the training set, the test set, and the validation set, details of which are provided in the paper of "An Enhanced Fake News Detection System With Fuzzy Deep Learning". The LIAR2 dataset can be accessed at Huggingface and Github, and statistical information for LIAR and LIAR2 is provided in the table below: | Statistics | LIAR | LIAR2 | |----------------------------------------------|----------|------------| | Training set size | 10,269 | 18,369 | | Validation set size | 1,284 | 2,297 | | Testing set size | 1,283 | 2,296 | | Avg. statement length (tokens) | 17.9 | 17.7 | | Avg. speaker description length (tokens) | \ | 39.4 | | Avg. justification length (tokens) | \ | 94.4 | | Labels | Pants on fire | 1,050 | 3,031 | | False | 2,511 | 6,605 | | Barely-true | 2,108 | 3,603 | | Half-true | 2,638 | 3,709 | | Mostly-true | 2,466 | 3,429 | | True | 2,063 | 2,585 | ## Ablation Experiment The LIAR2 dataset is an upgrade of the LIAR dataset, which inherits the ideas of the LIAR dataset, refines the details and architecture, and expands the size of the dataset to make it more responsive to the needs of fake news detection tasks. We believe that with the help of the LIAR2 dataset, it will be able to perform better fake news detection tasks. The analysis and baseline information about the LIAR2 dataset is provided in below. | Feature | Val. Accuracy | Val. F1-Macro | Val. F1-Micro | Test Accuracy | Test F1-Macro | Test F1-Micro | Mean | |--------------------------|-------------------|-------------------|-------------------|-------------------|-------------------|-------------------|----------| | Statement | 0.3174 | 0.1957 | 0.3117 | 0.3197 | 0.2380 | 0.3197 | 0.2837 | | Date | 0.2912 | 0.1879 | 0.2912 | 0.3079 | 0.1775 | 0.3079 | 0.2606 | | Subject | 0.3243 | 0.2311 | 0.3183 | 0.3267 | 0.2271 | 0.3267 | 0.2924 | | Speaker | 0.3283 | 0.2250 | 0.3174 | 0.3310 | 0.2462 | 0.3310 | 0.2965 | | Speaker Description | 0.3322 | 0.2444 | 0.3250 | 0.3280 | 0.2444 | 0.3280 | 0.3003 | | State Info | 0.2930 | 0.1577 | 0.2950 | 0.2979 | 0.1521 | 0.2979 | 0.2489 | | Credibility History | 0.5007 | 0.4696 | 0.4985 | 0.5057 | 0.4656 | 0.5057 | 0.4910 | | Context | 0.2982 | 0.1817 | 0.2982 | 0.3132 | 0.1791 | 0.3132 | 0.2639 | | Justification | 0.5964 | 0.5657 | 0.5827 | 0.6115 | 0.5968 | 0.6115 | 0.5941 | | All without |Statement | 0.7079 | 0.6734 | 0.6822 | 0.7182 | 0.7108 | 0.7182 | 0.7018 | | Date | 0.6931 | 0.6572 | 0.6680 | 0.7078 | 0.6993 | 0.7078 | 0.6889 | | Subject | 0.7000 | 0.6579 | 0.6681 | 0.7078 | 0.7013 | 0.7078 | 0.6905 | | Speaker | 0.6944 | 0.6648 | 0.6757 | 0.7043 | 0.6942 | 0.7043 | 0.6896 | | Speaker Description | 0.6892 | 0.6640 | 0.6739 | 0.7169 | 0.7073 | 0.7169 | 0.6947 | | State Info | 0.7074 | 0.6625 | 0.6729 | 0.7099 | 0.7016 | 0.7099 | 0.6940 | | Credibility History | 0.6025 | 0.5717 | 0.5900 | 0.6185 | 0.6046 | 0.6185 | 0.6010 | | Context | 0.7005 | 0.6622 | 0.6720 | 0.7043 | 0.6967 | 0.7043 | 0.6900 | | Justification | 0.5285 | 0.4898 | 0.5153 | 0.5340 | 0.5148 | 0.5340 | 0.5194 | | Statement + | Date | 0.3431 | 0.2540 | 0.3343 | 0.3380 | 0.2514 | 0.3380 | 0.3098 | | Subject | 0.3548 | 0.2759 | 0.3513 | 0.3375 | 0.2580 | 0.3375 | 0.3192 | | Speaker | 0.3618 | 0.2862 | 0.3539 | 0.3476 | 0.2640 | 0.3476 | 0.3269 | | Speaker Description | 0.3583 | 0.2814 | 0.3531 | 0.3667 | 0.2886 | 0.3667 | 0.3358 | | State Info | 0.3317 | 0.2367 | 0.3294 | 0.3328 | 0.2362 | 0.3328 | 0.2999 | | Credibility History | 0.5067 | 0.4737 | 0.5084 | 0.5244 | 0.5000 | 0.5244 | 0.5063 | | Context | 0.3361 | 0.2682 | 0.3391 | 0.3458 | 0.2560 | 0.3458 | 0.3152 | | Justification | 0.6017 | 0.5578 | 0.5796 | 0.6176 | 0.6026 | 0.6176 | 0.5962 | | All | 0.6974 | 0.6570 | 0.6676 | 0.7021 | 0.6961 | 0.7021 | 0.6871|

Medical Abstracts

The Medical Abstracts dataset contains 14,438 medical abstracts describing 5 different classes of patient conditions, with all of the dataset being annotated. The dataset is split into training and test sets.

MetaHate

MetaHate: A Dataset for Unifying Efforts on Hate Speech Detection This is MetaHate: a meta-collection of 36 hate speech datasets from social media comments. ## Dataset Structure The dataset contains 1,226,202 social media posts in a TSV file. Each element contains the following fields: | Field Name | Type | Possible Values | Description | |------------|------|-----------------|----------------------------------------------------------------------| | text | str | any | Social media post. Each post is unique. | | label | int | 0, 1 | Label of the post. 0 for non-hate speech posts, 1 for hate speech. |

MixSet

MIXSET comprises a total of 3.6k mixtext instances. The dataset features a blend of HWT(human-written text) and MGT(machine-generated text).

MN-DS

Multilabeled News Dataset (MN-DS) is a dataset for news classification. It consists of 10,917 articles in 17 first-level and 109 second-level categories from 215 media sources.

SV-Ident

SV-Ident comprises 4,248 sentences from social science publications in English and German. The data is the official data for the Shared Task: “Survey Variable Identification in Social Science Publications” (SV-Ident) 2022. Sentences are labeled with variables that are mentioned either explicitly or implicitly. The dataset supports the following tasks: - Variable Detection: identifying whether a sentence contains a variable mention or not. - Variable Disambiguation: identifying which variable from a given vocabulary is mentioned in a sentence.

TArC

A morpho-syntactically annotated Tunisian Arabish Corpus (TArC). Source: TArC: Incrementally and Semi-Automatically Collecting a Tunisian Arabish Corpus

Twitter Sentiment Analysis

This is an entity-level Twitter Sentiment Analysis dataset. For each message, the task is to judge the sentiment of the entire sentence towards a given entity. For example, A outperforms B is positive for entity A but negative for entity B. The dataset contains ~70K labeled training messages and 1K labeled validation messages. It is available online for free on Kaggle.

UIT-VSMEC

Emotion recognition is a higher approach or special case of sentiment analysis. In this task, the result is not produced in terms of either polarity: positive or negative or in the form of rating (from 1 to 5) but of a more detailed level of sentiment analysis in which the result are depicted in more expressions like sadness, enjoyment, anger, disgust, fear and surprise. Emotion recognition plays a critical role in measuring brand value of a product by recognizing specific emotions of customers’ comments. In this study, we have achieved two targets. First and foremost, we built a standard Vietnamese Social Media Emotion Corpus (UIT-VSMEC) with about 6,927 human-annotated sentences with six emotion labels, contributing to emotion recognition research in Vietnamese which is a low-resource language in Natural Language Processing (NLP). Secondly, we assessed and measured machine learning and deep neural network models on our UIT-VSMEC. As a result, Convolutional Neural Network (CNN) model achieved the highest performance with 57.61% of F1-score.

DAWT

The DAWT dataset consists of Densely Annotated Wikipedia Texts across multiple languages. The annotations include labeled text mentions mapping to entities (represented by their Freebase machine ids) as well as the type of the entity. The data set contains total of 13.6M articles, 5.0B tokens, 13.8M mention entity co-occurrences. DAWT contains 4.8 times more anchor text to entity links than originally present in the Wikipedia markup. Moreover, it spans several languages including English, Spanish, Italian, German, French and Arabic. Source: DAWT: Densely Annotated Wikipedia Texts across multiple languages

DICE: a Dataset of Italian Crime Event news

The dataset contains the main components of the news articles published online by the newspaper named <a href="https://gazzettadimodena.gelocal.it/modena">Gazzetta di Modena</a>: url of the web page, title, sub-title, text, date of publication, crime category assigned to each news article by the author. The news articles are written in Italian and describe 11 types of crime events occurred in the province of Modena between the end of 2011 and 2021. Moreover, the dataset includes data derived from the abovementioned components thanks to the application of Natural Language Processing techniques. Some examples are the place of the crime event occurrence (municipality, area, address and GPS coordinates), the date of the occurrence, and the type of the crime events described in the news article obtained by an automatic categorization of the text. In the end, news articles describing the same crime events (duplciates) are detected by calculating the document similarity. Now, we are working on the application of question answering to extract the 5W+1H and we plan to extend the current dataset with the obtained data. Other researchers can employ the dataset to apply other algorithms of text categorization and duplicate detection and compare their results with the benchmark. The dataset can be useful for several scopes, e.g., geo-localization of the events, text summarization, crime analysis, crime prediction, community detection, topic modeling.

FreSaDa

FreSaDa is a French satire dataset for cross-domain satire detection, which is composed of 11,570 articles from the news domain. The dataset samples have been split into training, validation and test, such that the training publication sources are distinct from the validation and test publication sources. This gives rise to a cross-domain (cross-source) satire detection task.

Hyperpartisan News Detection

Hyperpartisan News Detection was a dataset created for PAN @ SemEval 2019 Task 4. Given a news article text, decide whether it follows a hyperpartisan argumentation, i.e., whether it exhibits blind, prejudiced, or unreasoning allegiance to one party, faction, cause, or person. There are two parts: * byarticle: Labeled through crowdsourcing on an article basis. The data contains only articles for which a consensus among the crowdsourcing workers existed. * bypublisher: Labeled by the overall bias of the publisher as provided by BuzzFeed journalists or MediaBiasFactCheck.com.

LLeQA

LLeQA is a French native dataset for studying information retrieval and long-form question answering in the legal domain. It consists of a knowledge corpus of 27,941 statutory articles collected from the Belgian legislation, and 1,868 legal questions posed by Belgian citizens and labeled by experienced jurists with a comprehensive answer rooted in relevant articles from the corpus.

MuLD

MuLD (Multitask Long Document Benchmark) is a set of 6 NLP tasks where the inputs consist of at least 10,000 words. The benchmark covers a wide variety of task types including translation, summarization, question answering, and classification. Additionally there is a range of output lengths from a single word classification label all the way up to an output longer than the input text.

RESD

Russian dataset of emotional speech dialogues. This dataset was assembled from ~3.5 hours of live speech by actors who voiced pre-distributed emotions in the dialogue for ~3 minutes each. <br> Each sample of dataset contains name of part from the original dataset studio source, speech file (16000 or 44100Hz) of human voice, 1 of 7 labeled emotions and the speech-to-texted part of voice speech. <br> Emotions are represented in 7 states: anger, disgust, fear, enthusiasm, happiness, neutral and sadness. This dataset was created by Artem Amentes, Nikita Davidchuk and Ilya Lubenets @misc{Aniemore, author = {Артем Аментес, Илья Лубенец, Никита Давидчук}, title = {Открытая библиотека искусственного интеллекта для анализа и выявления эмоциональных оттенков речи человека}, year = {2022}, publisher = {Hugging Face}, journal = {Hugging Face Hub}, howpublished = {\url{https://huggingface.com/aniemore/Aniemore}}, email = {hello@socialcode.ru} }

SaRoCo

SaRoCo is a dataset for detecting satire in Romanian news containing 55,608 news articles from multiple real and satirical news sources, of which 27,980 are regular and 27,628 satirical news reports. We provide the data in csv format, in three files following the train/validation/test splits.

SDCNL (Suicide vs Depression Classification)

We develop a primary dataset based on our task of suicide or depression classification. This dataset is web-scraped from Reddit. We collect our data from subreddits using the Python Reddit API. We specifically scrape from two subreddits, r/SuicideWatch3 and r/Depression. The dataset contains 1,895 total posts. We utilize two fields from the scraped data: the original text of the post as our inputs, and the subreddit it belongs to as labels. Posts from r/SuicideWatch are labeled as suicidal, and posts from r/Depression are labeled as depressed. We make this dataset and the web-scraping script available in our code.

StockEmotions

This repository contains a financial-domain-focused dataset for financial sentiment/emotion classification and stock market time series prediction. It's based on our paper: StockEmotions: Discover Investor Emotions for Financial Sentiment Analysis and Multivariate Time Series accepted by AAAI 2023 Bridge (AI for Financial Services). - Data collection period: Jan 2020 - Dec 2020 - Number of Utterance: 10,000 (train 80%, val 10%, test 10%) - Sentiment classes: 2 [bullish (~positive), bearish (~negative)] - Emotion classes: 12 [ambiguous, amusement, anger, anxiety, belief, confusion, depression, disgust, excitement, optimism, panic, surprise] - tweet/processed.csv: 50,281 samples with text-processed data for Topic Modelling - tweet/train, val, test.csv: 10,000 samples in total. Each file has id, date, ticker, emo_label, senti_lable, original, and processed content. For the data curation, processing (e.g. emoji, CTAG, HTAG), and annotation, we refer to our paper. The dataset is used for Financial Sentiment/Emotion Classification tasks. - price/38 companies: historical price data in csv format. The tweet and price dataset together are used for Multivariate Time Series tasks.

TCAB

Text Classification Attack Benchmark (TCAB) is a dataset for analyzing, understanding, detecting, and labeling adversarial attacks against text classifiers. TCAB includes 1.5 million attack instances, generated by twelve adversarial attack targeting three classifiers trained on six source datasets for sentiment analysis and abuse detection in English. The process of generating attacks is automated, so that TCAB can easily be extended to incorporate new text attacks and better classifiers as they are developed. Source: TCAB: A Large-Scale Text Classification Attack Benchmark

Wikipedia Title

Wikipedia Title is a dataset for learning character-level compositionality from the character visual characteristics. It consists of a collection of Wikipedia titles in Chinese, Japanese or Korean labelled with the category to which the article belongs. Source: https://arxiv.org/abs/1704.04859

AG’s Corpus

Antonio Gulli’s corpus of news articles is a collection of more than 1 million news articles. The articles have been gathered from more than 2000 news sources by ComeToMyHead in more than 1 year of activity. ComeToMyHead is an academic news search engine which has been running since July, 2004. The dataset is provided by the academic comunity for research purposes in data mining (clustering, classification, etc), information retrieval (ranking, search, etc), xml, data compression, data streaming, and any other non - commercial activity. A subset of this corpus, AG News, consisting of the 4 largest classes is a popular topic classification dataset. Source: AG's corpus of news articles

AVCAffe

We introduce AVCAffe, the first Audio-Visual dataset consisting of Cognitive load and Affect attributes. We record AVCAffe by simulating remote work scenarios over a video-conferencing platform, where subjects collaborate to complete a number of cognitively engaging tasks. AVCAffe is the largest originally collected (not collected from the Internet) affective dataset in English language. We recruit 106 participants from 18 different countries of origin, spanning an age range of 18 to 57 years old, with a balanced male-female ratio. AVCAffe comprises a total of 108 hours of video, equivalent to more than 58,000 clips along with task-based self-reported ground truth labels for arousal, valence, and cognitive load attributes such as mental demand, temporal demand, effort, and a few others. We believe AVCAffe would be a challenging benchmark for the deep learning research community given the inherent difficulty of classifying affect and cognitive load in particular. Moreover, our dataset fills an existing timely gap by facilitating the creation of learning systems for better self-management of remote work meetings, and further study of hypotheses regarding the impact of remote work on cognitive load and affective states.

Biographical

Biographical is a semi-supervised dataset for RE. The dataset, which is aimed towards digital humanities (DH) and historical research, is automatically compiled by aligning sentences from Wikipedia articles with matching structured data from sources including Pantheon and Wikidata.

CIC

The dataset is annotated with stance towards one topic, namely, the independence of Catalonia. Source: Multilingual Stance Detection: The Catalonia Independence Corpus

FairPrism

FairPrism is a dataset of 5,000 examples of AI-generated English text with detailed human annotations covering a diverse set of harms relating to gender and sexuality. FairPrism aims to address several limitations of existing datasets for measuring and mitigating fairness-related harms, including improved transparency, clearer specification of dataset coverage, and accounting for annotator disagreement and harms that are context-dependent. FairPrism’s annotations include the extent of stereotyping and demeaning harms, the demographic groups targeted, and appropriateness for different applications. The annotations also include specific harms that occur in interactive contexts and harms that raise normative concerns when the “speaker” is an AI system. Due to its precision and granularity, FairPrism can be used to diagnose (1) the types of fairness-related harms that AI text generation systems cause, and (2) the potential limitations of mitigation methods.

HeriGraph

The dataset contains constructed multi-modal features (visual and textual), pseudo-labels (on heritage values and attributes), and graph structures (with temporal, social, and spatial links) constructed using User-Generated Content data collected from Flickr social media platform in three global cities containing UNESCO World Heritage property (Amsterdam, Suzhou, Venice). The motivation of data collection in this project is to provide datasets that could be both directly applicable for ML communities as test-bed, and theoretically informative for heritage and urban scholars to draw conclusions on for planning decision-making.

HLGD

The Headline Grouping dataset is a binary classification dataset on pairs of news headline. For each pair of headline, the binary label indicates whether the two headlines are part of the same group (and describe the same underlying event), or whether they are in distinct groups. The dataset contains a total of 20k annotated headline pairs, further split in a train, validation and test portions.

KanHope

KanHope is a code mixed hope speech dataset for equality, diversity, and inclusion in Kannada, an under-resourced Dravidian language. The dataset consists of 6,176 user-generated comments in code mixed Kannada crawled from YouTube and manually labelled as bearing hope speech or not-hope speech.

LatamXIX

A novel dataset of 19th-century Latin American press texts, which addresses the lack of specialized corpora for historical and linguistic analysis in this region.

Law Stack Exchange

Description Dataset from the Law Stack Exchange, as used in "Parameter-Efficient Legal Domain Adaptation" (Li et al., 2022). We introduce a dataset with data from the Law Stack Exchange. This dataset is composed of questions from the Law Stack Exchange, which is a community forum-based website containing questions with answers to legal questions. We link the questions with their associated tags (e.g., "copyright" or "criminal-law"), and perform a multi-label classification task ## Citation Information @inproceedings{li-etal-2022-parameter, title = "Parameter-Efficient Legal Domain Adaptation", author = "Li, Jonathan and Bhambhoria, Rohan and Zhu, Xiaodan", booktitle = "Proceedings of the Natural Legal Language Processing Workshop 2022", month = dec, year = "2022", address = "Abu Dhabi, United Arab Emirates (Hybrid)", publisher = "Association for Computational Linguistics", url = "https://aclanthology.org/2022.nllp-1.10", pages = "119--129", }

nEMO

Overview nEMO is a simulated dataset of emotional speech in the Polish language. The corpus contains over 3 hours of samples recorded with the participation of nine actors portraying six emotional states: anger, fear, happiness, sadness, surprise, and a neutral state. The text material used was carefully selected to represent the phonetics of the Polish language. The corpus is available for free under the Creative Commons license (CC BY-NC-SA 4.0). The dataset is available on Hugging Face and GitHub. ## Data Fields - file_id - filename, i.e. {speaker_id}_{emotion}_{sentence_id}, - audio (audio) - dictionary containing audio array, path and sampling rate (available when accessed via datasets library), - emotion - label corresponding to emotional state, - raw_text - original (orthographic) transcription of the audio, - normalized_text - normalized transcription of the audio, - speaker_id - id of speaker, - gender - gender of the speaker, - age - age of the speaker. ## Usage The nEMO dataset can be loaded and processed using the datasets library: python from datasets import load_dataset nemo = load_dataset("amu-cai/nEMO", split="train") To work with the nEMO dataset on GitHub, you may clone the repository and access the files directly within the samples folder. Corresponding metadata can be found in the data.tsv file. The nEMO dataset is provided as a whole, without predefined training and test splits. This allows researchers and developers flexibility in creating their splits based on the specific needs. ## Supported Tasks - Audio classification: This dataset was mainly created for the task of speech emotion recognition. Each recording is labeled with one of six emotional states (anger, fear, happiness, sadness, surprised, and neutral). Additionally, each sample is labeled with speaker id and speaker gender. Because of that, the dataset can also be used for different audio classification tasks. - Automatic Speech Recognition: The dataset includes orthographic and normalized transcriptions for each audio recording, making it a useful resource for automatic speech recognition (ASR) tasks. The sentences were carefully selected to cover a wide range of phonemes in the Polish language. - Text-to-Speech: The dataset contains emotional audio recordings with transcriptions, which can be valuable for developing TTS systems that produce emotionally expressive speech. ## Additional Information ### Licensing Information The dataset is available under the Creative Commons license (CC BY-NC-SA 4.0). ### Citation Information You can access the nEMO paper at arXiv. Please cite the paper when referencing the nEMO dataset as: @misc{christop2024nemo, title={nEMO: Dataset of Emotional Speech in Polish}, author={Iwona Christop}, year={2024}, eprint={2404.06292}, archivePrefix={arXiv}, primaryClass={cs.CL} } ### Contributions Thanks to @iwonachristop for adding this dataset.

NLP Taxonomy Classification Data

The dataset consists of titles and abstracts from NLP-related papers. Each paper is annotated with multiple fields of study from an NLP taxonomy. The training dataset contains 178,521 weakly annotated samples. The test dataset consists of 828 manually annotated samples from the EMNLP22 conference. The manually labeled test dataset might not contain all possible classes since it consists of EMNLP22 papers only, and some rarer classes haven’t been published there. Therefore, we advise creating an additional test or validation set from the train data that includes all the possible classes.

출처: Papers with Code · CC BY-SA 4.0. 아카이브 자료와 새로 수집한 논문은 별도로 관리합니다.