Persian NLP Hub
Click to add a brief description of the dataset (Markdown and LaTeX enabled). Provide: * a high-level explanation of the dataset characteristics * explain motivations and summary of its content * potential use cases of the dataset
태스크의 하위 분류와 논문·코드·평가·기법·데이터셋을 탐색하세요.
Text Classification is the task of assigning a sentence or document an appropriate category. The categories depend on the chosen dataset and can range from topics.
Text Classification problems include emotion classification, news classification, citation intent classification, among others. Benchmark datasets for evaluating text classification capabilities include GLUE, AGNews, among others.
In recent years, deep learning techniques like XLNet and RoBERTa have attained some of the biggest performance jumps for text classification problems.
<span style="color:grey; opacity: 0.6">( Image credit: Text Classification Algorithms: A Survey )</span>
출처에 같은 태스크 이름으로 연결된 데이터셋입니다. 동명 태스크가 포함될 수 있으므로, 비교할 때는 평가 결과의 조건도 확인하세요.
Click to add a brief description of the dataset (Markdown and LaTeX enabled). Provide: * a high-level explanation of the dataset characteristics * explain motivations and summary of its content * potential use cases of the dataset
Dataset with articles posted in the r/Liberal and r/Conservative subreddits. In total, we collected a corpus of 226,010 articles. We have collected news articles to understand political expression through the shared news articles.
Internet Archive Scholar Reference Dataset.
Reader Emotion News 20k Dataset
RVL-CDIP_MP is our first contribution to retrieve the original documents of the IIT-CDIP test collection which were used to create RVL-CDIP. Some PDFs or encoded images were corrupt, which explains that we have around 500 fewer instances. By leveraging metadata from OCR-IDL , we matched the original identifiers from IIT-CDIP and retrieved them from IDL using a conversion. It has the same label taxonomy as RVL-CDIP (16) with close to 400K documents in PDF format, averaging 5 pages per document.
RVL-CDIP_MP-N can serve its original goal as a covariate shift test set, now for multi-page document classification. We were able to retrieve the original full documents from DocumentCloud and Web Search. It has the same label taxonomy as RVL-CDIP (16) with close to 1K documents in PDF format, averaging 10 pages per document.
This resource contains 10.5 million paragraphs with associated statement labels, realized as one paragraph per file, one sentence per line. Each file is placed in a subdirectory named after its annotated class. The statements were extracted from author-annotated environments, where we only selected the first paragraph,immediately following the heading. Headings include both structural sections (e.g. Introduction), as well as scholarly statement annotations, (e.g. Definition, Proof, Remark). The annotated statement dataset is derived from arXMLiv, a machine-readable HTML5 representation of the arXiv corpus of scientific articles. ### Examples Definition with math lexemes (main data, single sentence, linebreaks for readability): a directed quantum turing automaton is a quadruple italic_T RELOP_equals OPEN_( caligraphic_H PUNCT_, caligraphic_K PUNCT_, caligraphic_L PUNCT_, italic_tau CLOSE_) PUNCT_, where caligraphic_H caligraphic_K and caligraphic_L are finite dimensional hilbert spaces over the complex field blackboard_C and italic_tau METARELOP_colon caligraphic_H MULOP_tensor_product caligraphic_K ARROW_rightarrow caligraphic_H MULOP_tensor_product caligraphic_L is an isometry in fdhilb source: definition/1e4a1aea317bbf363c5314fb25eaf72c8a350a1007bb8aafc542e188405b93d5.txt Same definition without math lexemes (nomath data, single sentence, linebreaks for readability): a directed quantum turing automaton is a quadruple where and are finite dimensional hilbert spaces over the complex field and is an isometry in fdhilb nomath source: definition/35b170bae4259a5c430846116142d4e4a45097e52daf818b78ea378d94d14a21.txt
The ShapeIt dataset introduced by Alper et al. (2023) consists of 109 nouns and noun phrases along with the basic shape normally associated with that item, chosen from the set {circle, rectangle, triangle}.
@inproceedings{tsai2025harmful, author = {Elisa Tsai and Neal Mangaokar and Boyuan Zheng and Haizhong Zheng and Atul Prakash}, title = {Harmful Terms and Where to Find Them: Measuring and Modeling Unfavorable Financial Terms and Conditions in Shopping Websites at Scale}, booktitle = {Proceedings of the ACM Web Conference 2025 (WWW ’25)}, year = {2025}, location = {Sydney, NSW, Australia}, publisher = {ACM}, address = {New York, NY, USA}, pages = {14}, month = {April 28-May 2}, doi = {10.1145/3696410.3714573} } ## Dataset Description The dataset consists of sanitized terms extracted from e-commerce websites with English terms and conditions. The websites were sourced from the Tranco list (as of April 2024).ShortPersianEmo is a new data set for emotion recognition in Persian short texts. The ShortPersianEmo dataset is a single-label dataset that contains 5472 short Persian texts collected from Twitter and Digikala. Our dataset is annotated according to Rachael Jack’s emotional model in five emotional classes happiness, sadness, anger, fear, and other. Unlike publicly accessible datasets that do not impose any restrictions on text length, ShortPersianEmo specifically focuses on short texts. The average text length in the ShortPersianEmo dataset is 56 words. Table 1 presents a comparison between the introduced ShortPersianEmo dataset and other datasets from the literature for emotion detection in Persian text. For more information on this dataset please read our paper. If you use this dataset in any research work, please cite our paper.
The Sequence labellIng evaLuatIon benChmark fOr spoken laNguagE (SILICONE) benchmark is a collection of resources for training, evaluating, and analyzing natural language understanding systems specifically designed for spoken language. All datasets are in the English language and covers a large variety of domains (e.g daily life, scripted scenarios, joint task completion, phone call conversations, and televsion dialogue). Some datasets additionally include emotion and/or sentiment labels.
SmokEng is a dataset of 3144 tweets, which are selected based on the presence of colloquial slang related to smoking and analyze it based on the semantics of the tweet. Source: SmokEng: Towards Fine-grained Classification of Tobacco-related Social Media Text
StEduCov, a dataset annotated for stances toward online education during the COVID-19 pandemic. StEduCov has 17,097 tweets gathered over 15 months, from March 2020 to May 2021, using Twitter API. The tweets are manually annotated into agree, disagree or neutral classes. We used a set of relevant hashtags and keywords. Specifically, we utilised a combination of hashtags, such as '#COVID 19' or '#Coronavirus' with keywords, such as 'education', 'online learning', 'distance learning' and 'remote learning'. To ensure high annotation quality, three different annotators annotated each tweet and at least one of the reviewers from three judges revised it. They were guided by some instructions, such as that in the case of disagree class, there should be a clear negative statement about online education or its impact. Also, if the tweet is negative but refers to other people (e.g. 'my children hate online learning').
Table-ACM12K (TACM12K) is a relational table dataset derived from the ACM heterogeneous graph dataset. It includes four tables: papers, authors, citations, and writings. The paper table features attributes such as year, title, and abstract, while the author table includes name and affiliation details. Additionally, some feature completion has been performed for the papers. The dataset also defines a standard classification task for predicting the conference to which a paper belongs.
The Mafia Dataset was created to model the behavior of deceptive actors in the context of the Mafia game, as described in the paper “Putting the Con in Context: Identifying Deceptive Actors in the Game of Mafia”. We hope that this dataset will be of use to others studying the effects of deception on language use.
We introduce a large semi-automatically generated dataset of ~400,000 descriptive sentences about commonsense knowledge that can be true or false in which negation is present in about 2/3 of the corpus in different forms that we use to evaluate LLMs. Source: ThisIsNotaDataset Image: ThisIsNotaDataset
Click to add a brief description of the dataset (Markdown and LaTeX enabled). Provide: * a high-level explanation of the dataset characteristics * explain motivations and summary of its content * potential use cases of the dataset
A prevalent use case of topic models is that of topic discovery. However, most of the topic model evaluation methods rely on abstract metrics such as perplexity or topic coherence. The topic coverage approach is to measure the models' performance by matching model-generated topics to topics discovered by humans. This way, the models are evaluated in the context of their use, by essentially simulating topic modeling in a fixed setting defined by a text collection and a set of reference topics. Reference topics represent a ground truth that can be used to evaluate both topic models and other measures of model performance. The coverage approach enables large-scale automatic evaluation of both existing and future topic models. The topic coverage dataset consists of two text collections and two sets of reference topics. These two sub-datasets correspond to two domains (news text and biological text) where topic models are used for topic discovery in large text collections. The reference topics consist of model-generated topics inspected, selected, and curated by humans. Each dataset contains a corpus of preprocessed (tokenized) texts and a set of reference topics, each represented by a list of words and text documents. The dataset details, including the instruction for the use of the data and supporting code, are here: https://github.com/dkorenci/topic_coverage/blob/main/data.readme.txt The coverage measures that can be used to evaluate topic models are described in the accompanying paper, whereas the code and the instructions can be found in the github repo.
6000 French user reviews from three applications on Google Play (Garmin Connect, Huawei Health, Samsung Health) are labelled manually. We selected four labels: rating, bug report, feature request and user experience. * Ratings are simple text which express the overall evaluation to that app, including praise, criticism, or dissuasion. * Bug reports show the problems that users have met while using the app, like loss of data, crash of app, connection error, etc. * Feature requests reflect the demande of users on new function, new content, new interface, etc. * In user experience, users describe their experience in relation to the functionality of the app, how does certain functions be helpful. As we can observe from the following table, that shows examples of labelled user reviews, each review belongs to one or more categories. | App | Total | Rating | Bug report | Feature request | User experience | | -------------- | ----- | ------ | ---------- | --------------- | --------------- | | Garmin Connect | 2000 | 1260 | 757 | 170 | 493 | | Huawei Health | 2000 | 1068 | 819 | 384 | 289 | | Samsung Health | 2000 | 1324 | 491 | 486 | 349 |
The Trilemma of Truth is a multiclass probing dataset for evaluating the veracity-tracking mechanism of large language models. It comprises three configurations— city_locations (statements about where cities are), med_indications (drug–condition assignments), and word_definitions - each annotated as true, false, or neither. Designed to plug straight into sAwMIL and various veracity probes.
TruthGen is a dataset of generated true and false statements, intended for research on truthfulness in reward models and language models, specifically in contexts where political bias is undesirable. This dataset contains 1,987 statement pairs (3,974 statements in total), with each pair containing one objectively true statement and one false statement. It spans a variety of everyday and scientific facts, excluding politically charged topics to the greatest extent possible. The dataset is particularly useful for evaluating reward models trained for alignment with truth, as well as for research on mitigating political bias while improving model accuracy on truth-related tasks.
This dataset for abusive content detection in Twitter consists of two sets of annotations for the same set of tweets, one where the human annotators had access to the tweet's content and one where they didn't know the context.
Education is increasingly data-driven, and the ability to analyse and adapt educational materials quickly and effectively is important for keeping materials contemporary and interesting. These approaches also have the potential to personalise learning experiences. One of the challenges in this domain is aligning new literature with the appropriate educational stages. This dataset aims to contribute to alleviating this knowledge gap. This dataset has been generated through literature in the public domain from Project Gutenberg, and cross-referenced by the UK Key Stage equivalents from the Lexile Reading Framework. The dataset contains a total of 20,000 rows evenly distributed across four educational stages - Key Stage 2 (KS2), Key Stage 3 (KS3), Key Stage 4 (KS4), and Key Stage 5 (KS5). The data has been split into Train (80%, 16,000 objects) and Test (20%, 4,000 objects) sets. The data is multimodal and contains: - Text - the cropped excerpt of text, which is limited to 512 tokens to the nearest complete sentence. - Linguistic Features - each extracted from the text excerpt
This dataset includes User Story (or Issue) text descriptions, User Story titles, and Story Points from 33 software development projects, comprising a total of 20,479 User Stories (or issues) extracted from GitLab repositories, amounting to 12,262.7 Story Points. The mining process focused on GitLab’s top open-source projects that use agile software development methodologies and record task sizes in Story Points. Only tasks with the State attribute set to Closed and with the Weight attribute filled in were collected. The Weight field in GitLab is used to record the effort in Story Points. The data was mined between January 2023 and April 2023. The projects in the dataset have diverse characteristics, covering different programming languages, business domains, and geographic locations of the teams. The paper below describe shortly this dataset in section 4.1 Dataset Citation Suggestion Neo, G. S.; Moura, J.; Almeida, H.; Neo, A. and Freitas Júnior, O. (2024). User Story Tutor (UST) to Support Agile Software Developers. In Proceedings of the 16th International Conference on Computer Supported Education - Volume 2: CSEDU; ISBN 978-989-758-697-2; ISSN 2184-5026, SciTePress, pages 51-62. DOI: 10.5220/0012619200003693
This research introduces the dataset that we created to test voice emotional recognition models with Vietnamese data. The data set is the result of research, testing, and filtering 250 emotional segments from movie, movie series and live show divided equally for 5 basic emotional states of humans: “anger, happiness, sadness, neutral and anxiety”, VNEMOS contains approximately 30 minutes long. This dataset brings a balanced and diverse emotional set to study emotional recognition issues related to understanding human mood and emotions. Through evaluation Our dataset achieved a 89% accuracy, showcasing the adeptness at capturing the essence of positive emotions and robust learning capabilities in real-world scenarios. By clarifying these nuances. Our research will contribute much to future work on emotion recognition and signal processing in Vietnamese and data using human psychological science. Detailed information of the dataset is in the following link: “bit.ly/VNEMOS”.
Wiki-en is an annotated English dataset for domain detection extracted from Wikipedia. It includes texts from 7 different domains: “Business and Commerce” (BUS), “Government and Politics” (GOV), “Physical and Mental Health” (HEA), “Law and Order” (LAW), “Lifestyle” (LIF), “Military” (MIL), and “General Purpose” (GEN). Source: https://arxiv.org/pdf/1907.11499.pdf
Wiki-Reliability is the first dataset of English Wikipedia articles annotated with a wide set of content reliability issues. Templates are tags used by expert Wikipedia editors to indicate content issues, such as the presence of "non-neutral point of view" or "contradictory articles", and serve as a strong signal for detecting reliability issues in a revision. We select the 10 most popular reliability-related templates on Wikipedia, and propose an effective method to label almost 1M samples of Wikipedia article revisions as positive or negative with respect to each template. Each positive/negative example in the dataset comes with the full article text and 20 features from the revision's metadata. We provide an overview of the possible downstream tasks enabled by such data, and show that Wiki-Reliability can be used to train large-scale models for content reliability prediction.
Wiki-zh is an annotated Chinese dataset for domain detection extracted from Wikipedia. It includes texts from 7 different domains: “Business and Commerce” (BUS), “Government and Politics” (GOV), “Physical and Mental Health” (HEA), “Law and Order” (LAW), “Lifestyle” (LIF), “Military” (MIL), and “General Purpose” (GEN). It contains 26,280 documents split into training, validation and test. Source: https://arxiv.org/pdf/1907.11499.pdf
The CIDII dataset is a binary classification, consisting of two classes of correct information and disinformation related to Islamic issues. The CIDII dataset belongs to our research (DISINFORMATION DETECTION ABOUT ISLAMIC ISSUES ON SOCIAL MEDIA USING DEEP LEARNING TECHNIQUES) published in MJCS journal in the link below: https://ejournal.um.edu.my/index.php/MJCS/article/view/41935 This dataset consists of five columns: 1- ID: Each article has a unique ID. 2- Article: The article contains text that is either facts related to Islamic issues if the information is correct, or posts targeting the Islamic religion if the information is false. Most posts contain only the body without a title. 3- Propagation Source: The source refers to the source of the article content, as it contains a Facebook link in the event that the post is disinformation, or it contains a link to Islamic websites in the event that the article refers to correct information (an explanation of a verse, a hadith, or an article related to the Islamic religion). 4- Article Type: This column contains the type of article published. Is it a post if the article is disinformation, or is it an Islamic article, a Quranic interpretation, or a hadith, in case of that the information is correct? 5- Class Type: This column shows whether the article belongs to the category of correct information or disinformation.
With the emergence of the COVID-19 pandemic, the political and the medical aspects of disinformation merged as the problem got elevated to a whole new level to become the first global infodemic. Fighting this infodemic has been declared one of the most important focus areas of the World Health Organization, with dangers ranging from promoting fake cures, rumors, and conspiracy theories to spreading xenophobia and panic. Addressing the issue requires solving a number of challenging problems such as identifying messages containing claims, determining their check-worthiness and factuality, and their potential to do harm as well as the nature of that harm, to mention just a few. To address this gap, we release a large dataset of 16K manually annotated tweets for fine-grained disinformation analysis that focuses on COVID-19, combines the perspectives and the interests of journalists, fact-checkers, social media platforms, policy makers, and society, and covers Arabic, Bulgarian, Dutch, and English. Finally, we show strong evaluation results using pretrained Transformers, thus confirming the practical utility of the dataset in monolingual multilingual, and single task vs. multitask settings.
A dataset specifically tailored to the biotech news sector, aiming to transcend the limitations of existing benchmarks. This dataset is rich in complex content, comprising various biotech news articles covering various events, thus providing a more nuanced view of information extraction challenges. A standout feature of our dataset is its dual-layered approach: it not only identifies the general theme of the articles but also extracts specific information about the target companies involved. This dual focus significantly elevates the dataset’s relevance for applications demanding a deeper insight into the interplay of events, companies, and the broader biotech industry. The dataset encompasses 31 classes, including a ‘None’ category, to cover various events and information types such as event organisation, executive statements, regulatory approvals, hiring announcements, and more. Key aspects: - Event extraction - Multi-class classification - Biotech news domain - 31 classes - 3140 total number of examples Classes: - alliance & partnership - forming an alliance or partnership - article publication - publishing an article - clinical trial sponsorship - Sponsoring or participating in a clinical trial - closing - shutting down a facility/office/division or ceasing an initiative - company description - describing or profiling the company - department establishment - establishing a new department or division - event organization - organizing or participating in an event like a conference, exhibition, etc - event organization - organizing or participating in an event - executive appointment - appointing a new executive - executive statement - a statement or quote from an executive of a company - expanding geography - expanding into new geographical areas - expanding industry - expanding into new industries or markets - foundation - establishing a new charitable foundation - funding round - raising a new round of funding - hiring - announcing new hires or appointments at the company - investment in public company - making an investment in a public company - ipo exit - having an initial public offering or acquisition exit - m&a - mergers, acquisitions, or divestitures - new initiatives & programs - announcing new initiatives or programs - new initiatives or programs - announcing new initiatives, programs, or campaigns - None - no label - other - other events that don't fit into defined categories - participation in an event - participating in an industry event, conference, etc - partnerships & alliances - forming partnerships or strategic alliances with other companies - patent publication - publication of a new patent filing - product launching & presentation - launching or unveiling a new product - product updates - announcing updates or new versions of existing products - regulatory approval - getting approval from regulatory bodies for products, services, trials, etc. - service & product providing - launching or expanding products or services - subsidiary establishment - establishing a new subsidiary company - support & philanthropy - philanthropic activities or donations
This dataset consists of 600+ items of faces with different emotions and mixed races that are ready to use for optimizing the accuracy of computer vision models. Human age range from 20 to 60 years old, balance in gender, no occlusion, with head direction (<45 degree up-down and left-right). All of the contents is sourced from PIXTA's stock library of 100M+ Asian-featured images and videos. PIXTA is the largest platform of visual materials in the Asia Pacific region offering fully-managed services, high quality contents and data, and powerful tools for businesses & organizations to enable their creative and machine learning projects. For more details, please refer to the link: https://www.pixta.ai/ Or send your inquiries to contact@pixta.ai
This dataset is an extremely challenging set of over 20,000+ original Number plate images captured and crowdsourced from over 700+ urban and rural areas, where each image is manually reviewed and verified by computer vision professionals at Datacluster Labs Dataset Features - Dataset size : 20,000+ - Captured by : Over 4000+ crowdsource contributors - Resolution : 100% of images HD and above (1920x1080 and above) - Location : Captured with 700+ cities and villages across India - Diversity : Various lighting conditions like day, night, varied distances, view points etc. - Device used : Captured using mobile phones in 2020-2022 - Usage : Number plate detection, ANPR, Number plate recognition, Self driving system, etc. Available Annotation formats - COCO, YOLO, PASCAL-VOC, Tf-Record To download full datasets or to submit a request for your dataset needs, please ping us at sales@datacluster.ai Visit www.datacluster.ai to know more.
출처: Papers with Code · CC BY-SA 4.0. 아카이브 자료와 새로 수집한 논문은 별도로 관리합니다.