AG News (AG’s News Corpus) is a subdataset of AG's corpus of news articles constructed by assembling titles and description fields of articles from the 4 largest classes (“World”, “Sports”, “Business”, “Sci/Tech”) of AG’s Corpus. The AG News contains 30,000 training and 1,900 test samples per class. Source: https://arxiv.org/pdf/1509.01626.pdf

DBpedia (from "DB" for "database") is a project aiming to extract structured content from the information created in the Wikipedia project. DBpedia allows users to semantically query relationships and properties of Wikipedia resources, including links to other related datasets. Source: https://en.wikipedia.org/wiki/DBpedia

The ATIS (Airline Travel Information Systems) is a dataset consisting of audio recordings and corresponding manual transcripts about humans asking for flight information on automated airline travel inquiry systems. The data consists of 17 unique intent categories. The original split contains 4478, 500 and 893 intent-labeled reference utterances in train, development and test set respectively. Source: Spoken Language Intent Detection using Confusion2Vec

The SNIPS Natural Language Understanding benchmark is a dataset of over 16,000 crowdsourced queries distributed among 7 user intents of various complexity: * SearchCreativeWork (e.g. Find me the I, Robot television show), * GetWeather (e.g. Is it windy in Boston, MA right now?), * BookRestaurant (e.g. I want to book a highly rated restaurant in Paris tomorrow night), * PlayMusic (e.g. Play the last track from Beyoncé off Spotify), * AddToPlaylist (e.g. Add Diamonds to my roadtrip playlist), * RateBook (e.g. Give 6 stars to Of Mice and Men), * SearchScreeningEvent (e.g. Check the showtimes for Wonder Woman in Paris). The training set contains of 13,084 utterances, the validation set and the test set contain 700 utterances each, with 100 queries per intent. Source: https://paperswithcode.com/paper/snips-voice-platform-an-embedded-spoken/

MTEB is a benchmark that spans 8 embedding tasks covering a total of 56 datasets and 112 languages. The 8 task types are Bitext mining, Classification, Clustering, Pair Classification, Reranking, Retrieval, Semantic Textual Similarity and Summarisation. The 56 datasets contain varying text lengths and they are grouped into three categories: Sentence to sentence, Paragraph to paragraph, and Sentence to paragraph. Check the latest leaderboards at HuggingFace.

Dataset composed of online banking queries annotated with their corresponding intents. BANKING77 dataset provides a very fine-grained set of intents in a banking domain. It comprises 13,083 customer service queries labeled with 77 intents. It focuses on fine-grained single-domain intent detection.

This dataset is for evaluating the performance of intent classification systems in the presence of "out-of-scope" queries, i.e., queries that do not fall into any of the system-supported intent classes. The dataset includes both in-scope and out-of-scope data. Source: CLINC150

The 20 Newsgroups data set is a collection of approximately 20,000 newsgroup documents, partitioned (nearly) evenly across 20 different newsgroups.

A large-scale curated dataset of over 152 million tweets, growing daily, related to COVID-19 chatter generated from January 1st to April 4th at the time of writing. Source: A large-scale COVID-19 Twitter chatter dataset for open scientific research -- an international collaboration

The dataset contains the main components of the news articles published online by the newspaper named <a href="https://gazzettadimodena.gelocal.it/modena">Gazzetta di Modena</a>: url of the web page, title, sub-title, text, date of publication, crime category assigned to each news article by the author. The news articles are written in Italian and describe 11 types of crime events occurred in the province of Modena between the end of 2011 and 2021. Moreover, the dataset includes data derived from the abovementioned components thanks to the application of Natural Language Processing techniques. Some examples are the place of the crime event occurrence (municipality, area, address and GPS coordinates), the date of the occurrence, and the type of the crime events described in the news article obtained by an automatic categorization of the text. In the end, news articles describing the same crime events (duplciates) are detected by calculating the document similarity. Now, we are working on the application of question answering to extract the 5W+1H and we plan to extend the current dataset with the obtained data. Other researchers can employ the dataset to apply other algorithms of text categorization and duplicate detection and compare their results with the benchmark. The dataset can be useful for several scopes, e.g., geo-localization of the events, text summarization, crime analysis, crime prediction, community detection, topic modeling.

Urdu News Headlines Dataset with VOA and BBC An Urdu news headlines dataset is a collection of news headlines in the Urdu language, typically scraped from news websites and social media platforms. These datasets can be valuable for researchers and developers working on a variety of tasks, such as: Machine translation: Training machine translation models to translate between Urdu and other languages. Text summarization: Developing algorithms for automatically summarizing Urdu news articles. Natural language processing: Studying the structure and grammar of the Urdu language. Clustering: Grouping news articles into similar categories based on their headlines. Benefits of using a dataset with VOA and BBC headlines There are several benefits to using a dataset that includes news headlines from VOA and BBC: High quality: VOA and BBC are well-respected news organizations known for their high-quality journalism. This means that the headlines in the dataset are likely to be accurate and unbiased. Diversity of topics: VOA and BBC cover a wide range of topics, including politics, business, sports, and entertainment. This means that the dataset will be representative of the different types of news that are available in the Urdu language. Large size: VOA and BBC have been publishing news in Urdu for many years. This means that there is a large amount of data available, which can be used to train machine learning models and to conduct research. Here are some examples of how a Urdu news headlines dataset with VOA and BBC headlines can be used: A machine translation model could be trained on the dataset to translate Urdu news headlines into English. This could be used to make Urdu news more accessible to a wider audience. A text summarization algorithm could be trained on the dataset to automatically generate summaries of Urdu news articles. This could be used to save people time and to help them to stay informed. A clustering algorithm could be used to group Urdu news articles into similar categories based on their headlines. This could be used to create personalized news feeds for users. Here are some limitations of using a Urdu news headlines dataset with VOA and BBC headlines: Limited scope: VOA and BBC are primarily English-language news organizations. This means that the dataset may not be representative of the full range of Urdu news that is available. Bias: VOA and BBC are both Western news organizations. This means that the dataset may be biased towards Western perspectives. Limited access: The data may not be readily available to everyone.

About the MNAD Dataset The MNAD corpus is a collection of over 1 million Moroccan news articles written in modern Arabic language. These news articles have been gathered from 11 prominent electronic news sources. The dataset is made available to the academic community for research purposes, such as data mining (clustering, classification, etc.), information retrieval (ranking, search, etc.), and other non-commercial activities. ## Dataset Fields - Title: The title of the article - Body: The body of the article - Category: The category of the article - Source: The Electronic News paper source of the article ## About Version 1 of the Dataset (MNAD.v1) Version 1 of the dataset comprises 418,563 articles classified into 19 categories. The data was collected from well-known electronic news sources, namely Akhbarona.ma, Hespress.ma, Hibapress.com, and Le360.com. The articles were stored in four separate CSV files, each corresponding to the news website source. Each CSV file contains three fields: Title, Body, and Category of the news article. The dataset is rich in Arabic vocabulary, with approximately 906,125 unique words. It has been utilized as a benchmark in the research paper: "A Moroccan News Articles Dataset (MNAD) For Arabic Text Categorization". In 2021 International Conference on Decision Aid Sciences and Application (DASA). This dataset is available for download from the following sources: - Kaggle Datasets : MNADv1 - Huggingface Datasets: MNADv1 ## About Version 2 of the Dataset (MNAD.v2) Version 2 of the MNAD dataset includes an additional 653,901 articles, bringing the total number of articles to over 1 million (1,069,489), classified into the same 19 categories as in version 1. The new documents were collected from seven additional prominent Moroccan news websites, namely al3omk.com, medi1news.com, alayam24.com, anfaspress.com, alyaoum24.com, barlamane.com, and SnrtNews.com. The newly collected articles have been merged with the articles from the previous version into a single CSV file named MNADv2.csv. This file includes an additional column called "Source" to indicate the source of each news article. Furthermore, MNAD.v2 incorporates improved pre-processing techniques and data cleaning methods. These enhancements involve removing duplicates, eliminating multiple spaces, discarding rows with NaN values, replacing new lines with "\n", excluding very long and very short articles, and removing non-Arabic articles. These additions and improvements aim to enhance the usability and value of the MNAD dataset for researchers and practitioners in the field of Arabic text analysis. This dataset is available for download from the following sources: - Kaggle Datasets : MNADv2 - Huggingface Datasets: MNADv2 ## Citation If you use our data, please cite the following paper: bibtex @inproceedings{MNAD2021, author = {Mourad Jbene and Smail Tigani and Rachid Saadane and Abdellah Chehri}, title = {A Moroccan News Articles Dataset ({MNAD}) For Arabic Text Categorization}, year = {2021}, publisher = {{IEEE}}, booktitle = {2021 International Conference on Decision Aid Sciences and Application ({DASA})} doi = {10.1109/dasa53625.2021.9682402}, url = {https://doi.org/10.1109/dasa53625.2021.9682402}, }