Fake News Detection

LIAR is a publicly available dataset for fake news detection. A decade-long of 12.8K manually labeled short statements were collected in various contexts from POLITIFACT.COM, which provides detailed analysis report and links to source documents for each case. This dataset can be used for fact-checking research as well. Notably, this new dataset is an order of magnitude larger than previously largest public fake news datasets of similar type. The LIAR dataset4 includes 12.8K human labeled short statements from POLITIFACT.COM’s API, and each statement is evaluated by a POLITIFACT.COM editor for its truthfulness. Source: “Liar, Liar Pants on Fire”: A New Benchmark Dataset for Fake News Detection

RealNews is a large corpus of news articles from Common Crawl. Data is scraped from Common Crawl, limited to the 5000 news domains indexed by Google News. The authors used the Newspaper Python library to extract the body and metadata from each article. News from Common Crawl dumps from December 2016 through March 2019 were used as training data; articles published in April 2019 from the April 2019 dump were used for evaluation. After deduplication, RealNews is 120 gigabytes without compression. Image Source: https://arxiv.org/pdf/1905.12616v3.pdf

The Weibo NER dataset is a Chinese Named Entity Recognition dataset drawn from the social media website Sina Weibo. Source: Chinese NER Using Lattice LSTM Image Source: https://en.wikipedia.org/wiki/Sina_Weibo

FakeNewsNet is collected from two fact-checking websites: GossipCop and PolitiFact containing news contents with labels annotated by professional journalists and experts, along with social context information. Source: Leveraging Multi-Source Weak Social Supervision for Early Detection of Fake News Image Source: https://arxiv.org/pdf/1809.01286.pdf

Fact-checking (FC) articles which contains pairs (multimodal tweet and a FC-article) from snopes.com. Source: Where Are the Facts? Searching for Fact-checked Information to Alleviate the Spread of Fake News

Fact-checking (FC) articles which contains pairs (multimodal tweet and a FC-article) from politifact.com. Source: Where Are the Facts? Searching for Fact-checked Information to Alleviate the Spread of Fake News

FNC-1 was designed as a stance detection dataset and it contains 75,385 labeled headline and article pairs. The pairs are labelled as either agree, disagree, discuss, and unrelated. Each headline in the dataset is phrased as a statement Source: Investigating Rumor News Using Agreement-Aware Search Image Source: http://www.fakenewschallenge.org/

Weibo21 is a benchmark of fake news dataset for multi-domain fake news detection (MFND) with domain label annotated, which consists of 4,488 fake news and 4,640 real news from 9 different domains.

Fakeddit is a novel multimodal dataset for fake news detection consisting of over 1 million samples from multiple categories of fake news. After being processed through several stages of review, the samples are labeled according to 2-way, 3-way, and 6-way classification categories through distant supervision. Source: https://fakeddit.netlify.app/

For benchmarking, please refer to its variant UPFD-POL and UPFD-GOS. The dataset has been integrated with Pytorch Geometric (PyG) and Deep Graph Library (DGL). You can load the dataset after installing the latest versions of PyG or DGL. The UPFD dataset includes two sets of tree-structured graphs curated for evaluating binary graph classification, graph anomaly detection, and fake/real news detection tasks. The dataset is dumped in the form of Pytorch-Geometric dataset object. You can easily load the data and run various GNN models using PyG. The dataset includes fake&real news propagation (retweet) networks on Twitter built according to fact-check information from Politifact and Gossipcop. The news retweet graphs were originally extracted by FakeNewsNet. Each graph is a hierarchical tree-structured graph where the root node represents the news; the leaf nodes are Twitter users who retweeted the root news. A user node has an edge to the news node if he/she retweeted the news tweet. Two user nodes have an edge if one user retweeted the news tweet from the other user. We crawled near 20 million historical tweets from users who participated in fake news propagation in FakeNewsNet to generate node features in the dataset. We incorporate four node feature types in the dataset, the 768-dimensional bert and 300-dimensional spacy features are encoded using pretrained BERT and spaCy word2vec, respectively. The 10-dimensional profile feature is obtained from a Twitter account's profile. You can refer to profile_feature.py for profile feature extraction. The 310-dimensional content feature is composed of a 300-dimensional user comment word2vec (spaCy) embedding plus a 10-dimensional profile feature. The dataset statistics is shown below: | Data | #Graphs | #Fake News| #Total Nodes | #Total Edges | #Avg. Nodes per Graph | |-------|--------|--------|--------|--------|--------| | Politifact | 314 | 157 | 41,054 | 40,740 | 131 | | Gossipcop | 5464 | 2732 | 314,262 | 308,798 | 58 | Please refer to the paper for more details about the UPFD dataset. Due to the Twitter policy, we could not release the crawled user's historical tweets publicly. To get the corresponding Twitter user information, you can refer to the news lists under \data in our github repo and map the news id to FakeNewsNet. Then, you can crawl the user information by following the instruction on FakeNewsNet. In the UPFD project, we use Tweepy and Twitter Developer API to get the user information.

Along with COVID-19 pandemic we are also fighting an `infodemic'. Fake news and rumors are rampant on social media. Believing in rumors can cause significant harm. This is further exacerbated at the time of a pandemic. To tackle this, we curate and release a manually annotated dataset of 10,700 social media posts and articles of real and fake news on COVID-19. We benchmark the annotated dataset with four machine learning baselines - Decision Tree, Logistic Regression , Gradient Boost , and Support Vector Machine (SVM). We obtain the best performance of 93.46% F1-score with SVM.

MM-COVID is a dataset for fake news detection related to COVID-19. This dataset provides the multilingual fake news and the relevant social context. It contains 3,981 pieces of fake news content and 7,192 trustworthy information from English, Spanish, Portuguese, Hindi, French and Italian, 6 different languages.

For LIAR-RAW, we extended the public dataset LIAR-PLUS (Alhindi et al., 2018) with relevant raw reports, containing fine-grained claims from Politifact. LIAR-RAW is based on LIAR, where gold labels refer to Politifact. To alleviate the dependency of fact-checked reports, we extended the public LIAR dataset with additional raw reports for each claim. Besides, we put these raw reports into a single file with the format of LIAR. Provide: * A novel benchmark dataset for Explainable fake news detection.

NELA-GT-2018 is a dataset for the study of misinformation that consists of 713k articles collected between 02/2018-11/2018. These articles are collected directly from 194 news and media outlets including mainstream, hyper-partisan, and conspiracy sources. It includes ground truth ratings of the sources from 8 different assessment sites covering multiple dimensions of veracity, including reliability, bias, transparency, adherence to journalistic standards, and consumer trust.

An annotated dataset of ~50K news that can be used for building automated fake news detection systems for a low resource language like Bangla. Source: BanFakeNews: A Dataset for Detecting Fake News in Bangla

For RAWFC, we constructed it from scratch by collecting the claims from Snopes and relevant raw reports by retrieving claim keywords. To alleviate the dependency of fact-checked reports, RAWFC was constructed by using raw reports (from scratch), where gold labels refer to Snopes. Each instance in the train/val/test set is presented as a signle file. Provide: * A novel benchmark dataset for explainable fake news detection

MuMiN is a misinformation graph dataset containing rich social media data (tweets, replies, users, images, articles, hashtags), spanning 21 million tweets belonging to 26 thousand Twitter threads, each of which have been semantically linked to 13 thousand fact-checked claims across dozens of topics, events and domains, in 41 different languages, spanning more than a decade. MuMiN fills a gap in the existing misinformation datasets in multiple ways: - By having a large amount of social media information which have been semantically linked to fact-checked claims on an individual basis. - By featuring 41 languages, enabling evaluation of multilingual misinformation detection models. - By featuring both tweets, articles, images, social connections and hashtags, enabling multimodal approaches to misinformation detection. MuMiN features two node classification tasks, related to the veracity of a claim: - Claim classification: Determine the veracity of a claim, given its social network context. - Tweet classification: Determine the likelihood that a social media post to be fact-checked is discussing a misleading claim, given its social network context. To use the dataset, see the "Getting Started" guide and tutorial at the MuMiN website.

NELA-GT-2019 is an updated version of the NELA-GT-2018 dataset. NELA-GT-2019 contains 1.12M news articles from 260 sources collected between January 1st 2019 and December 31st 2019. Just as with NELA-GT-2018, these sources come from a wide range of mainstream news sources and alternative news sources. Included with the dataset are source-level ground truth labels from 7 different assessment sites covering multiple dimensions of veracity.

The LIAR dataset has been widely followed by fake news detection researchers since its release, and along with a great deal of research, the community has provided a variety of feedback on the dataset to improve it. We adopted these feedbacks and released the LIAR2 dataset, a new benchmark dataset of ~23k manually labeled by professional fact-checkers for fake news detection tasks. We have used a split ratio of 8:1:1 to distinguish between the training set, the test set, and the validation set, details of which are provided in the paper of "An Enhanced Fake News Detection System With Fuzzy Deep Learning". The LIAR2 dataset can be accessed at Huggingface and Github, and statistical information for LIAR and LIAR2 is provided in the table below: | Statistics | LIAR | LIAR2 | |----------------------------------------------|----------|------------| | Training set size | 10,269 | 18,369 | | Validation set size | 1,284 | 2,297 | | Testing set size | 1,283 | 2,296 | | Avg. statement length (tokens) | 17.9 | 17.7 | | Avg. speaker description length (tokens) | \ | 39.4 | | Avg. justification length (tokens) | \ | 94.4 | | Labels | Pants on fire | 1,050 | 3,031 | | False | 2,511 | 6,605 | | Barely-true | 2,108 | 3,603 | | Half-true | 2,638 | 3,709 | | Mostly-true | 2,466 | 3,429 | | True | 2,063 | 2,585 | ## Ablation Experiment The LIAR2 dataset is an upgrade of the LIAR dataset, which inherits the ideas of the LIAR dataset, refines the details and architecture, and expands the size of the dataset to make it more responsive to the needs of fake news detection tasks. We believe that with the help of the LIAR2 dataset, it will be able to perform better fake news detection tasks. The analysis and baseline information about the LIAR2 dataset is provided in below. | Feature | Val. Accuracy | Val. F1-Macro | Val. F1-Micro | Test Accuracy | Test F1-Macro | Test F1-Micro | Mean | |--------------------------|-------------------|-------------------|-------------------|-------------------|-------------------|-------------------|----------| | Statement | 0.3174 | 0.1957 | 0.3117 | 0.3197 | 0.2380 | 0.3197 | 0.2837 | | Date | 0.2912 | 0.1879 | 0.2912 | 0.3079 | 0.1775 | 0.3079 | 0.2606 | | Subject | 0.3243 | 0.2311 | 0.3183 | 0.3267 | 0.2271 | 0.3267 | 0.2924 | | Speaker | 0.3283 | 0.2250 | 0.3174 | 0.3310 | 0.2462 | 0.3310 | 0.2965 | | Speaker Description | 0.3322 | 0.2444 | 0.3250 | 0.3280 | 0.2444 | 0.3280 | 0.3003 | | State Info | 0.2930 | 0.1577 | 0.2950 | 0.2979 | 0.1521 | 0.2979 | 0.2489 | | Credibility History | 0.5007 | 0.4696 | 0.4985 | 0.5057 | 0.4656 | 0.5057 | 0.4910 | | Context | 0.2982 | 0.1817 | 0.2982 | 0.3132 | 0.1791 | 0.3132 | 0.2639 | | Justification | 0.5964 | 0.5657 | 0.5827 | 0.6115 | 0.5968 | 0.6115 | 0.5941 | | All without |Statement | 0.7079 | 0.6734 | 0.6822 | 0.7182 | 0.7108 | 0.7182 | 0.7018 | | Date | 0.6931 | 0.6572 | 0.6680 | 0.7078 | 0.6993 | 0.7078 | 0.6889 | | Subject | 0.7000 | 0.6579 | 0.6681 | 0.7078 | 0.7013 | 0.7078 | 0.6905 | | Speaker | 0.6944 | 0.6648 | 0.6757 | 0.7043 | 0.6942 | 0.7043 | 0.6896 | | Speaker Description | 0.6892 | 0.6640 | 0.6739 | 0.7169 | 0.7073 | 0.7169 | 0.6947 | | State Info | 0.7074 | 0.6625 | 0.6729 | 0.7099 | 0.7016 | 0.7099 | 0.6940 | | Credibility History | 0.6025 | 0.5717 | 0.5900 | 0.6185 | 0.6046 | 0.6185 | 0.6010 | | Context | 0.7005 | 0.6622 | 0.6720 | 0.7043 | 0.6967 | 0.7043 | 0.6900 | | Justification | 0.5285 | 0.4898 | 0.5153 | 0.5340 | 0.5148 | 0.5340 | 0.5194 | | Statement + | Date | 0.3431 | 0.2540 | 0.3343 | 0.3380 | 0.2514 | 0.3380 | 0.3098 | | Subject | 0.3548 | 0.2759 | 0.3513 | 0.3375 | 0.2580 | 0.3375 | 0.3192 | | Speaker | 0.3618 | 0.2862 | 0.3539 | 0.3476 | 0.2640 | 0.3476 | 0.3269 | | Speaker Description | 0.3583 | 0.2814 | 0.3531 | 0.3667 | 0.2886 | 0.3667 | 0.3358 | | State Info | 0.3317 | 0.2367 | 0.3294 | 0.3328 | 0.2362 | 0.3328 | 0.2999 | | Credibility History | 0.5067 | 0.4737 | 0.5084 | 0.5244 | 0.5000 | 0.5244 | 0.5063 | | Context | 0.3361 | 0.2682 | 0.3391 | 0.3458 | 0.2560 | 0.3458 | 0.3152 | | Justification | 0.6017 | 0.5578 | 0.5796 | 0.6176 | 0.6026 | 0.6176 | 0.5962 | | All | 0.6974 | 0.6570 | 0.6676 | 0.7021 | 0.6961 | 0.7021 | 0.6871|

NELA-GT-2020 is an updated version of the NELA-GT-2019 dataset. NELA-GT-2020 contains nearly 1.8M news articles from 519 sources collected between January 1st, 2020 and December 31st, 2020. Just as with NELA-GT-2018 and NELA-GT-2019, these sources come from a wide range of mainstream news sources and alternative news sources. Included in the dataset are source-level ground truth labels from Media Bias/Fact Check (MBFC) covering multiple dimensions of veracity. Additionally, new in the 2020 dataset are the Tweets embedded in the collected news articles, adding an extra layer of information to the data.

Some Like it Hoax is a fake news detection dataset consisting of 15,500 Facebook posts and 909,236 users. Source: Some Like it Hoax: Automated Fake News Detection in Social Networks

The Gossipcop variant of the UPFD dataset for benchmarking. Please refer to the UPFD dataset for more details of the data.

AraCOVID19-MFH is a manually annotated multi-label Arabic COVID-19 fake news and hate speech detection dataset. The dataset contains 10,828 Arabic tweets annotated with 10 different labels.

A Dataset to Identify Manipulated Social Media News in Bangla We construct a publicly available Bangla dataset of 800 news-related social media items that are annotated as manipulated or not relative to 500 reference news articles. We present a semi-automatic (use both human and LLM) method for generating such a dataset, which allows scalable dataset collection using annotators efficiently for languages with few available NLP tools.

Search Engine Optimization (SEO) attributes provide strong signals for predicting news site reliability. We introduce a novel attributed webgraph dataset with labeled news domains and their connections to outlinking and backlinking domains. Finally, we introduce and evaluate a novel graph-based algorithm for discovering previously unknown misinformation news sources. This dataset is provided courtesy of Ahrefs.com.

The PolitiFact variant of the UPFD dataset for benchmarking. Please refer to the UPFD dataset for more details of the data.

Expertly-curated benchmark dataset for fake news detection in Filipino. Source: Localization of Fake News Detection via Multitask Transfer Learning

A novel corpus of 15,356 Polish web articles, including articles identified as containing disinformation. Our dataset enables a multifaceted understanding of disinformation. We present a distinctive multilayered methodology for annotating disinformation in texts. What sets our corpus apart is its focus on uncovering hidden intent and manipulation in disinformative content. A team of experts annotated each article with multiple labels indicating both disinformation creators’ intents and the manipulation techniques employed.

The task addresses the problem of the appearance and propagation of posts that share misleading multimedia content (images or video). In the context of the task, different types of misleading use are considered: 1. Reposting of real multimedia, such as real photos from the past re-posted as being associated to a current event, 2. Digitally manipulated multimedia, 3. Synthetic multimedia, such as artworks or snapshots presented as real imagery. Source: http://www.multimediaeval.org/mediaeval2016/verifyingmultimediause/

The CIDII dataset is a binary classification, consisting of two classes of correct information and disinformation related to Islamic issues. The CIDII dataset belongs to our research (DISINFORMATION DETECTION ABOUT ISLAMIC ISSUES ON SOCIAL MEDIA USING DEEP LEARNING TECHNIQUES) published in MJCS journal in the link below: https://ejournal.um.edu.my/index.php/MJCS/article/view/41935 This dataset consists of five columns: 1- ID: Each article has a unique ID. 2- Article: The article contains text that is either facts related to Islamic issues if the information is correct, or posts targeting the Islamic religion if the information is false. Most posts contain only the body without a title. 3- Propagation Source: The source refers to the source of the article content, as it contains a Facebook link in the event that the post is disinformation, or it contains a link to Islamic websites in the event that the article refers to correct information (an explanation of a verse, a hadith, or an article related to the Islamic religion). 4- Article Type: This column contains the type of article published. Is it a post if the article is disinformation, or is it an Islamic article, a Quranic interpretation, or a hadith, in case of that the information is correct? 5- Class Type: This column shows whether the article belongs to the category of correct information or disinformation.