Fact Checking

The Beyond the Imitation Game Benchmark (BIG-bench) is a collaborative benchmark intended to probe large language models and extrapolate their future capabilities. Big-bench include more than 200 tasks. Image source: https://arxiv.org/pdf/2206.04615.pdf

BEIR (Benchmarking IR) is a heterogeneous benchmark containing different information retrieval (IR) tasks. Through BEIR, it is possible to systematically study the zero-shot generalization capabilities of multiple neural retrieval approaches. The benchmark contains a total of 9 information retrieval tasks (Fact Checking, Citation Prediction, Duplicate Question Retrieval, Argument Retrieval, News Retrieval, Question Answering, Tweet Retrieval, Biomedical IR, Entity Retrieval) from 19 different datasets: * MS MARCO * TREC-COVID * NFCorpus * BioASQ * Natural Questions * HotpotQA * FiQA-2018 * Signal-1M * TREC-News * ArguAna * Touche 2020 * CQADupStack * Quora Question Pairs * DBPedia * SciDocs * FEVER * Climate-FEVER * SciFact * Robust04

The VitaminC dataset contains more than 450,000 claim-evidence pairs for fact verification and factual consistent generation. Based on over 100,000 revisions to popular Wikipedia pages, and additional "synthetic" revisions.

AVeriTeC (Automated Verification of Textual Claims) is a dataset of 4568 real-world claims covering fact-checks by 50 different organizations. Each claim is annotated with question-answer pairs supported by evidence available online, as well as textual justifications explaining how the evidence combines to produce a verdict. The Claims in AVeriTeC are classified into four labels: "Supported", "Refuted", "Not Enough Evidence", "Conflicting Evidence/Cherry-picking". The dataset also contains several fields of metadata such as the speaker of the claim, the publisher of the claim, the date the claim was published, and the location most relevant to the claim. These can be used to support questions, answers, and justifications.

A large-scale dataset that consists of 21,184 claims, where each claim is assigned a truthfulness label and ruling statement, with 58,523 pieces of evidence in the form of text and images. It supports the end-to-end multimodal fact-checking and explanation generation, where the input is a claim and a large collection of web sources, including articles, images, videos, and tweets, and the goal is to assess the truthfulness of the claim by retrieving relevant evidence and predicting a truthfulness label (i.e., support, refute and not enough information), and generate a rationalization statement to explain the reasoning and ruling process.

ataset format Each row in the dataset splits represents one instance and contains the following tab-separated columns: article_id - article id corresponding to the id of the claim in the LIAR dataset statement - the text of the claim author - the author of the claim ruling - a comma-separated list of the sentences in the Politifact ruling report (this excludes sentences from the summary in the end) url_sentences - a comma-separated list of ids of the sentences with a corresponding source url(s) relevant_text_url_sentences - a comma-separated list of ids of the sentences with a corresponding source url(s) that are actually relevant to the selected evidence sentences politifact_label - label assigned to the claim by PolitiFact fact-checkiers annotated_evidence - a json dict of the evidence chains (keys) and the sentences that belong to the chain (value, which is a list of sentence ids from the ruling) annotated_label - label annotated by annotators of PolitiFact - True, False, Half-True urls - a comma-separated list of source urls used in the corresponding PolitiFact article annotated_urls - a json dict mapping sentence ids to the corresponding urls ids. One sentence can have multiple urls

CoVERT is a fact-checked corpus of tweets with a focus on the domain of biomedicine and COVID-19-related (mis)information. The corpus consists of 300 tweets, each annotated with medical named entities and relations. Employs a novel crowdsourcing methodology to annotate all tweets with fact-checking labels and supporting evidence, which crowdworkers search for online. This methodology results in moderate inter-annotator agreement.

The LIAR dataset has been widely followed by fake news detection researchers since its release, and along with a great deal of research, the community has provided a variety of feedback on the dataset to improve it. We adopted these feedbacks and released the LIAR2 dataset, a new benchmark dataset of ~23k manually labeled by professional fact-checkers for fake news detection tasks. We have used a split ratio of 8:1:1 to distinguish between the training set, the test set, and the validation set, details of which are provided in the paper of "An Enhanced Fake News Detection System With Fuzzy Deep Learning". The LIAR2 dataset can be accessed at Huggingface and Github, and statistical information for LIAR and LIAR2 is provided in the table below: | Statistics | LIAR | LIAR2 | |----------------------------------------------|----------|------------| | Training set size | 10,269 | 18,369 | | Validation set size | 1,284 | 2,297 | | Testing set size | 1,283 | 2,296 | | Avg. statement length (tokens) | 17.9 | 17.7 | | Avg. speaker description length (tokens) | \ | 39.4 | | Avg. justification length (tokens) | \ | 94.4 | | Labels | Pants on fire | 1,050 | 3,031 | | False | 2,511 | 6,605 | | Barely-true | 2,108 | 3,603 | | Half-true | 2,638 | 3,709 | | Mostly-true | 2,466 | 3,429 | | True | 2,063 | 2,585 | ## Ablation Experiment The LIAR2 dataset is an upgrade of the LIAR dataset, which inherits the ideas of the LIAR dataset, refines the details and architecture, and expands the size of the dataset to make it more responsive to the needs of fake news detection tasks. We believe that with the help of the LIAR2 dataset, it will be able to perform better fake news detection tasks. The analysis and baseline information about the LIAR2 dataset is provided in below. | Feature | Val. Accuracy | Val. F1-Macro | Val. F1-Micro | Test Accuracy | Test F1-Macro | Test F1-Micro | Mean | |--------------------------|-------------------|-------------------|-------------------|-------------------|-------------------|-------------------|----------| | Statement | 0.3174 | 0.1957 | 0.3117 | 0.3197 | 0.2380 | 0.3197 | 0.2837 | | Date | 0.2912 | 0.1879 | 0.2912 | 0.3079 | 0.1775 | 0.3079 | 0.2606 | | Subject | 0.3243 | 0.2311 | 0.3183 | 0.3267 | 0.2271 | 0.3267 | 0.2924 | | Speaker | 0.3283 | 0.2250 | 0.3174 | 0.3310 | 0.2462 | 0.3310 | 0.2965 | | Speaker Description | 0.3322 | 0.2444 | 0.3250 | 0.3280 | 0.2444 | 0.3280 | 0.3003 | | State Info | 0.2930 | 0.1577 | 0.2950 | 0.2979 | 0.1521 | 0.2979 | 0.2489 | | Credibility History | 0.5007 | 0.4696 | 0.4985 | 0.5057 | 0.4656 | 0.5057 | 0.4910 | | Context | 0.2982 | 0.1817 | 0.2982 | 0.3132 | 0.1791 | 0.3132 | 0.2639 | | Justification | 0.5964 | 0.5657 | 0.5827 | 0.6115 | 0.5968 | 0.6115 | 0.5941 | | All without |Statement | 0.7079 | 0.6734 | 0.6822 | 0.7182 | 0.7108 | 0.7182 | 0.7018 | | Date | 0.6931 | 0.6572 | 0.6680 | 0.7078 | 0.6993 | 0.7078 | 0.6889 | | Subject | 0.7000 | 0.6579 | 0.6681 | 0.7078 | 0.7013 | 0.7078 | 0.6905 | | Speaker | 0.6944 | 0.6648 | 0.6757 | 0.7043 | 0.6942 | 0.7043 | 0.6896 | | Speaker Description | 0.6892 | 0.6640 | 0.6739 | 0.7169 | 0.7073 | 0.7169 | 0.6947 | | State Info | 0.7074 | 0.6625 | 0.6729 | 0.7099 | 0.7016 | 0.7099 | 0.6940 | | Credibility History | 0.6025 | 0.5717 | 0.5900 | 0.6185 | 0.6046 | 0.6185 | 0.6010 | | Context | 0.7005 | 0.6622 | 0.6720 | 0.7043 | 0.6967 | 0.7043 | 0.6900 | | Justification | 0.5285 | 0.4898 | 0.5153 | 0.5340 | 0.5148 | 0.5340 | 0.5194 | | Statement + | Date | 0.3431 | 0.2540 | 0.3343 | 0.3380 | 0.2514 | 0.3380 | 0.3098 | | Subject | 0.3548 | 0.2759 | 0.3513 | 0.3375 | 0.2580 | 0.3375 | 0.3192 | | Speaker | 0.3618 | 0.2862 | 0.3539 | 0.3476 | 0.2640 | 0.3476 | 0.3269 | | Speaker Description | 0.3583 | 0.2814 | 0.3531 | 0.3667 | 0.2886 | 0.3667 | 0.3358 | | State Info | 0.3317 | 0.2367 | 0.3294 | 0.3328 | 0.2362 | 0.3328 | 0.2999 | | Credibility History | 0.5067 | 0.4737 | 0.5084 | 0.5244 | 0.5000 | 0.5244 | 0.5063 | | Context | 0.3361 | 0.2682 | 0.3391 | 0.3458 | 0.2560 | 0.3458 | 0.3152 | | Justification | 0.6017 | 0.5578 | 0.5796 | 0.6176 | 0.6026 | 0.6176 | 0.5962 | | All | 0.6974 | 0.6570 | 0.6676 | 0.7021 | 0.6961 | 0.7021 | 0.6871|

Stanceosaurus is a corpus of 28,033 tweets in English, Hindi, and Arabic annotated with stance towards 251 misinformation claims. The claims in Stanceosaurus originate from 15 fact-checking sources that cover diverse geographical regions and cultures. Unlike existing stance datasets, it introduces a more fine-grained 5-class labeling strategy with additional subcategories to distinguish implicit stance. Source: Stanceosaurus: Classifying Stance Towards Multilingual Misinformation Image Source: https://arxiv.aorg/pdf/2210.15954v1.pdf

• CFEVER is a Chinese Fact Extraction and VERification dataset published at AAAI 2024. - CFEVER comprises 30,012 manually created claims based on content in Chinese Wikipedia. - Inspired by the FEVER dataset (Thorne et al., 2018), we provide class labels (Supports, Refutes, or Not Enough Information) and evidence for each claim in the CFEVER dataset. - Download the dataset: https://github.com/IKMLab/CFEVER-data

Spiced is a paraphrase dataset of scientific findings annotated for degree of information change. Spiced contains 6,000 scientific finding pairs extracted from news stories, social media discussions, and full texts of original papers. Source: Modeling Information Change in Science Communication with Semantically Matched Paraphrases Image Source: https://arxiv.org/pdf/2210.13001v1.pdf

The peer-reviewed publication for this dataset has been presented in the 2022 Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), and can be accessed here: https://arxiv.org/abs/2205.02596. Please cite this when using the dataset. This dataset contains a heterogeneous set of True and False COVID claims and online sources of information for each claim. The claims have been obtained from online fact-checking sources, existing datasets and research challenges. It combines different data sources with different foci, thus enabling a comprehensive approach that combines different media (Twitter, Facebook, general websites, academia), information domains (health, scholar, media), information types (news, claims) and applications (information retrieval, veracity evaluation). The processing of the claims included an extensive de-duplication process eliminating repeated or very similar claims. The dataset is presented in a LARGE and a SMALL version, accounting for different degrees of similarity between the remaining claims (excluding respectively claims with a 90% and 99% probability of being similar, as obtained through the MonoT5 model). The similarity of claims was analysed using BM25 (Robertson et al., 1995; Crestani et al., 1998; Robertson and Zaragoza, 2009) with MonoT5 re-ranking (Nogueira et al., 2020), and BERTScore (Zhang et al., 2019). The processing of the content also involved removing claims making only a direct reference to existing content in other media (audio, video, photos); automatically obtained content not representing claims; and entries with claims or fact-checking sources in languages other than English. The claims were analysed to identify types of claims that may be of particular interest, either for inclusion or exclusion depending on the type of analysis. The following types were identified: (1) Multimodal; (2) Social media references; (3) Claims including questions; (4) Claims including numerical content; (5) Named entities, including: PERSON − People, including fictional; ORGANIZATION − Companies, agencies, institutions, etc.; GPE − Countries, cities, states; FACILITY − Buildings, highways, etc. These entities have been detected using a RoBERTa base English model (Liu et al., 2019) trained on the OntoNotes Release 5.0 dataset (Weischedel et al., 2013) using Spacy. The original labels for the claims have been reviewed and homogenised from the different criteria used by each original fact-checker into the final True and False labels. The data sources used are: - The CoronaVirusFacts/DatosCoronaVirus Alliance Database. https://www.poynter.org/ifcn-covid-19-misinformation/ - CoAID dataset (Cui and Lee, 2020) https://github.com/cuilimeng/CoAID - MM-COVID (Li et al., 2020) https://github.com/bigheiniu/MM-COVID - CovidLies (Hossain et al., 2020) https://github.com/ucinlp/covid19-data - TREC Health Misinformation track https://trec-health-misinfo.github.io/ - TREC COVID challenge (Voorhees et al., 2021; Roberts et al., 2020) https://ir.nist.gov/covidSubmit/data.html The LARGE dataset contains 5,143 claims (1,810 False and 3,333 True), and the SMALL version 1,709 claims (477 False and 1,232 True). The entries in the dataset contain the following information: - Claim. Text of the claim. - Claim label. The labels are: False, and True. - Claim source. The sources include mostly fact-checking websites, health information websites, health clinics, public institutions sites, and peer-reviewed scientific journals. - Original information source. Information about which general information source was used to obtain the claim. - Claim type. The different types, previously explained, are: Multimodal, Social Media, Questions, Numerical, and Named Entities. Funding. This work was supported by the UK Engineering and Physical Sciences Research Council (grant no. EP/V048597/1, EP/T017112/1). ML and YH are supported by Turing AI Fellowships funded by the UK Research and Innovation (grant no. EP/V030302/1, EP/V020579/1). References - Arana-Catania M., Kochkina E., Zubiaga A., Liakata M., Procter R., He Y.. Natural Language Inference with Self-Attention for Veracity Assessment of Pandemic Claims. NAACL 2022 https://arxiv.org/abs/2205.02596 - Stephen E Robertson, Steve Walker, Susan Jones, Micheline M Hancock-Beaulieu, Mike Gatford, et al. 1995. Okapi at trec-3. Nist Special Publication Sp,109:109. - Fabio Crestani, Mounia Lalmas, Cornelis J Van Rijsbergen, and Iain Campbell. 1998. “is this document relevant?. . . probably” a survey of probabilistic models in information retrieval. ACM Computing Surveys (CSUR), 30(4):528–552. - Stephen Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: BM25 and beyond. Now Publishers Inc. - Rodrigo Nogueira, Zhiying Jiang, Ronak Pradeep, and Jimmy Lin. 2020. Document ranking with a pre-trained sequence-to-sequence model. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pages 708–718. - Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations. - Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. - Ralph Weischedel, Martha Palmer, Mitchell Marcus, Eduard Hovy, Sameer Pradhan, Lance Ramshaw, Nianwen Xue, Ann Taylor, Jeff Kaufman, Michelle Franchini, et al. 2013. Ontonotes release 5.0 ldc2013t19. Linguistic Data Consortium, Philadelphia, PA, 23. - Limeng Cui and Dongwon Lee. 2020. Coaid: Covid-19 healthcare misinformation dataset. arXiv preprint arXiv:2006.00885. - Yichuan Li, Bohan Jiang, Kai Shu, and Huan Liu. 2020. Mm-covid: A multilingual and multimodal data repository for combating covid-19 disinformation. - Tamanna Hossain, Robert L. Logan IV, Arjuna Ugarte, Yoshitomo Matsubara, Sean Young, and Sameer Singh. 2020. COVIDLies: Detecting COVID-19 misinformation on social media. In Proceedings of the 1st Workshop on NLP for COVID-19 (Part 2) at EMNLP 2020, Online. Association for Computational Linguistics. - Ellen Voorhees, Tasmeer Alam, Steven Bedrick, Dina Demner-Fushman, William R Hersh, Kyle Lo, Kirk Roberts, Ian Soboroff, and Lucy Lu Wang. 2021. Trec-covid: constructing a pandemic information retrieval test collection. In ACM SIGIR Forum, volume 54, pages 1–12. ACM New York, NY, USA.

STVD-FC is the largest public dataset on the political content analysis and fact-checking tasks. It consists of more than 1,200 fact-checked claims that have been scraped from a fact-checking service with associated metadata. For the video counterpart, the dataset contains nearly 6,730 TV programs, having a total duration of 6,540 hours, with metadata. These programs have been collected during the 2022 French presidential election with a dedicated workstation and protocol. The dataset is delivered as different parts for accessibility of the 2 TB of data and proper indexes. More information about the STVD-FC dataset can be found into the publication [1]. [1] F. Rayar, M. Delalandre and V.H. Le. A large-scale TV video and metadata database for French political content analysis and fact-checking. Conference on Content-Based Multimedia Indexing (CBMI), pp. 181–185, 2022. Source: http://mathieu.delalandre.free.fr/publications/CBMI2022.pdf