Papers with Code · CC BY-SA 4.0
데이터셋 탐색
데이터셋의 설명·홈페이지를 찾고, 관련 태스크의 평가 기록을 살펴보세요.
데이터셋의 설명·홈페이지를 찾고, 관련 태스크의 평가 기록을 살펴보세요.
.tsv file with the following structure: | premise | hypothesis | label | |:------------|:---------------------------------------------------|:-----:| | A sentence. | An equivalent, non-negated sentence (paraphrased). | 0 | | A sentence. | The sentence negated. | 1 | The dataset can be easily loaded into a Pandas DataFrame by running: Python import pandas as pd dataset = pd.read_csv('negation_dataset_v1.0.tsv', sep='\t') <br> ## Dataset Creation The dataset has been created by cleaning up and merging the following datasets: 1. Not another Negation Benchmark: The NaN-NLI Test Suite for Sub-clausal Negation (see datasets/nan-nli). 2. GLUE Diagnostic Dataset (see datasets/glue-diagnostic). 3. Automated Fact-Checking of Claims from Wikipedia (see datasets/wikifactcheck-english). 4. From Group to Individual Labels Using Deep Features (see datasets/sentiment-labelled-sentences). In this case, the negated sentences were obtained by using the Python module negate. 5. It Is Not Easy To Detect Paraphrases: Analysing Semantic Similarity With Antonyms and Negation Using the New SemAntoNeg Benchmark (see datasets/antonym-substitution). Once processed, the number of remaining samples in each of the datasets above are: | Dataset | Samples | |:--------------------------------------------------------------------------|-----------:| | Not another Negation Benchmark | 118 | | GLUE Diagnostic Dataset | 154 | | Automated Fact-Checking of Claims from Wikipedia | 14,970 | | From Group to Individual Labels Using Deep Features | 2,110 | | It Is Not Easy To Detect Paraphrases | 8,597 | | <div align="right"><b>Total</b></div> | 25,949 | Additionally, for each of the negated samples, another pair of non-negated sentences has been added by paraphrasing them with the pre-trained model 🤗tuner007/pegasus_paraphrase. Finally, the swapped version of each pair (premise ⇋ hypothesis) has also been included, and any duplicates have been removed. With this, the number of premises/hypothesis in the CANNOT dataset that appear in the original datasets are: | <div align="left"><b>Dataset</b></div> | <div align="center"><b>Sentences</b></div> | |:--------------------------------------------------------------------------|----------------------:| | Not another Negation Benchmark | 552 (0.36 %) | | GLUE Diagnostic Dataset | 586 (0.38 %) | | Automated Fact-Checking of Claims from Wikipedia | 89,728 (59.98 %) | | From Group to Individual Labels Using Deep Features | 12,626 (8.16 %) | | It Is Not Easy To Detect Paraphrases | 17,198 (11.11 %) | | <div align="right"><b>Total</b></div> | 120,690 (77.99 %) | The percentages above are in relation to the total number of premises and hypothesis in the CANNOT dataset. The remaining 22.01 % (34,062 sentences) are the novel premises/hypothesis added through paraphrase and rule-based negation. <br> ## Additional Information <br> ### Licensing Information The CANNOT dataset is released under CC BY-SA 4.0. <a rel="license" href="http://creativecommons.org/licenses/by-sa/4.0/"> <img alt="Creative Commons License" width="100px" src="https://i.creativecommons.org/l/by-sa/4.0/88x31.png"/> </a> <br> ### Citation Please cite our INLG 2023 paper, if you use our dataset. BibTeX: bibtex @misc{anschütz2023correct, title={This is not correct! Negation-aware Evaluation of Language Generation Systems}, author={Miriam Anschütz and Diego Miguel Lozano and Georg Groh}, year={2023}, eprint={2307.13989}, archivePrefix={arXiv}, primaryClass={cs.CL} } <br> ### Contributions Contributions to the dataset can be submitted through the project repository.