Jericho Environment Commonsense Comprehension (JECC) is a dataset for commonsense reasoning. It consists of 29 games in multiple domains from the Jericho Environment hausknecht2019interactive. Source: JECC: Commonsense Reasoning Tasks Derived from Interactive Fictions Image Source: https://arxiv.org/pdf/2210.15456v1.pdf

JustLogic is a natural language deductive reasoning dataset. JustLogic is (i) highly complex, capable of generating a diverse range of linguistic patterns, vocabulary, and argument structures; (ii) prior knowledge independent, eliminating the advantage of models possessing prior knowledge and ensuring that only deductive reasoning is used to answer questions; and (iii) capable of in-depth error analysis on the heterogeneous effects of reasoning depth and argument form on model accuracy.

A large-scale machine comprehension dataset (based on the COCO images and captions). Source: Understanding Image and Text Simultaneously: a Dual Vision-Language Machine Comprehension Task

NEREL-BIO is an annotation scheme and corpus of PubMed abstracts in Russian and English. It contains annotations for 700+ Russian and 100+ English abstracts. All English PubMed annotations have corresponding Russian counterparts. NEREL-BIO comprises the following specific features: annotation of nested named entities, it can be used as a benchmark for cross-domain (NEREL -> NEREL-BIO) and cross-language (English -> Russian) transfer. Source: NEREL-BIO: A Dataset of Biomedical Abstracts Annotated with Nested Named Entities Image Source: https://arxiv.org/pdf/2210.11913v1.pdf

PersianQA: a dataset for Persian Question Answering Persian Question Answering (PersianQA) Dataset is a reading comprehension dataset on Persian Wikipedia. The crowd-sourced the dataset consists of more than 9,000 entries. Each entry can be either an impossible-to-answer or a question with one or more answers spanning in the passage (the context) from which the questioner proposed the question. Much like the SQuAD2.0 dataset, the impossible or unanswerable questions can be utilized to create a system which "knows that it doesn't know the answer". Moreover, the dataset has 900 test data available. On top of that, the very first models trained on the dataset, Transformers, are available online. All the crowd workers of the dataset are native Persian speakers. Also, it worth mentioning that the contexts are collected from all categories of the Wiki (Historical, Religious, Geography, Science, etc).

A high-quality large-scale dataset consisting of 49,000+ data samples for the task of Chinese query-based document summarization. Source: QBSUM: a Large-Scale Query-Based Document Summarization Dataset from Real-world Applications

To collect WikiSuggest, Google Suggest API is used to harvest natural language questions and submit them to Google Search. Whenever Google Search returns a box with a short answer from Wikipedia, an example from the question, answer, and the Wikipedia document are created. If the answer string is missing from the document this often implies a spurious question-answer pair, such as (‘what time is half time in rugby’, ‘80 minutes, 40 minutes’). Question-answer pairs without the exact answer string are pruned. Fifty examples after filtering are examined and 54% were found to be well-formed question-answer pairs where answers in the document can be grounded, 20% contained answers without textual evidence in the document (the answer string exists in an irreleveant context), and 26% contain incorrect QA pairs. Source: Coarse-to-Fine Question Answering for Long Documents

IndirectRequests is an LLM-generated dataset of user utterances in a task-oriented dialogue setting where the user does not directly specify their preferred slot value. IndirectRequests was generated by crowdsourcing human labels over a dataset generated using a combination of GPT-3.5 (turbo) and GPT-4. Each utterance is labelled along two dimensions: - World Understanding (the degree of world understanding it takes to understand the utterance) - Unambiguity (whether or not the generated utterance unambiguously entails a single target slot value among a set of candidate possible values).