Logical Reasoning Reading Comprehension

A Winograd schema is a pair of sentences that differ in only one or two words and that contain an ambiguity that is resolved in opposite ways in the two sentences and requires the use of world knowledge and reasoning for its resolution. The schema takes its name from a well-known example by Terry Winograd. The set would then be presented as a challenge for AI programs, along the lines of the Turing test. The strengths of the challenge are that it is clear-cut, in that the answer to each schema is a binary choice; vivid, in that it is obvious to non-experts that a program that fails to get the right answers clearly has serious gaps in its understanding; and difficult, in that it is far beyond the current state of the art. ### Task Type Logic and Reasoning, World knowledge. Binary Classification: true/false ### Example { "text": "Кубок не помещается в коричневый чемодан, потому что он слишком большой." "label": false, "idx": 5, "target": { "span1_text": "чемодан", "span2_text": "он слишком большой", "span1_index": 5, "span2_index": 8 }, } ### How did we collect data? All text examples were collected manually translating and adapting original Winograd dataset for Russian. Human assessment was carried out on Yandex.Toloka.

JustLogic is a natural language deductive reasoning dataset. JustLogic is (i) highly complex, capable of generating a diverse range of linguistic patterns, vocabulary, and argument structures; (ii) prior knowledge independent, eliminating the advantage of models possessing prior knowledge and ensuring that only deductive reasoning is used to answer questions; and (iii) capable of in-depth error analysis on the heterogeneous effects of reasoning depth and argument form on model accuracy.