Papers with Code 아카이브

Common Sense Reasoning

태스크의 하위 분류와 논문·코드·평가·기법·데이터셋을 탐색하세요.

하위 태스크
17
평가표
24
보존된 평가 기록
397

Common sense reasoning tasks are intended to require the model to go beyond pattern recognition. Instead, the model should use "common sense" or world knowledge to make inferences.

데이터셋

출처에 같은 태스크 이름으로 연결된 데이터셋입니다. 동명 태스크가 포함될 수 있으므로, 비교할 때는 평가 결과의 조건도 확인하세요.

SMART-101

Recent times have witnessed an increasing number of applications of deep neural networks towards solving tasks that require superior cognitive abilities, e.g., playing Go, generating art, ChatGPT, etc. Such a dramatic progress raises the question: how generalizable are neural networks in solving problems that demand broad skills? To answer this question, we propose SMART: a Simple Multimodal Algorithmic Reasoning Task (and the associated SMART-101 dataset) for evaluating the abstraction, deduction, and generalization abilities of neural networks in solving visuo-linguistic puzzles designed specifically for children of younger age (6--8). Our dataset consists of 101 unique puzzles; each puzzle comprises a picture and a question, and their solution needs a mix of several elementary skills, including pattern recognition, algebra, and spatial reasoning, among others. To train deep neural networks, we programmatically augment each puzzle to 2,000 new instances; each instance varied in appearance, associated natural language question, and its solution. To foster research and make progress in the quest for artificial general intelligence, we are publicly releasing our SMART-101 dataset, consisting of the full set of programmatically-generated instances of 101 puzzles and their solutions. The dataset was introduced in our paper Are Deep Neural Networks SMARTer than Second Graders? by Anoop Cherian, Kuan-Chuan Peng, Suhas Lohit, Kevin A. Smith, and Joshua B. Tenenbaum, CVPR 2023

NumerSense

Contains 13.6k masked-word-prediction probes, 10.5k for fine-tuning and 3.1k for testing. Source: Birds have four legs?! NumerSense: Probing Numerical Commonsense Knowledge of Pre-trained Language Models

RuCoS

Russian reading comprehension with Commonsense reasoning (RuCoS) is a large-scale reading comprehension dataset that requires commonsense reasoning. RuCoS consists of queries automatically generated from CNN/Daily Mail news articles; the answer to each query is a text span from a summarizing passage of the corresponding news. The goal of RuCoS is to evaluate a machine`s ability of commonsense reasoning in reading comprehension. Example {'source': 'Lenta', 'passage': { 'text': 'Мать двух мальчиков, брошенных отцом в московском аэропорту Шереметьево, забрала их. Об этом сообщили ТАСС в пресс-службе министерства образования и науки Хабаровского края. Сейчас младший ребенок посещает детский сад, а старший ходит в школу. В учебных заведениях с ними по необходимости работают штатные психологи. Также министерство социальной защиты населения рассматривает вопрос о бесплатном оздоровлении детей в летнее время. Через несколько дней после того, как Виктор Гаврилов бросил своих детей в аэропорту, он явился с повинной к следователям в городе Батайске Ростовской области.\n@context\nБросившего детей в Шереметьево отца задержали за насилие над женой\n@context\nРоссиянина заподозрили в истязании брошенных в Шереметьево детей\n@context\nОставивший двоих детей в Шереметьево россиянин сам пришел к следователям', 'entities': [ {'start': 60, 'end': 71, 'text': 'Шереметьево'}, {'start': 102, 'end': 106, 'text': 'ТАСС'}, {'start': 155, 'end': 172, 'text': 'Хабаровского края'}, {'start': 470, 'end': 485, 'text': 'Виктор Гаврилов'}, {'start': 563, 'end': 571, 'text': 'Батайске'}, {'start': 572, 'end': 590, 'text': 'Ростовской области'}, {'start': 620, 'end': 631, 'text': 'Шереметьево'}, {'start': 725, 'end': 736, 'text': 'Шереметьево'}, {'start': 778, 'end': 789, 'text': 'Шереметьево'} ] }, 'qas': [ { 'query': '26 января @placeholder бросил сыновей в возрасте пяти и семи лет в Шереметьево.', 'answers': [ {'start': 470, 'end': 485, 'text': 'Виктор Гаврилов'} ], 'idx': 0 } ], 'idx': 0 } ### How did we collect data? All text examples were collected from open news sources, then automatically filtered with QA systems to prevent obvious questions to infiltrate the dataset. The texts were then filtered by IPM frequency of the contained words and, finally, manually reviewed.

WinoGAViL

This dataset is collected via the WinoGAViL game to collect challenging vision-and-language associations. Inspired by the popular card game Codenames, a “spymaster” gives a textual cue related to several visual candidates, and another player has to identify them. We use the game to collect 3.5K instances, finding that they are intuitive for humans (>90% Jaccard index) but challenging for state-of-the-art AI models, where the best model (ViLT) achieves a score of 52%, succeeding mostly where the cue is visually salient. Researchers are welcome to evaluate models on this dataset. A simple intended use is zero-shot prediction: run vision-and-language model, producing a score for the (cue,image) pair, and taking the K pairs with the highest scores. A supervised setting is also possible, code for re-running the experiments is available in the github repository. https://github.com/WinoGAViL/WinoGAViL-experiments

Mindgames

We generate epistemic reasoning problems using modal logic to target theory of mind (tom) in natural language processing models.

BCOPA-CE

We provide the BCOPA-CE test set, which has balanced token distribution in the correct and wrong alternatives and increases the difficulty of being aware of cause and effect. ### construction 1. for each premise of the 500 samples in COPA-test set, we generate one event manually which is a plausible answer to the opposite question type of the original sample. 2. obtain 500 triplets of <*premise*, *cause*, *effect*> 3. construct 1000 samples by giving two different questions (cause or effect) to each triplet.

CICEROv2

The CICEROv2 dataset can be found in the data directory. Each line of the files is a json object indicating a single instance. The json objects have the following key-value pairs: | Key | Value | |:----------:| :-----:| | ID | Dialogue ID with dataset indicator. | | Dialogue | Utterances of the dialogue in a list. | | Target | Target utterance. | | Question | One of the five questions (inference types). | | Choices | Five possible answer choices in a list. One of the answers is<br>human written. The other four answers are machine generated<br>and selected through the Adversarial Filtering (AF) algorithm. | | Human Written Answer | Index of the human written answer in a<br>single element list. Index starts from 0. | | Correct Answers | List of all correct answers indicated as plausible<br>or speculatively correct by the human annotators.<br>Includes the index of the human written answer. | --------------------------------------------------------------------------- An example of the data is shown below. { "ID": "daily-dialogue-0404", "Dialogue": [ "A: Dad , why are you taping the windows ?", "B: Honey , a typhoon is coming .", "A: Really ? Wow , I don't have to go to school tomorrow .", "B: Jenny , come and help , we need to prepare more food .", "A: OK . Dad ! I'm coming ." ], "Target": "Jenny , come and help , we need to prepare more food .", "Question": "What subsequent event happens or could happen following the target?", "Choices": [ "Jenny and her father stockpile food for the coming days.", "The speaker and the listener go outside to purchase more food material for precaution.", "Jenny and her father give away all their food.", "Jenny and her father eat all the food in their refrigerator." ], "Correct Answers": [ 0, 1 ] }

Physical Audiovisual CommonSense

PACS (Physical Audiovisual CommonSense) is the first audiovisual benchmark annotated for physical commonsense attributes. PACS contains a total of 13,400 question-answer pairs, involving 1,377 unique physical commonsense questions and 1,526 videos. The dataset provides new opportunities to advance the research field of physical reasoning by bringing audio as a core component of this multimodal problem.

CheGeKa

CheGeKa is a Jeopardy!-like Russian QA dataset collected from the official Russian quiz database ChGK. Motivation The task can be considered the most challenging in terms of reasoning, knowledge and logic, as the task implies the QA pairs with a free response form (no answer choices); however, a long chain of causal relationships between facts and associations forms the correct answer. The original corpus of the CheGeKa game was introduced in Mikhalkova (2021). An example in English for illustration purposes: { 'question_id': 3665, 'question': 'THIS MAN replaced John Lennon when the Beatles got together for the last time.', 'answer': 'Julian Lennon', 'topic': 'The Liverpool Four', 'author': 'Bayram Kuliyev', 'tour_name': 'Jeopardy!. Ashgabat-1996', 'tour_link': 'https://db.chgk.info/tour/ash96sv', 'episode': [16], 'perturbation': 'chegeka' } Data Fields - question_id: an integer corresponding to the question id in the database - question: a string containing the question text - answer: a string containing the correct answer to the question - topic: a string containing the question category - author: a string with the full name of the author - tour_name: a string with the title of a tournament - tour link: a string containing the link to a tournament (None for the test set) - perturbation: a string containing the name of the perturbation applied to text. If no perturbation was applied, the dataset name is used - episode: a list of episodes in which the instance is used. Only used for the train set Data Splits The dataset consists of a training set with labeled examples and a test set in two configurations: - raw data: includes the original data with no additional sampling - episodes: data is split into evaluation episodes and includes several perturbations of test for robustness evaluation Test Perturbations Each training episode in the dataset corresponds to seven test variations, including the original test data and six adversarial test sets, acquired through the modification of the original test through the following text perturbations: - ButterFingers: randomly adds noise to data by mimicking spelling mistakes made by humans through character swaps based on their keyboard distance - Emojify: replaces the input words with the corresponding emojis, preserving their original meaning - EDAdelete: randomly deletes tokens in the text - EDAswap: randomly swaps tokens in the text - BackTranslation: generates variations of the context through back-translation (ru -> en -> ru) - AddSent: generates extra words or a sentence at the end of the question

COFAR

The COFAR (COmmonsense and FActual Reasoning) dataset is a collection of images and text queries specifically designed to challenge and evaluate image search models that aim to go beyond simple visual matching. It focuses on the ability of these models to perform commonsense and factual reasoning, a capability currently lacking in most existing image search technology. Key Features of COFAR: * Named Visual Entities as a Gateway to Knowledge: The dataset consists of images containing prominent named visual entities, such as: * Business Brands: Examples include "Rolex", "Starbucks", and "KFC". * Celebrities: Examples include "Lionel Messi" and "J.K. Rowling". * Landmarks: Examples include "Eiffel Tower", "Taj Mahal", and "Parthenon". * Complex Queries Requiring Reasoning: Instead of simple object-based queries, COFAR uses carefully crafted textual queries that require a deeper understanding of the image content. These queries incorporate two main elements: * Factual Knowledge: This refers to specific facts about the named entities present in the image, often derived from external knowledge sources like Wikipedia. For instance, a query might state that "Lionel Messi is the captain of the Argentina national football team". * Commonsense Reasoning: Queries also involve commonsense assumptions about the scene, activities, and relationships depicted in the image. An example is "a queue of customers patiently waiting to buy ice cream", where the model needs to infer that people standing in line are likely "customers" who are "waiting". * Structured for Evaluation: COFAR is organised into distinct sets to facilitate model training and evaluation: * Training Set: This subset contains images and corresponding queries used to train image search models, allowing them to learn the relationships between visual content, textual descriptions, and external knowledge. * Gallery Sets: These sets contain images with entities unseen during training. Models are evaluated on their ability to retrieve relevant images from these galleries given a query, testing their generalisation to new, unseen entities. What Makes COFAR Unique? * Focus on Reasoning over Visual Recognition: COFAR shifts the emphasis from purely visual aspects of an image to the reasoning required to understand the query. This means that models cannot rely solely on detecting objects or scenes; they need to incorporate external knowledge and make commonsense inferences. * Diversity and Real-World Applicability: The dataset covers a diverse range of named entities and scenarios, making it more representative of real-world image search tasks. The inclusion of business brands, celebrities, and landmarks makes COFAR relevant to various practical applications, like e-commerce, news search, and tourism. * Promoting Advanced Image Search Technology: COFAR aims to stimulate the development of more sophisticated image search engines capable of understanding user intent and context. By providing a challenging benchmark, COFAR encourages researchers to explore new techniques and architectures that go beyond the limitations of existing methods. Limitations of COFAR: The sources also acknowledge some limitations of the COFAR dataset: * Single Entity Focus: Each image in COFAR contains only one prominent named visual entity. This might not accurately reflect real-world scenarios where multiple entities often co-exist within an image. * Dataset Size: While COFAR is larger than some previous datasets in this area, it is still limited in size compared to datasets used for training large vision-language models. This could potentially limit the generalisability of models trained on COFAR. Overall, COFAR is a valuable resource for researchers working on image search that involves complex reasoning. It presents a challenging benchmark and encourages the development of novel techniques that can leverage external knowledge and commonsense understanding to improve image search accuracy and relevance.

Machine Number Sense

Consists of visual arithmetic problems automatically generated using a grammar model--And-Or Graph (AOG). These visual arithmetic problems are in the form of geometric figures: each problem has a set of geometric shapes as its context and embedded number symbols. Source: Machine Number Sense: A Dataset of Visual Arithmetic Problems for Abstract and Relational Reasoning

QuRe

Introduction Generalized quantifiers (e.g., few, most) are used to indicate the proportions predicates are satisfied. QuRe is quantifier reasoning dataset from Pragmatic Reasoning Unlocks Quantifier Semantics for Foundation Models. It includes real-world sentences from Wikipedia and human annotations of generalized quantifiers from English speakers. ### Sample { "orig_sentence": "In order for a steel to be considered stainless it must have a Chromium content of at least 10.5%.", "percentage": "10.50%", "percentage_index": 0, "math_expr": ">=0.105", "quant_sent": "In order for a steel to be considered stainless it must have some Chromium content.", "quantifier": "some", "quantifier_position": 12, "specificity": "unable", "wiki_entity": "List of blade materials", "topics": "metallurgy; steel; composition" } * orig_sentence: the original sentence appeared in Wikipedia. * percentage: the percentage mentioned in the orig_sentence. * percentage_index: the index of the mentioned percentage in the orig_sentence. * math_expr: the percentage expression generated. * quant_sent: the annotated quantified sentence. * quantifier_position: the position of quantifier mentioned. * specificity: the difficulty of deciphering the percentage scope of the quantifier from the sentence excluding the quantifier. * wiki_entity: the wikipedia entity that includes <i>orig_sentence</i> in the wikipage content. * topics: sentence topics. ### Load Dataset from datasets import load_dataset ds = load_dataset("billli/QuRe") ### Reference @inproceedings{li-etal-2023-pragmatic, title = "Pragmatic Reasoning Unlocks Quantifier Semantics for Foundation Models", author = "Li, Yiyuan and Menon, Rakesh and Ghosh, Sayan and Srivastava, Shashank", editor = "Bouamor, Houda and Pino, Juan and Bali, Kalika", booktitle = "Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing", month = dec, year = "2023", address = "Singapore", publisher = "Association for Computational Linguistics", url = "https://aclanthology.org/2023.emnlp-main.38", pages = "573--591", }

Situation Puzzle

LLMs' lateral thinking capabilities remain under-explored and challenging to measure due to the complexity of assessing creative thought processes and the scarcity of relevant data. To address these challenges, we introduce SPLAT, a benchmark leveraging Situation Puzzles to evaluate and elicit LAteral Thinking of LLMs. This benchmark, containing 975 graded situation puzzles across three difficulty levels.

AC-Bench

AC-Bench: A Benchmark for Actual Causality Reasoning ## Dataset Description AC-Bench is designed to evaluate the actual causality (AC) reasoning capabilities of large language models (LLMs). It contains a collection of carefully annotated samples, each consisting of a story, a query related to actual causation, detailed reasoning steps, and a binary answer. The dataset aims to provide a comprehensive benchmark for assessing the ability of LLMs to perform formal and interpretable AC reasoning. ## Dataset Structure Each sample in the dataset is formatted as a JSON object with the following structure: json { "story": "A narrative describing a real-world scenario involving causal events.", "question": "A query asking whether a specific event caused an outcome.", "reasoning": { "causal_events": { "Event Description": { "occur": true/false, // Whether the event actually occurs. "order": integer, // The temporal order of the event. "focal": true/false, // Whether the event is the focal causal event. "sufficient": true/false, // Whether the event is a sufficient cause. "necessary": true/false, // Whether the event is a necessary cause. "halpern_pearl": true/false, // Whether the event is an actual cause. "norm_violated": true/false, // Whether the event violates a norm. "behavior_intended": true/false // Whether the event is intentional. }, ... }, "outcome_event": { "Outcome Description": { "occur": true/false, // Whether the outcome event occurs. "order": integer // The temporal order of the outcome event. } } }, "answer": "Yes/No" // The binary answer to the question. } ## Example Sample json { "story": "Bill's wife, Sue, is out of town for the weekend. She leaves Bill a message that says, 'I just saw this marvelous bookend. It's called a Bartlett bookend. So pretty! I'm going to go back tomorrow and get one. It will be perfect for the left side of our bookshelf'. Bill goes and visits his friend. Bill and his friend talk for a while, and when Bill asks if his friend is willing to sell the bookend, his friend is happy to sell it. Bill makes an offer, but his friend insists on him not paying so much. Finally, Bill buys the right-side Bartlett bookend from his friend and goes home. Then the next day, Sue goes and buys the left-side Bartlett bookend. So, when Sue got home, they had the paired set of bookends.", "question": "Did Bill cause them to possess the paired set of bookends?", "reasoning": { "causal_events": { "Bill buys the right-side Bartlett bookend": { "occur": true, "order": 0, "focal": true, "sufficient": false, "necessary": true, "halpern_pearl": true, "norm_violated": false, "behavior_intended": true }, "Sue buys the left-side Bartlett bookend": { "occur": true, "order": 1, "focal": false, "sufficient": false, "necessary": true, "halpern_pearl": true, "norm_violated": false, "behavior_intended": true } }, "outcome_event": { "Bill and Sue possess the paired set of bookends": { "occur": true, "order": 2 } } }, "answer": "Yes" } ## Dataset Features - Comprehensive Annotations: Each sample includes detailed reasoning steps for actual causality reasoning. - Focus on Actual Causation: The dataset is specifically designed to evaluate reasoning about actual causality, distinguishing it from type causality. - Challenging and Diverse Samples: The dataset includes a variety of causal scenarios, and presents greater challenge for LLMs compared with Big-Bench Hard Causal Judgment. ## Usage This dataset can be used to train, evaluate, and analyze the performance of LLMs on actual causality reasoning tasks. Researchers can leverage this benchmark to develop more interpretable and accurate causal reasoning models. ## License This dataset is released under the Creative Commons Attribution 4.0 International License. ## Acknowledgments This dataset is based on the work presented in the paper "AC-Reason: Towards Theory-Guided Actual Causality Reasoning with Large Language Models" by Yanxi Zhang, Xin Cong, Zhong Zhang, Xiao Liu, Dongyan Zhao, and Yesai Wu. We thank the authors for their contributions to the field of causal reasoning.

ACCORD CSQA 0-5

ACCORD CSQA is an extension of the popular CommonsenseQA (CSQA) dataset using ACCORD, a scalable framework for disentangling the commonsense grounding and reasoning abilities of large language models (LLMs) through controlled, multi-hop counterfactuals. ACCORD closes the measurability gap between commonsense and formal reasoning tasks for LLMs. A detailed understanding of LLMs' commonsense reasoning abilities is severely lagging compared to our understanding of their formal reasoning abilities, since commonsense benchmarks are difficult to construct in a manner that is rigorously quantifiable. Specifically, prior commonsense reasoning benchmarks and datasets are limited to one- or two-hop reasoning or include an unknown (i.e., non-measurable) number of reasoning hops and/or distractors. Arbitrary scalability via compositional construction is also typical of formal reasoning tasks but lacking in commonsense reasoning. Finally, most prior commonsense benchmarks either are limited to a single reasoning skill or do not control skills. ACCORD aims to address all these gaps by introducing formal elements to commonsense reasoning to explicitly control and quantify reasoning complexity beyond the typical 1 or 2 reasoning hops. Uniquely, ACCORD can automatically generate benchmarks of arbitrary reasoning complexity, and so it scales with future LLM improvements. ACCORD CSQA is a benchmark suite comprising problem with 6 levels of reasoning difficulty, ACCORD CSQA 0 to ACCORD CSQA 5. Experiments on state-of-the-art LLMs show performance degrading to random chance with only moderate scaling, leaving substantial headroom for improvement.

Advice Seeking Questions

The Advice-Seeking Questions (ASQ) dataset is a collection of personal narratives with advice-seeking questions. The dataset has been split into train, test, heldout sets, with 8865, 2500, 10000 test instances each. This dataset is used to train and evaluate methods that can infer what is the advice-seeking goal behind a personal narrative. This task is formulated as a cloze test, where the goal is to identify which of two advice-seeking questions was removed from a given narrative. Source: https://github.com/CornellNLP/ASQ

DpgMedia2019

DpgMedia2019 is a Dutch news dataset for partisanship detection. It contains more than 100K articles that are labelled on the publisher level and 776 articles that were crowdsourced using an internal survey platform and labelled on the article level. Source: https://github.com/dpgmedia/partisan-news2019

Russian Event2Mind

The work provides a comprehensive overview of the corpus for the Russian language for the commonsense inference task. Namely, we construct event phrases, which cover a wide range of everyday situations with labelled intents and reactions of the event main participant and emotions of other people involved. Example: The main contribution of the paper is a text corpus suitable for event2mind training in Russian which consists of two parts: 1. 6,756 event phrases covering a diverse range of everyday events and situations in Russian, 2. a subset of 23,409 event phrases from English corpus translated via Google translator.

This is not a Dataset

We introduce a large semi-automatically generated dataset of ~400,000 descriptive sentences about commonsense knowledge that can be true or false in which negation is present in about 2/3 of the corpus in different forms that we use to evaluate LLMs. Source: ThisIsNotaDataset Image: ThisIsNotaDataset

Winograd Automatic

The Winograd schema challenge composes tasks with syntactic ambiguity, which can be resolved with logic and reasoning. Motivation The dataset presents an extended version of a traditional Winograd challenge (Levesque et al., 2012): each sentence contains unresolved homonymy, which can be resolved based on commonsense and reasoning. The Winograd scheme is extendable with the real-life sentences filtered out of the National Corpora with a set of 11 syntactic queries, extracting sentences like "Katya asked Masha if she..." (two possible references to a pronoun), "A change of scenery that..." (Noun phrase & subordinate clause with "that" in the same gender and number), etc. The extraction pipeline can be adjusted to various languages depending on the set of ambiguous syntactic constructions possible. An example in English for illustration purposes: { ‘text’: ‘But then I was glad, because in the end the singer from Turkey who performed something national, although in a modern version, won.’, ‘answer’: ‘singer’, ‘label’: 1, ‘options’: [‘singer’, ‘Turkey’], ‘reference’: ‘who’, ‘homonymia_type’: ‘1.1’, episode: [15], ‘perturbation’ : ‘winograd’ } Data Fields - text: a string containing the sentence text - answer: a string with a candidate for the coreference resolution - options: a list of all the possible candidates present in the text - reference: a string containing an anaphor (a word or phrase that refers back to an earlier word or phrase) - homonymia_type: a float corresponding to the type of the structure with syntactic homonymy - label: an integer, either 0 or 1, indicating whether the homonymy is resolved correctly or not - perturbation: a string containing the name of the perturbation applied to text. If no perturbation was applied, the dataset name is used - episode: a list of episodes in which the instance is used. Only used for the train set Data Splits The dataset consists of a training set with labeled examples and a test set in two configurations: - raw data: includes the original data with no additional sampling - episodes: data is split into evaluation episodes and includes several perturbations of test for robustness evaluation The train and test sets are disjoint with respect to the sentence-candidate answer pairs but may include overlaps in individual sentences and homonymy type. Test Perturbations Each training episode in the dataset corresponds to six test variations, including the original test data and five adversarial test sets, acquired through the modification of the original test through the following text perturbations: - ButterFingers: randomly adds noise to data by mimicking spelling mistakes made by humans through character swaps based on their keyboard distance - Emojify: replaces the input words with the corresponding emojis, preserving their original meaning - EDAdelete: randomly deletes tokens in the text - EDAswap: randomly swaps tokens in the text - AddSent: generates extra words or a sentence at the end of the text

Compositional Visual Reasoning (CVR)

A fundamental component of human vision is our ability to parse complex visual scenes and judge the relations between their constituent objects. AI benchmarks for visual reasoning have driven rapid progress in recent years with state-of-the-art systems now reaching human accuracy on some of these benchmarks. Yet, there remains a major gap between humans and AI systems in terms of the sample efficiency with which they learn new visual reasoning tasks. Humans' remarkable efficiency at learning has been at least partially attributed to their ability to harness compositionality -- allowing them to efficiently take advantage of previously gained knowledge when learning new tasks. Here, we introduce a novel visual reasoning benchmark, Compositional Visual Relations (CVR), to drive progress towards the development of more data-efficient learning algorithms. We take inspiration from fluidic intelligence and non-verbal reasoning tests and describe a novel method for creating compositions of abstract rules and generating image datasets corresponding to these rules at scale. Our proposed benchmark includes measures of sample efficiency, generalization, compositionality, and transfer across task rules. We systematically evaluate modern neural architectures and find that convolutional architectures surpass transformer-based architectures across all performance measures in most data regimes. However, all computational models are much less data efficient than humans, even after learning informative visual representations using self-supervision. Overall, we hope our challenge will spur interest in developing neural architectures that can learn to harness compositionality for more efficient learning.

출처: Papers with Code · CC BY-SA 4.0. 아카이브 자료와 새로 수집한 논문은 별도로 관리합니다.