The Beyond the Imitation Game Benchmark (BIG-bench) is a collaborative benchmark intended to probe large language models and extrapolate their future capabilities. Big-bench include more than 200 tasks. Image source: https://arxiv.org/pdf/2206.04615.pdf

By perturbing the widely used GSM8K dataset, an adversarial dataset for grade-school math called GSM-Plus is created. Motivated by the capability taxonomy for solving math problems mentioned in Polya's principles, this paper identifies 5 perspectives to guide the development of GSM-Plus: * Numerical Variation refers to altering the numerical data or its types, including 3 subcategories: Numerical Substitution, Digit Expansion, and Integer-decimal-fraction Conversion. * Arithmetic Variation refers to reversing or introducing additional operations (e.g., addition, subtraction, multiplication, and division) to math problems, including 2 subcategories: Adding Operation and Reversing Operation. * Problem Understanding refers to rephrasing the text description of the math problems. * Distractor Insertion refers to inserting topic-related but useless sentences to the problems. * Critical Thinking focuses on question or doubt ability when the question lacks necessary statements. GSM-Plus can be used to evaluate the robustness of current LLMs in mathematical reasoning.

MathBench is an All in One math dataset for language model evaluation, with: A Sophisticated Five-Stage Difficulty Mechanism: Unlike the usual mathematical datasets that can only evaluate a single difficulty level or have a mix of unclear difficulty levels, MathBench provides 3709 questions with a gradient difficulty division by education stages, ranging from basic arithmetic to primary, middle, high school, and college levels, allowing you to get a clear overview of the comprehensive difficulty evaluation results. Bilingual Gradient Evaluation: Apart from the basic calculation part which is language-independent, MathBench provides questions in both Chinese and English for the four-stage difficulty datasets from primary to college. Implementation of the Robust Circular Evaluation (CE) Method: MathBench use CE as the main evaluation method for questions. Compared to the general Accuracy evaluation method, CE requires the model to answer the same multiple-choice question multiple times, with the order of the options changing each time. The model is considered correct on this question only if all answers are correct. The results of CE can reflect the model's capabilities more realistically, providing more valuable evaluation results. Support for Basic Theory Questions: For every stage, MathBench provides questions that cover the basic theory knowledge points of the corresponding stage, to ascertain whether the model has genuinely mastered the fundamental concepts of each stage or merely memorized the answers.

Recent times have witnessed an increasing number of applications of deep neural networks towards solving tasks that require superior cognitive abilities, e.g., playing Go, generating art, ChatGPT, etc. Such a dramatic progress raises the question: how generalizable are neural networks in solving problems that demand broad skills? To answer this question, we propose SMART: a Simple Multimodal Algorithmic Reasoning Task (and the associated SMART-101 dataset) for evaluating the abstraction, deduction, and generalization abilities of neural networks in solving visuo-linguistic puzzles designed specifically for children of younger age (6--8). Our dataset consists of 101 unique puzzles; each puzzle comprises a picture and a question, and their solution needs a mix of several elementary skills, including pattern recognition, algebra, and spatial reasoning, among others. To train deep neural networks, we programmatically augment each puzzle to 2,000 new instances; each instance varied in appearance, associated natural language question, and its solution. To foster research and make progress in the quest for artificial general intelligence, we are publicly releasing our SMART-101 dataset, consisting of the full set of programmatically-generated instances of 101 puzzles and their solutions. The dataset was introduced in our paper Are Deep Neural Networks SMARTer than Second Graders? by Anoop Cherian, Kuan-Chuan Peng, Suhas Lohit, Kevin A. Smith, and Joshua B. Tenenbaum, CVPR 2023

DBE-KT22 contains student exercise answering activities collected through an online practicing platform for the database systems course taught at the Australian National University within the period 2018-2021. The dataset is useful for research targeting students' knowledge tracing given historical sequences of exercise answering.

A deductive reasoning benchmark based on formal logic theory. A model is required to generate a proof that (dis-) proves a given hypothesis based on a given set of facts.

This dataset is a benchmark for complex reasoning abilities in large language models, drawing on United Kingdom Linguistics Olympiad problems which cover a wide range of languages.

MultiQ is a multi-hop QA dataset for Russian, suitable for general open-domain question answering, information retrieval, and reading comprehension tasks. Motivation Question-answering has been an essential task in natural language processing and information retrieval. However, certain areas in QA remain quite challenging for modern approaches, including the multi-hop one, which is traditionally considered an intersection of graph methods, knowledge representation, and SOTA language modeling. Multi-hop reasoning has been the least addressed QA direction for Russian. The task is represented by the MuSeRC dataset (Fenogenova et al., 2020) and only a few dozen questions in SberQUAD (Efimov et al., 2020) and RuBQ (Rybin et al., 2021). In response, we have developed a semi-automatic pipeline for multi-hop dataset generation based on Wikidata. An example in English for illustration purposes: { 'support_text': 'Gerard McBurney (b. June 20, 1954, Cambridge) is a British arranger, musicologist, television and radio presenter, teacher, and writer. He was born in the family of American archaeologist Charles McBurney and secretary Anna Frances Edmonston, who combined English, Scottish and Irish roots. Gerard's brother Simon McBurney is an English actor, writer, and director. He studied at Cambridge and the Moscow State Conservatory with Edison Denisov and Roman Ledenev.', 'main_text': 'Simon Montague McBurney (born August 25, 1957, Cambridge) is an English actor, screenwriter, and director. Biography. Father is an American archaeologist who worked in the UK. Simon graduated from Cambridge with a degree in English Literature. After his father's death (1979) he moved to France, where he studied theater at the Jacques Lecoq Institute. In 1983 he created the theater company "Complicity". Actively works as an actor in film and television, and acts as a playwright and screenwriter.', 'question': 'Where was Gerard McBurney's brother born?', 'bridge_answers': [{'label': 'passage', 'length': 14, 'offset': 300, 'segment': 'Simon McBurney'}], 'main_answers': [{'label': 'passage', 'length': 9, 'offset': 47, 'segment': Cambridge'}], 'episode': [15], 'perturbation': 'multiq' } Data Fields - question: a string containing the question text - support_text: a string containing the first text passage relating to the question - main_text: a string containing the main answer text - bridge_answers: a list of entities required to hop from the support text to the main text - main_answers: a list of answers to the question - perturbation: a string containing the name of the perturbation applied to text. If no perturbation was applied, the dataset name is used - episode: a list of episodes in which the instance is used. Only used for the train set Data Splits The dataset consists of a training set with labeled examples and a test set in two configurations: raw data: includes the original data with no additional sampling - episodes: data is split into evaluation episodes and includes several perturbations of test for robustness evaluation Test and train data sets are disjoint with respect to individual - questions, but may include overlaps in support and main texts. Test Perturbations Each training episode in the dataset corresponds to seven test variations, including the original test data and six adversarial test sets, acquired through the modification of the original test through the following text perturbations: - ButterFingers: randomly adds noise to data by mimicking spelling mistakes made by humans through character swaps based on their keyboard distance - Emojify: replaces the input words with the corresponding emojis, preserving their original meaning - EDAdelete: randomly deletes tokens in the text - EDAswap: randomly swaps tokens in the text - BackTranslation: generates variations of the context through back-translation (ru -> en -> ru) - AddSent: generates an extra sentence at the end of the text

POPGym is designed to benchmark memory in deep reinforcement learning. It contains a set of environments and a collection of memory model baselines. The environments are all Partially Observable Markov Decision Process (POMDP) environments following the Openai Gym interface. Our environments follow a few basic tenets: 1. Painless Setup - popgym environments require only gym, numpy, and mazelib as dependencies 2. Laptop-Sized Tasks - Most tasks can be solved in less than a day on the CPU 3. True Generalization - All environments are heavily randomized. The paper uses 15M environment steps for each trial.

CheGeKa is a Jeopardy!-like Russian QA dataset collected from the official Russian quiz database ChGK. Motivation The task can be considered the most challenging in terms of reasoning, knowledge and logic, as the task implies the QA pairs with a free response form (no answer choices); however, a long chain of causal relationships between facts and associations forms the correct answer. The original corpus of the CheGeKa game was introduced in Mikhalkova (2021). An example in English for illustration purposes: { 'question_id': 3665, 'question': 'THIS MAN replaced John Lennon when the Beatles got together for the last time.', 'answer': 'Julian Lennon', 'topic': 'The Liverpool Four', 'author': 'Bayram Kuliyev', 'tour_name': 'Jeopardy!. Ashgabat-1996', 'tour_link': 'https://db.chgk.info/tour/ash96sv', 'episode': [16], 'perturbation': 'chegeka' } Data Fields - question_id: an integer corresponding to the question id in the database - question: a string containing the question text - answer: a string containing the correct answer to the question - topic: a string containing the question category - author: a string with the full name of the author - tour_name: a string with the title of a tournament - tour link: a string containing the link to a tournament (None for the test set) - perturbation: a string containing the name of the perturbation applied to text. If no perturbation was applied, the dataset name is used - episode: a list of episodes in which the instance is used. Only used for the train set Data Splits The dataset consists of a training set with labeled examples and a test set in two configurations: - raw data: includes the original data with no additional sampling - episodes: data is split into evaluation episodes and includes several perturbations of test for robustness evaluation Test Perturbations Each training episode in the dataset corresponds to seven test variations, including the original test data and six adversarial test sets, acquired through the modification of the original test through the following text perturbations: - ButterFingers: randomly adds noise to data by mimicking spelling mistakes made by humans through character swaps based on their keyboard distance - Emojify: replaces the input words with the corresponding emojis, preserving their original meaning - EDAdelete: randomly deletes tokens in the text - EDAswap: randomly swaps tokens in the text - BackTranslation: generates variations of the context through back-translation (ru -> en -> ru) - AddSent: generates extra words or a sentence at the end of the question

🎯 DART-Math > Difficulty-Aware Rejection Tuning for Mathematical Problem-Solving 📝 Paper@arXiv | 🤗 Datasets&Models@HF | 🐱 Code@GitHub 🐦 Thread@X(Twitter) | 🐶 中文博客@知乎 | 📊 Leaderboard@PapersWithCode | 📑 BibTeX ## Datasets: DART-Math DART-Math datasets are the state-of-the-art and data-efficient open-source instruction tuning datasets for mathematical reasoning. DART-Math-Hard contains ~585k mathematical QA pair samples constructed by applying DARS-Prop2Diff to the query set from MATH and GSK8K training sets, achieves SOTA on many challenging mathematical reasoning benchmarks. It introduces a deliberate bias towards hard queries, opposite to vanilla rejection sampling. Performance produced by DART-Math-Hard is usually but not necessarily slightly better (~1% absolutely) than DART-Math-Uniform, which contains ~591k samples constructed by applying DARS-Uniform. ### Comparison between Mathematical Instruction Tuning Datasets Most of previous datasets are constructed with ChatGPT, and many of them are not open-source, especially for ones of the best performance. | Math SFT Dataset | # of Samples | MATH | GSM8K | College | Synthesis Agent(s) | Open-Source | | :--------------------------------------------------------------------------------- | -----------: | -----------------------------------------------------------------: | ---------------------------------------------: | -----------------------------------------------------------------------------------------------------------: | :---------------------- | :-------------------------------------------------------------------------: | | WizardMath | 96k | 32.3 | 80.4 | 23.1 | GPT-4 | ✗ | | MetaMathQA | 395k | 29.8 | 76.5 | 19.3 | GPT-3.5 | ✓ | | MMIQC | 2294k | 37.4 | 75.4 | 28.5 | GPT-4+GPT-3.5+Human | ✓ | | Orca-Math | 200k | -- | -- | -- | GPT-4 | ✓ | | Xwin-Math-V1.1 | 1440k | 45.5 | 84.9 | 27.6 | GPT-4 | ✗ | | KPMath-Plus | 1576k | 46.8 | 82.1 | -– | GPT-4 | ✗ | | MathScaleQA | 2021k | 35.2 | 74.8 | 21.8 | GPT-3.5+Human | ✗ | | DART-Math-Uniform | 591k | 43.5 | 82.6 | 26.9 | DeepSeekMath-7B-RL | ✓ | | DART-Math-Hard | 585k | 45.5 | 81.1 | 29.4 | DeepSeekMath-7B-RL | ✓ | ^{MATH and GSM8K are in-domain, while College(Math) is out-of-domain. Performance here are of models fine-tuned from Mistral-7B, except for Xwin-Math-V1.1 based on Llama2-7B. Bold/Italic means the best/second best score here.} ## Dataset Construction: DARS - Difficulty-Aware Rejection Sampling Previous works usually synthesize data from proprietary models to augment existing datasets, followed by instruction tuning to achieve top-tier results. However, our analysis of these datasets reveals severe biases towards easy queries, with frequent failures to generate any correct response for the most challenging queries. Motivated by the observation above, we propose to Difficulty-Aware Rejection Sampling (DARS), to collect more responses for more difficult queries. Specifically, we introduce two strategies to increase the number of correct responses for difficult queries: 1) Uniform, which involves sampling responses for each query until each query accumulates $k_u$ correct responses, where $k_u$ is a preset hyperparameter determined by the desired size of the synthetic dataset; 2) Prop2Diff, where we continue sampling responses until the number of correct responses for each query is proportional to its difficulty score. The most challenging queries will receive $k_p$ responses and kp is a hyperparameter. This method introduces a deliberate bias in the opposite direction to vanilla rejection sampling, towards more difficult queries, inspired by previous works that demonstrate difficult samples can be more effective to enhance model capabilities (Sorscher et al., 2022; Liu et al., 2024b). See Figure 1 (Right) for examples of DART-Math-Uniform by DARS-Uniform and DART-Math-Hard by DARS-Prop2Diff. ## Citation If you find our data, model or code useful for your work, please kindly cite our paper: latex @article{tong2024dartmath, title={DART-Math: Difficulty-Aware Rejection Tuning for Mathematical Problem-Solving}, author={Yuxuan Tong and Xiwen Zhang and Rui Wang and Ruidong Wu and Junxian He}, year={2024}, eprint={2407.13690}, archivePrefix={arXiv}, primaryClass={cs.CL}, url={https://arxiv.org/abs/2407.13690}, }

Ethics (per ethics) dataset is created to test the knowledge of the basic concepts of morality. The task is to predict human ethical judgments about diverse text situations in a multi-label classification setting. The main objective of the task is to evaluate the positive or negative implementation of five concepts in normative with ‘yes’ and ‘no’ ratings. The included concepts are as follows: virtue, law, moral, justice, and utilitarianism. Motivation There are a multitude of approaches to evaluating ethics in machine learning. The Ethics dataset for Russian is created from scratch for the first time, relying on the design compatible with (Hendrycks et al., 2021). Our Ethics dataset would go through community validation and discussion as it is the first ethics dataset for Russian based on the established methodology. We acknowledge that the work (Hendrycks et al., 2021) has flaws; thus, we do not reproduce the generative approach. We construct the dataset using a similar annotation scheme: we avoid the direct question of whether the deed is good or bad. Instead, we make annotations according to five criteria that describe the aspects of the annotators' attitude to the deed. An example in English for illustration purposes: { 'source': 'gazeta', 'text': '100-year-old Greta Ploech gave handmade cookies to a toddler who helped her cross a busy highway at a pedestrian crossing. The video was posted on the Readers Channel.', 'sit_virtue': 1, 'sit_moral': 0, 'sit_law': 0, 'sit_justice': 1, 'sit_util': 1, 'episode': [5], 'perturbation': 'sit_ethics' } Data Fields - text: a string containing the body of a news article or a fiction text - source: a string containing the source of the text - per_virtue: an integer, either 0 or 1, indicating whether virtue standards are violated in the text - per_moral: an integer, either 0 or 1, indicating whether moral standards are violated in the text - per_law: an integer, either 0 or 1, indicating whether any laws are violated in the text - per_justice: an integer, either 0 or 1, indicating whether justice norms are violated in the text - per_util: an integer, either 0 or 1, indicating whether utilitarianism norms are violated in the text - perturbation: a string containing the name of the perturbation applied to text. If no perturbation was applied, the dataset name is used - episode: a list of episodes in which the instance is used. Only used for the train set Data Splits The dataset consists of a training set with labeled examples and a test set in two configurations: - raw data: includes the original data with no additional sampling - episodes: data is split into evaluation episodes and includes several perturbations of test for robustness evaluation Test Perturbations Each training episode in the dataset corresponds to seven test variations, including the original test data and six adversarial test sets, acquired through the modification of the original test through the following text perturbations: - ButterFingers: randomly adds noise to data by mimicking spelling mistakes made by humans through character swaps based on their keyboard distance - Emojify: replaces the input words with the corresponding emojis, preserving their original meaning - EDAdelete: randomly deletes tokens in the text - EDAswap: randomly swaps tokens in the text - BackTranslation: generates variations of the context through back-translation (ru -> en -> ru) - AddSent: generates an extra sentence at the end of the text

General-purpose Visual Understanding Evaluation (G-VUE) is a comprehensive benchmark covering the full spectrum of visual cognitive abilities with four functional domains -- Perceive, Ground, Reason, and Act. The four domains are embodied in 11 carefully curated tasks, from 3D reconstruction to visual reasoning and manipulation. Source: Perceive, Ground, Reason, and Act: A Benchmark for General-purpose Visual Representation Image Source: https://github.com/wllmzhu/G-VUE

Introduction Generalized quantifiers (e.g., few, most) are used to indicate the proportions predicates are satisfied. QuRe is quantifier reasoning dataset from Pragmatic Reasoning Unlocks Quantifier Semantics for Foundation Models. It includes real-world sentences from Wikipedia and human annotations of generalized quantifiers from English speakers. ### Sample { "orig_sentence": "In order for a steel to be considered stainless it must have a Chromium content of at least 10.5%.", "percentage": "10.50%", "percentage_index": 0, "math_expr": ">=0.105", "quant_sent": "In order for a steel to be considered stainless it must have some Chromium content.", "quantifier": "some", "quantifier_position": 12, "specificity": "unable", "wiki_entity": "List of blade materials", "topics": "metallurgy; steel; composition" } * orig_sentence: the original sentence appeared in Wikipedia. * percentage: the percentage mentioned in the orig_sentence. * percentage_index: the index of the mentioned percentage in the orig_sentence. * math_expr: the percentage expression generated. * quant_sent: the annotated quantified sentence. * quantifier_position: the position of quantifier mentioned. * specificity: the difficulty of deciphering the percentage scope of the quantifier from the sentence excluding the quantifier. * wiki_entity: the wikipedia entity that includes <i>orig_sentence</i> in the wikipage content. * topics: sentence topics. ### Load Dataset from datasets import load_dataset ds = load_dataset("billli/QuRe") ### Reference @inproceedings{li-etal-2023-pragmatic, title = "Pragmatic Reasoning Unlocks Quantifier Semantics for Foundation Models", author = "Li, Yiyuan and Menon, Rakesh and Ghosh, Sayan and Srivastava, Shashank", editor = "Bouamor, Houda and Pino, Juan and Bali, Kalika", booktitle = "Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing", month = dec, year = "2023", address = "Singapore", publisher = "Association for Computational Linguistics", url = "https://aclanthology.org/2023.emnlp-main.38", pages = "573--591", }

RuOpenBookQA is a QA dataset with multiple-choice elementary-level science questions which probe the understanding of core science facts. Motivation RuOpenBookQA is mainly based on the work of (Mihaylov et al., 2018): it is a QA dataset with multiple-choice elementary-level science questions, which probe the understanding of 1k+ core science facts. Very similar to the pipeline of the RuWorldTree, the dataset includes a corpus of factoids, factoid questions and correct answer. Only one fact is enough to find the correct answer, so this task can be considered easier. { 'ID': '7-674', 'question': 'If a person walks in the direction opposite to the compass needle, they are going (A) west (B) north (C) east (D) south', 'answer': 'D', 'episode': [11], 'perturbation': 'ru_openbook' } Data Fields - ID: a string containing a unique question id - question: a string containing question text with answer options - answer: a string containing the correct answer key (A, B, C or D) - perturbation: a string containing the name of the perturbation applied to text. If no perturbation was applied, the dataset name is used - episode: a list of episodes in which the instance is used. Only used for the train set Data Splits The dataset consists of a training set with labeled examples and a test set in two configurations: - raw data: includes the original data with no additional sampling - episodes: data is split into evaluation episodes and includes several perturbations of test for robustness evaluation Test Perturbations Each training episode in the dataset corresponds to seven test variations, including the original test data and six adversarial test sets, acquired through the modification of the original test through the following text perturbations: - ButterFingers: randomly adds noise to data by mimicking spelling mistakes made by humans through character swaps based on their keyboard distance - Emojify: replaces the input words with the corresponding emojis, preserving their original meaning - EDAdelete: randomly deletes tokens in the text - EDAswap: randomly swaps tokens in the text - BackTranslation: generates variations of the context through back-translation (ru -> en -> ru) - AddSent: replaces one or more choice options with a generated one

RuWorldTree is a QA dataset with multiple-choice elementary-level science questions, which evaluate the understanding of core science facts. Motivation The WorldTree dataset starts the triad of the Reasoning and Knowledge tasks. The data includes the corpus of factoid utterances of various kinds, complex factoid questions and a corresponding causal chain of facts from the corpus resulting in a correct answer. The WorldTree design was originally proposed in (Jansen et al., 2018). An example in English for illustration purposes: { 'question': 'A bottle of water is placed in the freezer. What property of water will change when the water reaches the freezing point? (A) color (B) mass (C) state of matter (D) weight', 'answer': 'C', 'exam_name': 'MEA', 'school_grade': 5, 'knowledge_type': 'NO TYPE', 'perturbation': 'ru_worldtree', 'episode': [18, 10, 11] } Data Fields - text: a string containing the sentence text - answer: a string with a candidate for the coreference resolution - options: a list of all the possible candidates present in the text - reference: a string containing an anaphor (a word or phrase that refers back to an earlier word or phrase) - homonymia_type: a float corresponding to the type of the structure with syntactic homonymy - label: an integer, either 0 or 1, indicating whether the homonymy is resolved correctly or not perturbation: a string containing the name of the perturbation applied to text. If no perturbation was applied, the dataset name is used - episode: a list of episodes in which the instance is used. Only used for the train set Data Splits The dataset consists of a training set with labeled examples and a test set in two configurations: - raw data: includes the original data with no additional sampling - episodes: data is split into evaluation episodes and includes several perturbations of test for robustness evaluation We use the same splits of data as in the original English version. Test Perturbations Each training episode in the dataset corresponds to seven test variations, including the original test data and six adversarial test sets, acquired through the modification of the original test through the following text perturbations: - ButterFingers: randomly adds noise to data by mimicking spelling mistakes made by humans through character swaps based on their keyboard distance - Emojify: replaces the input words with the corresponding emojis, preserving their original meaning - EDAdelete: randomly deletes tokens in the text - EDAswap: randomly swaps tokens in the text - BackTranslation: generates variations of the context through back-translation (ru -> en -> ru) - AddSent: replaces one or more choice options with a generated one

A benchmark for suppositional reasoning based on the principles of knights and knaves puzzles. Knights and knaves problems represent a classic genre of logical puzzles where characters either tell the truth or lie. The objective is to logically deduce each character's identity based on their statements. The challenge arises from the truth-telling or lying behavior, which influences the logical implications of each statement. TruthQuest encompasses problems of varying complexity, considering both the number of characters and the types of logical statements involved. The dataset comprises 2,400 instances. Each instance has a unique solution.

ASyMOB (pronounced Asimov, in tribute to the renowned author), is a novel assessment framework focused exclusively on symbolic manipulation, featuring 17,092 unique math challenges, organized by similarity and complexity. ASyMOB enables analysis of LLM failure root-causes and generalization capabilities by comparing performance in problems that differ by simple numerical or symbolic "perturbations".

🎯 DART-Math > Difficulty-Aware Rejection Tuning for Mathematical Problem-Solving 📝 Paper@arXiv | 🤗 Datasets&Models@HF | 🐱 Code@GitHub 🐦 Thread@X(Twitter) | 🐶 中文博客@知乎 | 📊 Leaderboard@PapersWithCode | 📑 BibTeX ## Datasets: DART-Math DART-Math datasets are the state-of-the-art and data-efficient open-source instruction tuning datasets for mathematical reasoning. DART-Math-Hard contains ~585k mathematical QA pair samples constructed by applying DARS-Prop2Diff to the query set from MATH and GSK8K training sets, achieves SOTA on many challenging mathematical reasoning benchmarks. It introduces a deliberate bias towards hard queries, opposite to vanilla rejection sampling. Performance produced by DART-Math-Hard is usually but not necessarily slightly better (~1% absolutely) than DART-Math-Uniform, which contains ~591k samples constructed by applying DARS-Uniform. ### Comparison between Mathematical Instruction Tuning Datasets Most of previous datasets are constructed with ChatGPT, and many of them are not open-source, especially for ones of the best performance. | Math SFT Dataset | # of Samples | MATH | GSM8K | College | Synthesis Agent(s) | Open-Source | | :--------------------------------------------------------------------------------- | -----------: | -----------------------------------------------------------------: | ---------------------------------------------: | -----------------------------------------------------------------------------------------------------------: | :---------------------- | :-------------------------------------------------------------------------: | | WizardMath | 96k | 32.3 | 80.4 | 23.1 | GPT-4 | ✗ | | MetaMathQA | 395k | 29.8 | 76.5 | 19.3 | GPT-3.5 | ✓ | | MMIQC | 2294k | 37.4 | 75.4 | 28.5 | GPT-4+GPT-3.5+Human | ✓ | | Orca-Math | 200k | -- | -- | -- | GPT-4 | ✓ | | Xwin-Math-V1.1 | 1440k | 45.5 | 84.9 | 27.6 | GPT-4 | ✗ | | KPMath-Plus | 1576k | 46.8 | 82.1 | -– | GPT-4 | ✗ | | MathScaleQA | 2021k | 35.2 | 74.8 | 21.8 | GPT-3.5+Human | ✗ | | DART-Math-Uniform | 591k | 43.5 | 82.6 | 26.9 | DeepSeekMath-7B-RL | ✓ | | DART-Math-Hard | 585k | 45.5 | 81.1 | 29.4 | DeepSeekMath-7B-RL | ✓ | ^{MATH and GSM8K are in-domain, while College(Math) is out-of-domain. Performance here are of models fine-tuned from Mistral-7B, except for Xwin-Math-V1.1 based on Llama2-7B. Bold/Italic means the best/second best score here.} ## Dataset Construction: DARS - Difficulty-Aware Rejection Sampling Previous works usually synthesize data from proprietary models to augment existing datasets, followed by instruction tuning to achieve top-tier results. However, our analysis of these datasets reveals severe biases towards easy queries, with frequent failures to generate any correct response for the most challenging queries. Motivated by the observation above, we propose to Difficulty-Aware Rejection Sampling (DARS), to collect more responses for more difficult queries. Specifically, we introduce two strategies to increase the number of correct responses for difficult queries: 1) Uniform, which involves sampling responses for each query until each query accumulates $k_u$ correct responses, where $k_u$ is a preset hyperparameter determined by the desired size of the synthetic dataset; 2) Prop2Diff, where we continue sampling responses until the number of correct responses for each query is proportional to its difficulty score. The most challenging queries will receive $k_p$ responses and kp is a hyperparameter. This method introduces a deliberate bias in the opposite direction to vanilla rejection sampling, towards more difficult queries, inspired by previous works that demonstrate difficult samples can be more effective to enhance model capabilities (Sorscher et al., 2022; Liu et al., 2024b). See Figure 1 (Right) for examples of DART-Math-Uniform by DARS-Uniform and DART-Math-Hard by DARS-Prop2Diff. ## Citation If you find our data, model or code useful for your work, please kindly cite our paper: latex @article{tong2024dartmath, title={DART-Math: Difficulty-Aware Rejection Tuning for Mathematical Problem-Solving}, author={Yuxuan Tong and Xiwen Zhang and Rui Wang and Ruidong Wu and Junxian He}, year={2024}, eprint={2407.13690}, archivePrefix={arXiv}, primaryClass={cs.CL}, url={https://arxiv.org/abs/2407.13690}, }

This is a dataset for predicting 5-Formylcytidine Modifications on mRNA.

A collection of large languge model responses to tasks of propositional logic. The responses are annotated according to the following criteria: - Inferential strategy employed by the model. Strategies considered are: supposition following, chain construction, compound strategy, concatenation strategy, and the symbolic strategy. Binary labels are assigned to each strategy, indicating whether the strategy is present in the model's response. - Assessment of the validity of the model's final conclusion. Binary labels are assigned to each response, indicating whether the model's conclusion is accurate ("valid_conclusion"). - Evaluation of the soundness of the model's rationale. Binary labels are assigned to each response, indicating whether the rationale provided by the model is sound ("sound_reasoning"). - A description of the model's reasoning error. This information is provided in the form of a string ("reasoning_errors").

JustLogic is a natural language deductive reasoning dataset. JustLogic is (i) highly complex, capable of generating a diverse range of linguistic patterns, vocabulary, and argument structures; (ii) prior knowledge independent, eliminating the advantage of models possessing prior knowledge and ensuring that only deductive reasoning is used to answer questions; and (iii) capable of in-depth error analysis on the heterogeneous effects of reasoning depth and argument form on model accuracy.

NCTE Transcripts consists of 1,660 45-60 minute long 4th and 5th grade elementary mathematics observations collected by the National Center for Teacher Effectiveness (NCTE) between 2010-2013. The anonymized transcripts represent data from 317 teachers across 4 school districts that serve largely historically marginalized students. The transcripts come with rich metadata, including turn-level annotations for dialogic discourse moves, classroom observation scores, demographic information, survey responses and student test scores. Source: The NCTE Transcripts: A Dataset of Elementary Math Classroom Transcripts Image Source: https://arxiv.org/pdf/2211.11772v1.pdf

The ontology files, readme and statistical information can be found and browsed in the ontology library. Because many of the ontologies make use of imports, we have "localised" ontologies by parsing them, resolving and parsing all imports, merging the main and imported ontologies together, and re-serialising the ontology, all using the OWL API. The original main ontology and the imported ontologies are saved in the "sources/". Each localised ontology is assigned a unique ID and is companied with a hard link. All the hard links are stored here. In the readme, there is some information on all the localised ontologies--their IDs, ontology file directories, and statistics.

Panoramic Video Panoptic Segmentation Dataset is a large-scale dataset that offers high-quality panoptic segmentation labels for autonomous driving. The dataset has labels for 28 semantic categories and 2,860 temporal sequences that were captured by five cameras mounted on autonomous vehicles driving in three different geographical locations, leading to a total of 100k labeled camera images. Source: Waymo Open Dataset: Panoramic Video Panoptic Segmentation Image Source: https://arxiv.org/pdf/2206.07704v1.pdf

PolyMATH, a challenging benchmark aimed at evaluating the general cognitive reasoning abilities of MLLMs. PolyMATH comprises 5,000 manually collected high-quality images of cognitive textual and visual challenges across 10 distinct categories, including pattern recognition, spatial reasoning, and relative reasoning Paper Code Project

Recent advances in large language models have led to the development of multimodal LLMs (MLLMs), which take both image data and text as an input. Virtually all of these models have been announced within the past year, leading to a significant need for benchmarks evaluating the abilities of these models to reason truthfully and accurately on a diverse set of tasks. When Google announced Gemini (Gemini Team et al., 2023), they showcased its ability to solve rebuses—wordplay puzzles which involve creatively adding and subtracting letters from words derived from text and images. The diversity of rebuses allows for a broad evaluation of multimodal reasoning capabilities, including image recognition, multi- step reasoning, and understanding the human creator’s intent. We present REBUS: a collection of 333 hand-crafted rebuses spanning 13 diverse cate- gories, including hand-drawn and digital images created by nine contributors. Samples are presented in Table 1. Notably, GPT-4V, the most powerful model we evaluated, answered only 24% of puzzles correctly, highlighting the poor capabilities of MLLMs in new and unex- pected domains to which human reasoning generalizes with comparative ease. Open-source models perform even worse, with a median accuracy below 1%. We notice that models often give faithless explanations, fail to change their minds after an initial approach doesn’t work, and remain highly uncalibrated on their own abilities.

The Winograd schema challenge composes tasks with syntactic ambiguity, which can be resolved with logic and reasoning. Motivation The dataset presents an extended version of a traditional Winograd challenge (Levesque et al., 2012): each sentence contains unresolved homonymy, which can be resolved based on commonsense and reasoning. The Winograd scheme is extendable with the real-life sentences filtered out of the National Corpora with a set of 11 syntactic queries, extracting sentences like "Katya asked Masha if she..." (two possible references to a pronoun), "A change of scenery that..." (Noun phrase & subordinate clause with "that" in the same gender and number), etc. The extraction pipeline can be adjusted to various languages depending on the set of ambiguous syntactic constructions possible. An example in English for illustration purposes: { ‘text’: ‘But then I was glad, because in the end the singer from Turkey who performed something national, although in a modern version, won.’, ‘answer’: ‘singer’, ‘label’: 1, ‘options’: [‘singer’, ‘Turkey’], ‘reference’: ‘who’, ‘homonymia_type’: ‘1.1’, episode: [15], ‘perturbation’ : ‘winograd’ } Data Fields - text: a string containing the sentence text - answer: a string with a candidate for the coreference resolution - options: a list of all the possible candidates present in the text - reference: a string containing an anaphor (a word or phrase that refers back to an earlier word or phrase) - homonymia_type: a float corresponding to the type of the structure with syntactic homonymy - label: an integer, either 0 or 1, indicating whether the homonymy is resolved correctly or not - perturbation: a string containing the name of the perturbation applied to text. If no perturbation was applied, the dataset name is used - episode: a list of episodes in which the instance is used. Only used for the train set Data Splits The dataset consists of a training set with labeled examples and a test set in two configurations: - raw data: includes the original data with no additional sampling - episodes: data is split into evaluation episodes and includes several perturbations of test for robustness evaluation The train and test sets are disjoint with respect to the sentence-candidate answer pairs but may include overlaps in individual sentences and homonymy type. Test Perturbations Each training episode in the dataset corresponds to six test variations, including the original test data and five adversarial test sets, acquired through the modification of the original test through the following text perturbations: - ButterFingers: randomly adds noise to data by mimicking spelling mistakes made by humans through character swaps based on their keyboard distance - Emojify: replaces the input words with the corresponding emojis, preserving their original meaning - EDAdelete: randomly deletes tokens in the text - EDAswap: randomly swaps tokens in the text - AddSent: generates extra words or a sentence at the end of the text