데이터셋 탐색

The NumGLUE dataset is a valuable resource developed by the Allen Institute for AI. It focuses on evaluating the performance of AI systems in mathematical reasoning tasks that involve numbers within natural language text. Here are the key details about NumGLUE: 1. Purpose and Inspiration: - Drawing inspiration from the GLUE benchmark, which was designed for natural language understanding, NumGLUE aims to assess AI systems' ability to reason with numbers. - Unlike GLUE, which covers a wide range of NLP tasks, NumGLUE specifically targets tasks that require simple arithmetic understanding. 2. Tasks: - NumGLUE consists of eight different tasks, each involving numerical reasoning: - Commonsense + Arithmetic Reasoning - Domain Specific + Arithmetic Reasoning - Commonsense + Quantitative Comparison - Fill-in-the-blanks Format - Reading Comprehension (RC) + Explicit Numerical Reasoning - Reading Comprehension (RC) + Implicit Numerical Reasoning 3. Challenges and Performance: - Despite the availability of neural models, including state-of-the-art large-scale language models, NumGLUE remains unsolved. - These models perform significantly worse than humans, with an average gap of 46.4%. - The dataset encourages knowledge sharing across tasks, especially for those with limited training data. Joint training on all tasks yields superior performance. 4. Importance: - NumGLUE promotes the development of systems capable of robust and general arithmetic reasoning within language. - It serves as a stepping stone toward more complex mathematical reasoning. (1) NumGLUE Dataset — Allen Institute for AI. https://allenai.org/data/numglue. (2) GitHub - allenai/numglue: NumGLUE: A Suite of Fundamental yet .... https://github.com/allenai/numglue. (3) nyu-mll/glue · Datasets at Hugging Face. https://huggingface.co/datasets/nyu-mll/glue.

The MNumGLUESub dataset is a multi-task arithmetic reasoning benchmark. It is part of the NumGLUE collection, which focuses on evaluating mathematical reasoning abilities in various languages. Let's break down the details: 1. NumGLUE: This broader benchmark suite includes several tasks related to mathematical reasoning. One of its subsets is MNumGLUESub. 2. MNumGLUESub: - Tasks Selection: To create this dataset, specific tasks from the larger NumGLUE collection were chosen. Specifically, tasks 1, 4, and 8 were selected. - Mathematical World Problems: These tasks involve solving mathematical world problems. The questions in MNumGLUESub cover various mathematical concepts and require reasoning skills. - Translation: The questions from MNumGLUESub were translated into nine different languages using ChatGPT. These languages align with the languages used in the broader GSM8KInstruct dataset. 3. Purpose: MNumGLUESub serves as an evaluation dataset for assessing the mathematical reasoning capabilities of models across different languages. It helps researchers analyze how well models perform on arithmetic reasoning tasks in multilingual settings. For further details, you can refer to the research paper on Advancing Multilingual Reasoning through Multilingual Alignment-as-Preference Optimization (MAPO), which introduces the MAPO framework for enhancing multilingual reasoning abilities ¹². Source: Conversation with Bing, 3/17/2024 (1) MAPO: Advancing Multilingual Reasoning through Multilingual-Alignment .... https://arxiv.org/html/2401.06838v2. (2) MAPO: Advancing Multilingual Reasoning through Multilingual Alignment .... https://arxiv.org/html/2401.06838v1. (3) [2401.06838] MAPO: Advancing Multilingual Reasoning through .... https://arxiv.org/abs/2401.06838.