데이터셋 탐색

General Language Understanding Evaluation (GLUE) benchmark is a collection of nine natural language understanding tasks, including single-sentence tasks CoLA and SST-2, similarity and paraphrasing tasks MRPC, STS-B and QQP, and natural language inference tasks MNLI, QNLI, RTE and WNLI. Source: Align, Mask and Select: A Simple Method for Incorporating Commonsense Knowledge into Language Representation Models Image Source: https://gluebenchmark.com/

SuperGLUE is a benchmark dataset designed to pose a more rigorous test of language understanding than GLUE. SuperGLUE has the same high-level motivation as GLUE: to provide a simple, hard-to-game measure of progress toward general-purpose language understanding technologies for English. SuperGLUE follows the basic design of GLUE: It consists of a public leaderboard built around eight language understanding tasks, drawing on existing data, accompanied by a single-number performance metric, and an analysis toolkit. However, it improves upon GLUE in several ways: - More challenging tasks: SuperGLUE retains the two hardest tasks in GLUE. The remaining tasks were identified from those submitted to an open call for task proposals and were selected based on difficulty for current NLP approaches. - More diverse task formats: The task formats in GLUE are limited to sentence- and sentence-pair classification. The authors expand the set of task formats in SuperGLUE to include coreference resolution and question answering (QA). - Comprehensive human baselines: the authors include human performance estimates for all benchmark tasks, which verify that substantial headroom exists between a strong BERT-based baseline and human performance. - Improved code support: SuperGLUE is distributed with a new, modular toolkit for work on pretraining, multi-task learning, and transfer learning in NLP, built around standard tools including PyTorch (Paszke et al., 2017) and AllenNLP (Gardner et al., 2017). - Refined usage rules: The conditions for inclusion on the SuperGLUE leaderboard were revamped to ensure fair competition, an informative leaderboard, and full credit assignment to data and task creators.

CodeXGLUE is a benchmark dataset and open challenge for code intelligence. It includes a collection of code intelligence tasks and a platform for model evaluation and comparison. CodeXGLUE stands for General Language Understanding Evaluation benchmark for CODE. It includes 14 datasets for 10 diversified code intelligence tasks covering the following scenarios: - code-code (clone detection, defect detection, cloze test, code completion, code repair, and code-to-code translation) - text-code (natural language code search, text-to-code generation) - code-text (code summarization) - text-text (documentation translation) A brief summary of CodeXGLUE is provided in the figure, including tasks, datasets, language, sizes in various states, baseline systems, providers, and short definitions of each task. Datasets highlighted in BLUE are newly introduced. Image source: https://github.com/microsoft/CodeXGLUE

Legal General Language Understanding Evaluation (LexGLUE) benchmark is a collection of datasets for evaluating model performance across a diverse set of legal NLU tasks in a standardized way. Image source: https://arxiv.org/pdf/2110.00976v1.pdf

Adversarial GLUE (AdvGLUE) is a new multi-task benchmark to quantitatively and thoroughly explore and evaluate the vulnerabilities of modern large-scale language models under various types of adversarial attacks. In particular, we systematically apply 14 textual adversarial attack methods to GLUE tasks to construct AdvGLUE, which is further validated by humans for reliable annotations. Description from: Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Models

The Image-Grounded Language Understanding Evaluation (IGLUE) benchmark brings together—by both aggregating pre-existing datasets and creating new ones—visual question answering, cross-modal retrieval, grounded reasoning, and grounded entailment tasks across 20 diverse languages. The benchmark enables the evaluation of multilingual multimodal models for transfer learning, not only in a zero-shot setting, but also in newly defined few-shot learning setups.

The NumGLUE dataset is a valuable resource developed by the Allen Institute for AI. It focuses on evaluating the performance of AI systems in mathematical reasoning tasks that involve numbers within natural language text. Here are the key details about NumGLUE: 1. Purpose and Inspiration: - Drawing inspiration from the GLUE benchmark, which was designed for natural language understanding, NumGLUE aims to assess AI systems' ability to reason with numbers. - Unlike GLUE, which covers a wide range of NLP tasks, NumGLUE specifically targets tasks that require simple arithmetic understanding. 2. Tasks: - NumGLUE consists of eight different tasks, each involving numerical reasoning: - Commonsense + Arithmetic Reasoning - Domain Specific + Arithmetic Reasoning - Commonsense + Quantitative Comparison - Fill-in-the-blanks Format - Reading Comprehension (RC) + Explicit Numerical Reasoning - Reading Comprehension (RC) + Implicit Numerical Reasoning 3. Challenges and Performance: - Despite the availability of neural models, including state-of-the-art large-scale language models, NumGLUE remains unsolved. - These models perform significantly worse than humans, with an average gap of 46.4%. - The dataset encourages knowledge sharing across tasks, especially for those with limited training data. Joint training on all tasks yields superior performance. 4. Importance: - NumGLUE promotes the development of systems capable of robust and general arithmetic reasoning within language. - It serves as a stepping stone toward more complex mathematical reasoning. (1) NumGLUE Dataset — Allen Institute for AI. https://allenai.org/data/numglue. (2) GitHub - allenai/numglue: NumGLUE: A Suite of Fundamental yet .... https://github.com/allenai/numglue. (3) nyu-mll/glue · Datasets at Hugging Face. https://huggingface.co/datasets/nyu-mll/glue.

XGLUE is an evaluation benchmark XGLUE,which is composed of 11 tasks that span 19 languages. For each task, the training data is only available in English. This means that to succeed at XGLUE, a model must have a strong zero-shot cross-lingual transfer capability to learn from the English data of a specific task and transfer what it learned to other languages. Comparing to its concurrent work XTREME, XGLUE has two characteristics: First, it includes cross-lingual NLU and cross-lingual NLG tasks at the same time; Second, besides including 5 existing cross-lingual tasks (i.e. NER, POS, MLQA, PAWS-X and XNLI), XGLUE selects 6 new tasks from Bing scenarios as well, including News Classification (NC), Query-Ad Matching (QADSM), Web Page Ranking (WPR), QA Matching (QAM), Question Generation (QG) and News Title Generation (NTG). Such diversities of languages, tasks and task origin provide a comprehensive benchmark for quantifying the quality of a pre-trained model on cross-lingual natural language understanding and generation.

DialoGLUE is a natural language understanding benchmark for task-oriented dialogue designed to encourage dialogue research in representation-based transfer, domain adaptation, and sample-efficient task learning. It consisting of 7 task-oriented dialogue datasets covering 4 distinct natural language understanding tasks. Source: DialoGLUE: A Natural Language Understanding Benchmark for Task-Oriented Dialogue

We now introduce IndicGLUE, the Indic General Language Understanding Evaluation Benchmark, which is a collection of various NLP tasks as de- scribed below. The goal is to provide an evaluation benchmark for natural language understanding ca- pabilities of NLP models on diverse tasks and mul- tiple Indian languages.

FewGLUE consists of a random selection of 32 training examples from the SuperGLUE training sets and up to 20,000 unlabeled examples for each SuperGLUE task. Source: FewGlue

GLUE-X is a benchmark dataset used to evaluate the out-of-distribution (OOD) robustness of Natural Language Understanding (NLU) models. It was created to address the OOD generalization problem, which remains a challenge in many NLP tasks and limits the real-world deployment of these methods. The GLUE-X dataset consists of 14 publicly available datasets used as OOD test data. Evaluations are conducted on 8 classic NLP tasks over popularly used models. The findings from these evaluations highlight the need for improved OOD accuracy in NLP tasks, as significant performance degradation was observed in all settings compared to in-distribution (ID) accuracy. The creators of GLUE-X hope that this dataset will help highlight the importance of OOD robustness and provide insights on how to measure the robustness of a model and how to improve it.

JGLUE, Japanese General Language Understanding Evaluation, is built to measure the general NLU ability in Japanese.

The ASR-GLUE benchmark is a collection of 6 different NLU (Natural Language Understanding) tasks for evaluating the performance of models under automatic speech recognition (ASR) error across 3 different levels of background noise and 6 speakers with various voice characteristics.

GeoGLUE is a GeoGraphic Language Understanding Evaluation benchmark, which consists of six geographic text-related tasks, including geographic textual similarity on recall, geotagged geographic elements tagging, geographic composition analysis, geographic where what cut, and geographic entity alignment. All tasks' datasets are collected from open-released resources. Source: GeoGLUE: A GeoGraphic Language Understanding Evaluation Benchmark Image Source: GeoGLUE: A GeoGraphic Language Understanding Evaluation Benchmark

Introduction The FewGLUE_64_labeled dataset is a new version of FewGLUE dataset. It contains a 64-sample training set, a development set (the original SuperGLUE development set), a test set, and an unlabeled set. It is constructed to facilitate the research of few-shot learning for natural language understanding tasks. Compared with the original FewGLUE dataset, it differs in the number of labeled data examples in the training set, where the original FewGLUE has 32 training examples while FewGLUE_64_labeled has 64 labeled examples. Purposes for constructing a new version of FewGLUE dataset include: 1. To answer the questions that what is the best performance that few-shot learning can achieve and whether it is possible to further close the performance gap between few-shot learning and fully-supervised systems. 2. To explore to which degree the number of labeled training examples influences the few-shot performance. Please refer to the FewNLU paper as well as the FewNLU leaderboard for more details. ### Acknowledgement Part of the FewGLUE_64_labeled dataset is based on the original 32-sample version of FewGLUE. We collect them together in one package for the convenience of usage. We appreciate all the contributors who made their dataset public, which greatly advanced few-shot learning as well as the FewNLU project.

The MNumGLUESub dataset is a multi-task arithmetic reasoning benchmark. It is part of the NumGLUE collection, which focuses on evaluating mathematical reasoning abilities in various languages. Let's break down the details: 1. NumGLUE: This broader benchmark suite includes several tasks related to mathematical reasoning. One of its subsets is MNumGLUESub. 2. MNumGLUESub: - Tasks Selection: To create this dataset, specific tasks from the larger NumGLUE collection were chosen. Specifically, tasks 1, 4, and 8 were selected. - Mathematical World Problems: These tasks involve solving mathematical world problems. The questions in MNumGLUESub cover various mathematical concepts and require reasoning skills. - Translation: The questions from MNumGLUESub were translated into nine different languages using ChatGPT. These languages align with the languages used in the broader GSM8KInstruct dataset. 3. Purpose: MNumGLUESub serves as an evaluation dataset for assessing the mathematical reasoning capabilities of models across different languages. It helps researchers analyze how well models perform on arithmetic reasoning tasks in multilingual settings. For further details, you can refer to the research paper on Advancing Multilingual Reasoning through Multilingual Alignment-as-Preference Optimization (MAPO), which introduces the MAPO framework for enhancing multilingual reasoning abilities ¹². Source: Conversation with Bing, 3/17/2024 (1) MAPO: Advancing Multilingual Reasoning through Multilingual-Alignment .... https://arxiv.org/html/2401.06838v2. (2) MAPO: Advancing Multilingual Reasoning through Multilingual Alignment .... https://arxiv.org/html/2401.06838v1. (3) [2401.06838] MAPO: Advancing Multilingual Reasoning through .... https://arxiv.org/abs/2401.06838.