Papers with Code · CC BY-SA 4.0

데이터셋 탐색

데이터셋의 설명·홈페이지를 찾고, 관련 태스크의 평가 기록을 살펴보세요.

관련 논문 10편

CriticBench

CriticBench 설명 보기

CriticBench is a comprehensive benchmark designed to assess the abilities of Large Language Models (LLMs) to critique and rectify their reasoning across various tasks. It encompasses five reasoning domains: 1. Mathematical 2. Commonsense 3. Symbolic 4. Coding 5. Algorithmic CriticBench compiles 15 datasets and incorporates responses from three LLM families. By utilizing CriticBench, researchers evaluate and dissect the performance of 17 LLMs in generation, critique, and correction reasoning (referred to as GQC reasoning). Notable findings include: 1. A linear relationship in GQC capabilities, with critique-focused training significantly enhancing performance. 2. Task-dependent variation in correction effectiveness, with logic-oriented tasks being more amenable to correction. 3. GQC knowledge inconsistencies that decrease as model size increases. 4. An intriguing inter-model critiquing dynamic, where stronger models excel at critiquing weaker ones, while weaker models surprisingly surpass stronger ones in self-critique. (1) CriticBench: Benchmarking LLMs for Critique-Correct Reasoning. https://arxiv.org/abs/2402.14809. (2) CriticBench: Benchmarking LLMs for Critique-Correct Reasoning. http://export.arxiv.org/abs/2402.14809. (3) CriticBench: Benchmarking LLMs for Critique-Correct Reasoning. https://openreview.net/forum?id=sc5i7q6DQO. (4) CriticBench: Benchmarking LLMs for Critique-Correct Reasoning - arXiv.org. https://arxiv.org/html/2402.14809v2. (5) undefined. https://doi.org/10.48550/arXiv.2402.14809.

관련 논문 1편

open-compass/CriticBench

open-compass/CriticBench 설명 보기

arXiv license [Dataset on HF] [Project Page] [Subjective LeaderBoard] [Objective LeaderBoard] CriticBench is a novel benchmark designed to comprehensively and reliably evaluate the critique abilities of Large Language Models (LLMs). These critique abilities are crucial for scalable oversight and self-improvement of LLMs. While many recent studies explore how LLMs can judge and refine flaws in their generated outputs, the measurement of critique abilities remains under-explored. Here are the key aspects of CriticBench: 1. Purpose: To assess LLMs' critique abilities across four dimensions: - Feedback: How well an LLM provides constructive feedback. - Comparison: The ability to compare and contrast different responses. - Refinement: How effectively an LLM can refine flawed or suboptimal outputs. - Meta-feedback: The LLM's ability to reflect on its own performance. 2. Tasks: CriticBench encompasses nine diverse tasks, each evaluating LLMs' critique abilities at varying levels of quality granularity. 3. Evaluation: The benchmark evaluates both open-source and closed-source LLMs, revealing intriguing relationships between critique abilities, response qualities, and model scales. 4. Resources: Datasets, resources, and an evaluation toolkit for CriticBench will be publicly released. In summary, CriticBench aims to provide a comprehensive framework for assessing LLMs' critique and self-improvement capabilities, contributing to the advancement of large-scale language models in various applications.