Papers with Code · CC BY-SA 4.0

데이터셋 탐색

데이터셋의 설명·홈페이지를 찾고, 관련 태스크의 평가 기록을 살펴보세요.

관련 논문 22편

EXAMS

EXAMS 설명 보기

A new benchmark dataset for cross-lingual and multilingual question answering for high school examinations. Collects more than 24,000 high-quality high school exam questions in 16 languages, covering 8 language families and 24 school subjects from Natural Sciences and Social Sciences, among others. EXAMS offers a fine-grained evaluation framework across multiple languages and subjects, which allows precise analysis and comparison of various models. Source: EXAMS: A Multi-Subject High School Examinations Dataset for Cross-Lingual and Multilingual Question Answering

관련 논문 3편

OAB Exams

OAB Exams 설명 보기

The OAB Exams dataset is a valuable resource used in the context of legal information systems. In Brazil, all legal professionals must demonstrate their knowledge of the law and its application by passing the OAB exams, which are the national bar exams. These exams serve as a benchmark for evaluating the performance of legal information systems. If a system can achieve a level of legal reasoning comparable to that of a human lawyer who successfully passes the OAB exam, it indicates significant progress. Here are some key details about the OAB Exams dataset: 1. Purpose: The dataset is designed to address the problem of finding justifications for the answers to legal questions posed in the OAB exams. 2. Construction: Researchers have constructed this dataset to facilitate experiments related to legal reasoning. It provides a foundation for evaluating the performance of legal information systems. 3. Baseline Performance: Preliminary experiments using this dataset have established a baseline performance measure. Future improvements can be evaluated against this baseline. 4. Challenges: Despite its importance, achieving high performance on this dataset remains challenging due to the complexity of legal reasoning. Source: Conversation with Bing, 3/16/2024 (1) Passing the Brazilian OAB Exam: data preparation and some experiments. https://paperswithcode.com/paper/passing-the-brazilian-oab-exam-data. (2) eduagarcia/oab_exams at main - Hugging Face. https://huggingface.co/datasets/eduagarcia/oab_exams/tree/main. (3) eduagarcia-temp/llm_pt_leaderboard_requests · Datasets at Hugging Face. https://huggingface.co/datasets/eduagarcia-temp/llm_pt_leaderboard_requests.

관련 논문 2편

Kaleidoscope: In-language Exams for Massively Multilingual Vision Evaluation

Kaleidoscope
Kaleidoscope 설명 보기

The evaluation of vision-language models (VLMs) has mainly relied on English-language benchmarks, leaving significant gaps in both multilingual and multicultural coverage. While multilingual benchmarks have expanded, both in size and languages, many rely on translations of English datasets, failing to capture cultural nuances. In this work, we propose Kaleidoscope, as the most comprehensive exam benchmark to date for the multilingual evaluation of vision-language models. Kaleidoscope is a large-scale, in-language multimodal benchmark designed to evaluate VLMs across diverse languages and visual inputs. Kaleidoscope covers 18 languages and 14 different subjects, amounting to a total of 20,911 multiple-choice questions. Built through an open science collaboration with a diverse group of researchers worldwide, Kaleidoscope ensures linguistic and cultural authenticity. We evaluate top-performing multilingual vision-language models and find that they perform poorly on low-resource languages and in complex multimodal scenarios. Our results highlight the need for progress on culturally inclusive multimodal evaluation frameworks. Summary: 18 languages (Bengali → Spanish), 14 subjects (STEM to Humanities), 20,911 questions (55% requiring image understanding