Beyond Static Models and Test Sets: Benchmarking the Potential of Pre-trained Models Across Tasks and Languages
원문 초록
Although recent Massively Multilingual Language Models (MMLMs) like mBERT and XLMR support around 100 languages, most existing multilingual NLP benchmarks provide evaluation data in only a handful of these languages with little linguistic diversity. We argue that this makes the existing practices in multilingual evaluation unreliable and does not provide a full picture of the performance of MMLMs across the linguistic landscape. We propose that the recent work done in Performance Prediction for NLP tasks can serve as a potential solution in fixing benchmarking in Multilingual NLP by utilizing features related to data and language typology to estimate the performance of an MMLM on different languages. We compare performance prediction with translating test data with a case study on four different multilingual datasets, and observe that these methods can provide reliable estimates of the performance that are often on-par with the translation based approaches, without the need for any additional translation as well as evaluation costs.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Benchmarking histopathology foundation models in a multi-center dataset for skin cancer subtyping
Statistical Multicriteria Evaluation of LLM-Generated Text
Visual Place Recognition for Large-Scale UAV Applications
TransLaw: Benchmarking Large Language Models in Multi-Agent Simulation of the Collaborative Translation
GRAIL: A Benchmark for GRaph ActIve Learning in Dynamic Sensing Environments
Summarizing Videos using Concentrated Attention and Considering the Uniqueness and Diversity of the Video Frames
토론
이 논문과 연결된 포럼 글이 아직 없습니다.