What do we expect from Multiple-choice QA Systems?
원문 초록
The recent success of machine learning systems on various QA datasets could be interpreted as a significant improvement in models' language understanding abilities. However, using various perturbations, multiple recent works have shown that good performance on a dataset might not indicate performance that correlates well with human's expectations from models that "understand" language. In this work we consider a top performing model on several Multiple Choice Question Answering (MCQA) datasets, and evaluate it against a set of expectations one might have from such a model, using a series of zero-information perturbations of the model's inputs. Our results show that the model clearly falls short of our expectations, and motivates a modified training approach that forces the model to better attend to the inputs. We show that the new training paradigm leads to a model that performs on par with the original model while better satisfying our expectations.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
CP-Router: An Uncertainty-Aware Router Between LLM and LRM
Training-free LLM Merging for Multi-task Learning
VAU-R1: Advancing Video Anomaly Understanding via Reinforcement Fine-Tuning
How Far Can Off-the-Shelf Multimodal Large Language Models Go in Online Episodic Memory Question Answering?
WikiMixQA: A Multimodal Benchmark for Question Answering over Tables and Charts
Instruction Tuning and CoT Prompting for Contextual Medical QA with LLMs
토론
이 논문과 연결된 포럼 글이 아직 없습니다.