Triangulating LLM Progress through Benchmarks, Games, and Cognitive Tests
원문 초록
We examine three evaluation paradigms: large question-answering benchmarks (e.g., MMLU and BBH), interactive games (e.g., Signalling Games or Taboo), and cognitive tests (e.g., for working memory or theory of mind). First, we investigate which of the former two-benchmarks or games-is most effective at discriminating LLMs of varying quality. Then, inspired by human cognitive assessments, we compile a suite of targeted tests that measure cognitive abilities deemed essential for effective language use, and we investigate their correlation with model performance in benchmarks and games. Our analyses reveal that interactive games are superior to standard benchmarks in discriminating models. Causal and logical reasoning correlate with both static and interactive tests, while differences emerge regarding core executive functions and social/emotional skills, which correlate more with games. We advocate the development of new interactive benchmarks and targeted cognitive tasks inspired by assessing human abilities but designed specifically for LLMs.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Interleaved Reasoning for Large Language Models via Reinforcement Learning
Climate Finance Bench
CAPO: Reinforcing Consistent Reasoning in Medical Decision-Making
CP-Router: An Uncertainty-Aware Router Between LLM and LRM
LogiQA 2.0—An Improved Dataset for Logical Reasoning in Natural Language Understanding
From Roots to Rewards: Dynamic Tree Reasoning with RL
토론
이 논문과 연결된 포럼 글이 아직 없습니다.