Pseudointelligence: A Unifying Framework for Language Model Evaluation
원문 초록
With large language models surpassing human performance on an increasing number of benchmarks, we must take a principled approach for targeted evaluation of model capabilities. Inspired by pseudorandomness, we propose pseudointelligence, which captures the maxim that "(perceived) intelligence lies in the eye of the beholder". That is, that claims of intelligence are meaningful only when their evaluator is taken into account. Concretely, we propose a complexity-theoretic framework of model evaluation cast as a dynamic interaction between a model and a learned evaluator. We demonstrate that this framework can be used to reason about two case studies in language model evaluation, as well as analyze existing evaluation methods.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Enterprise Large Language Model Evaluation Benchmark
Finance Language Model Evaluation (FLaME)
Specification-based Security Evaluation of Real-World Agents Tool-Calling under Long-Horizon Context Tasks
Making Language Model a Hierarchical Classifier and Generator
InstructFLIP: Exploring Unified Vision-Language Model for Face Anti-spoofing
Describe Anything Model for Visual Question Answering on Text-rich Images
토론
이 논문과 연결된 포럼 글이 아직 없습니다.