Contrastive Entropy: A new evaluation metric for unnormalized language models
원문 초록
Perplexity (per word) is the most widely used metric for evaluating language models. Despite this, there has been no dearth of criticism for this metric. Most of these criticisms center around lack of correlation with extrinsic metrics like word error rate (WER), dependence upon shared vocabulary for model comparison and unsuitability for unnormalized language model evaluation. In this paper, we address the last problem and propose a new discriminative entropy based intrinsic metric that works for both traditional word level models and unnormalized language models like sentence level models. We also propose a discriminatively trained sentence level interpretation of recurrent neural network based language model (RNN) as an example of unnormalized sentence level model. We demonstrate that for word level models, contrastive entropy shows a strong correlation with perplexity. We also observe that when trained at lower distortion levels, sentence level RNN considerably outperforms traditional RNNs on this new metric.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
NeuralNexus at BEA 2025 Shared Task: Retrieval-Augmented Prompting for Mistake Identification in AI Tutors
Enterprise Large Language Model Evaluation Benchmark
Finance Language Model Evaluation (FLaME)
ConECT Dataset: Overcoming Data Scarcity in Context-Aware E-Commerce MT
DIBERT: Dependency Injected Bidirectional Encoder Representations from Transformers
A Bit of Progress in Language Modeling
토론
이 논문과 연결된 포럼 글이 아직 없습니다.