Detecting Benchmark Contamination Through Watermarking
원문 초록
Benchmark contamination poses a significant challenge to the reliability of Large Language Models (LLMs) evaluations, as it is difficult to assert whether a model has been trained on a test set. We introduce a solution to this problem by watermarking benchmarks before their release. The embedding involves reformulating the original questions with a watermarked LLM, in a way that does not alter the benchmark utility. During evaluation, we can detect ``radioactivity'', \ie traces that the text watermarks leave in the model during training, using a theoretically grounded statistical test. We test our method by pre-training 1B models from scratch on 10B tokens with controlled benchmark contamination, and validate its effectiveness in detecting contamination on ARC-Easy, ARC-Challenge, and MMLU. Results show similar benchmark utility post-watermarking and successful contamination detection when models are contaminated enough to enhance performance, e.g. $p$-val $=10^{-3}$ for +5$\%$ on ARC-Easy.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
HELM: Hyperbolic Large Language Models via Mixture-of-Curvature Experts
TODO: Enhancing LLM Alignment with Ternary Preferences
From Threat to Tool: Leveraging Refusal-Aware Injection Attacks for Safety Alignment
Self-Reasoning Language Models: Unfold Hidden Reasoning Chains with Few Reasoning Catalyst
Elastic Weight Consolidation for Full-Parameter Continual Pre-Training of Gemma2
DataDecide: How to Predict Best Pretraining Data with Small Experiments
토론
이 논문과 연결된 포럼 글이 아직 없습니다.