Mitigating Shortcuts in Language Models with Soft Label Encoding
원문 초록
Recent research has shown that large language models rely on spurious correlations in the data for natural language understanding (NLU) tasks. In this work, we aim to answer the following research question: Can we reduce spurious correlations by modifying the ground truth labels of the training data? Specifically, we propose a simple yet effective debiasing framework, named Soft Label Encoding (SoftLE). We first train a teacher model with hard labels to determine each sample's degree of relying on shortcuts. We then add one dummy class to encode the shortcut degree, which is used to smooth other dimensions in the ground truth label to generate soft labels. This new ground truth label is used to train a more robust student model. Extensive experiments on two NLU benchmark tasks demonstrate that SoftLE significantly improves out-of-distribution generalization while maintaining satisfactory in-distribution accuracy.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Vision Language Action Models in Robotic Manipulation: A Systematic Review
A Survey on Vision-Language-Action Models for Autonomous Driving
Dialect Normalization using Large Language Models and Morphological Rules
Positional Encoding meets Persistent Homology on Graphs
Pruning Spurious Subgraphs for Graph Out-of-Distribtuion Generalization
EdgeProfiler: A Fast Profiling Framework for Lightweight LLMs on Edge Using Analytical Model
토론
이 논문과 연결된 포럼 글이 아직 없습니다.