From Robustness to Improved Generalization and Calibration in Pre-trained Language Models
원문 초록
Enhancing generalization and uncertainty quantification in pre-trained language models (PLMs) is crucial for their effectiveness and reliability. Building on machine learning research that established the importance of robustness for improving generalization, we investigate the role of representation smoothness, achieved via Jacobian and Hessian regularization, in enhancing PLM performance. Although such regularization methods have proven effective in computer vision, their application in natural language processing (NLP), where PLM inputs are derived from a discrete domain, poses unique challenges. We introduce a novel two-phase regularization approach, JacHess, which minimizes the norms of the Jacobian and Hessian matrices within PLM intermediate representations relative to their inputs. Our evaluation using the GLUE benchmark demonstrates that JacHess significantly improves in-domain generalization and calibration in PLMs, outperforming unregularized fine-tuning and other similar regularization methods.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
From Physics to Foundation Models: A Review of AI-Driven Quantitative Remote Sensing Inversion
Statistical Machine Learning for Astronomy -- A Textbook
InstructFLIP: Exploring Unified Vision-Language Model for Face Anti-spoofing
Interpretable Bayesian Tensor Network Kernel Machines with Automatic Rank and Feature Selection
Estimating prevalence with precision and accuracy
Prompt-Free Conditional Diffusion for Multi-object Image Augmentation
토론
이 논문과 연결된 포럼 글이 아직 없습니다.