Enhancing Sharpness-Aware Minimization by Learning Perturbation Radius
원문 초록
Sharpness-aware minimization (SAM) is to improve model generalization by searching for flat minima in the loss landscape. The SAM update consists of one step for computing the perturbation and the other for computing the update gradient. Within the two steps, the choice of the perturbation radius is crucial to the performance of SAM, but finding an appropriate perturbation radius is challenging. In this paper, we propose a bilevel optimization framework called LEarning the perTurbation radiuS (LETS) to learn the perturbation radius for sharpness-aware minimization algorithms. Specifically, in the proposed LETS method, the upper-level problem aims at seeking a good perturbation radius by minimizing the squared generalization gap between the training and validation losses, while the lower-level problem is the SAM optimization problem. Moreover, the LETS method can be combined with any variant of SAM. Experimental results on various architectures and benchmark datasets in computer vision and natural language processing demonstrate the effectiveness of the proposed LETS method in improving the performance of SAM.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
GuiLoMo: Allocating Expert Number and Rank for LoRA-MoE via Bilevel Optimization with GuidedSelection Vectors
MISLEADER: Defending against Model Extraction with Ensembles of Distilled Models
Learning Where to Learn: Training Distribution Selection for Provable OOD Performance
GCAL: Adapting Graph Models to Evolving Domain Shifts
HBO: Hierarchical Balancing Optimization for Fine-Tuning Large Language Models
Optimizing Decomposition for Optimal Claim Verification
토론
이 논문과 연결된 포럼 글이 아직 없습니다.