Break a Lag: Triple Exponential Moving Average for Enhanced Optimization
원문 초록
The performance of deep learning models is critically dependent on sophisticated optimization strategies. While existing optimizers have shown promising results, many rely on first-order Exponential Moving Average (EMA) techniques, which often limit their ability to track complex gradient trends accurately. This fact can lead to a significant lag in trend identification and suboptimal optimization, particularly in highly dynamic gradient behavior. To address this fundamental limitation, we introduce Fast Adaptive Moment Estimation (FAME), a novel optimization technique that leverages the power of Triple Exponential Moving Average. By incorporating an advanced tracking mechanism, FAME enhances responsiveness to data dynamics, mitigates trend identification lag, and optimizes learning efficiency. Our comprehensive evaluation encompasses different computer vision tasks including image classification, object detection, and semantic segmentation, integrating FAME into 30 distinct architectures ranging from lightweight CNNs to Vision Transformers. Through rigorous benchmarking against state-of-the-art optimizers, FAME demonstrates superior accuracy and robustness. Notably, it offers high scalability, delivering substantial improvements across diverse model complexities, architectures, tasks, and benchmarks.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
A Robust Feature Downsampling Module for Remote Sensing Visual Tasks
MUPAX: Multidimensional Problem Agnostic eXplainable AI
Hashed Watermark as a Filter: Defeating Forging and Overwriting Attacks in Weight-based Neural Network Watermarking
Automatic Classification and Segmentation of Tunnel Cracks Based on Deep Learning and Visual Explanations
Federated Learning for Commercial Image Sources
Adversarial attacks to image classification systems using evolutionary algorithms
토론
이 논문과 연결된 포럼 글이 아직 없습니다.