Who breaks early, looses: goal oriented training of deep neural networks based on port Hamiltonian dynamics
원문 초록
The highly structured energy landscape of the loss as a function of parameters for deep neural networks makes it necessary to use sophisticated optimization strategies in order to discover (local) minima that guarantee reasonable performance. Overcoming less suitable local minima is an important prerequisite and often momentum methods are employed to achieve this. As in other non local optimization procedures, this however creates the necessity to balance between exploration and exploitation. In this work, we suggest an event based control mechanism for switching from exploration to exploitation based on reaching a predefined reduction of the loss function. As we give the momentum method a port Hamiltonian interpretation, we apply the 'heavy ball with friction' interpretation and trigger breaking (or friction) when achieving certain goals. We benchmark our method against standard stochastic gradient descent and provide experimental evidence for improved performance of deep neural networks when our strategy is applied.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Frictional Agent Alignment Framework: Slow Down and Don't Break Things
Physics-Informed Deep Learning for Nonlinear Friction Model of Bow-string Interaction
PyScrew: A Comprehensive Dataset Collection from Industrial Screw Driving Experiments
Barrier-Free Microhabitats: Self-Organized Seclusion in Microbial Communities
PUGS: Zero-shot Physical Understanding with Gaussian Splatting
Know Your Mistakes: Towards Preventing Overreliance on Task-Oriented Conversational AI Through Accountability Modeling
토론
이 논문과 연결된 포럼 글이 아직 없습니다.