LARC: Low-Rank Adaptive Residual Connections for Learning in Frozen Models

원문 초록

Low-Rank Adaptive Residual Connections (LARC) give a frozen model a compact numerical state that can learn from feedback. The map $h+BAh$ changes the hidden input to a pretrained mixer. An episode copies a learned initialization $\\rho$ into a private state $\\Phi$, updates its factors when feedback arrives, and resets to the bound $\\rho$. We analyze the factor-space updates and trace this input-side policy carrier into Orbit, the post-training realization of Memory-Mediated Learning Architecture studied here. In Orbit's released default, the learned residual shapes predictions while memory adaptation changes the retained evidence.\n\nThe controlled studies use a rank-4 residual with 12,288 factor entries on frozen MiniCPM5-1B-SFT. Two feedback updates reduce expected query execution error by 24.65 and 36.65 percentage points relative to the respective trained reset states in a four-candidate development task with 16 parameter groups and three paired seeds. A direct support-loss rule reaches 0.78125\\% error. In repository-balanced chronological replay of public continuous-integration jobs, retaining updates raises half-Brier loss from 0.1274 to 0.1808. A follow-up intervention observes same-batch non-descent and inconsistent future benefit from shrinking steps. These comparisons measure when numerical state helps a later read and when the trained reference is more effective.