PyTorchKR Papers

업데이트된 논문

총 3편
태그·기간 등 상세 필터

카테고리 탐색

로그인 후 이 태그 구독

arXiv:2602.19143v3cs.LG

트랜스포머에서 희소 어텐션 패턴의 점진적 학습

Incremental Learning of Sparse Attention Patterns in Transformers

Transformers integrate information from multiple past positions through sparse attention patterns that copy from specific locations. We study how these patterns form during training in a high-order Markov-chain setting, where the next token depends on several groups of past positions with differing statistical importance. Learning proceeds in discrete stages, each marked by the emergence of a sparse attention pattern for the next most important group. The dynamics are initially competitive, with all attention heads focusing on the dominant pattern, but later become cooperative, with different heads specializing in distinct patterns. For a tractable surrogate, the dynamics reduce to those of a tensor factorization problem, enabling us to prove stagewise convergence. Empirically, the number of stages completed depends on training set size, suggesting a form of implicit complexity control through early stopping. Overall, we give a theoretical account of how structured attention patterns and head specialization emerge without an explicit curriculum.

arXiv:2610.02283v2cs.LG

MLoRA:Continual Low-Rank Adaptation via Relative-Response Subspace Selection and Momentum Orthogonalization

Low-rank adaptation (LoRA) provides a parameter-efficient approach to continual learning, but its nominal rank can conceal a loss of effective adaptation capacity. We identify \emph{spectral plasticity collapse}: during sequential adaptation, update energy becomes concentrated in a small subset of singular modes, leaving much of the available low-rank space underutilized. This exposes a limitation of interference avoidance alone: protecting historical representations does not ensure that the remaining adaptation capacity is responsive to new tasks or effectively utilized. To address this problem, we propose \texttt{MLoRA}, which jointly controls capacity allocation and utilization. First, historical whitening identifies input directions with strong current-task response relative to accumulated historical response, yielding a task-adaptive basis that remains fixed during training. Second, approximate polar orthogonalization of momentum updates reduces spectral concentration within theselected space. An orthonormal basis connects these mechanisms by transferring the factor-update spectrum exactly tothe induced weight update. We establish a max--min characterization of exact subspace selection and derive cumulative spectral bounds under controlled cross-step anisotropy. Across five class-incremental benchmarks and eight incremental settings, \texttt{MuLoRA} achieves the highest mean accuracy in 15 of 16 reported metrics.

arXiv:2606.30266v2cs.LG

지속적 모션-언어 에이전트를 향하여: 점진적 모션 이해 및 생성을 위한 LoRA 변형

Towards Continual Motion-Language Agents: LoRA Variants for Incremental Motion Understanding and Generation

동적 환경에서 작동하는 모션-언어 에이전트의 파국적 망각(catastrophic forgetting) 문제를 해결하기 위해, 고정된 LLM 백본 위에 오토인코더 기반 라우터를 활용한 전문가 혼합(MoE) LoRA 변형을 제안하여 태스크 레이블 없이 추론 시 태스크별 전문가를 선택한다. HumanML3D 기반 5개 태스크 벤치마크에서 M2T·T2M 양방향 모두 망각이 거의 없으며, 소프트 혼합보다 하드 라우팅 방식이 생성 품질 지표에서 유의미하게 우수함을 실험적으로 입증했다.