MoRAL: MoE Augmented LoRA for LLMs' Lifelong Learning
원문 초록
Adapting large language models (LLMs) to new domains/tasks and enabling them to be efficient lifelong learners is a pivotal challenge. In this paper, we propose MoRAL, i.e., Mixture-of-Experts augmented Low-Rank Adaptation for Lifelong Learning. MoRAL combines the multi-tasking abilities of MoE with the fine-tuning abilities of LoRA for effective life-long learning of LLMs. In contrast to the conventional approaches that use factual triplets as inputs MoRAL relies on simple question-answer pairs, which is a more practical and effective strategy for robust and efficient learning. Owing to new data settings, we introduce a new evaluation benchmark namely: Life Long Learning of LLM (5L-bench) encompassing a newly curated dataset of question-answer pairs, and a set of evaluation metrics for rigorous evaluation of MoRAL in open-book and closed-book settings. Experimental evaluation shows (i) LLMs learn fast in open-book settings with up to 30.15% improvement in "RA" for Phi-2-2.7B compared to closed-book (for models fine-tuned with MoRAL); (ii) MoRAL shows higher performance improvement for models with a greater number of parameters; (iii) MoRAL is robust to catastrophic forgetting offering better knowledge retention compared to baselines.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
MoFE-Time: Mixture of Frequency Domain Experts for Time-Series Forecasting Models
A Survey on Prompt Tuning
Growing Transformers: Modular Composition and Layer-wise Expansion on a Frozen Substrate
Learning Robust Stereo Matching in the Wild with Selective Mixture-of-Experts
Learning to Skip the Middle Layers of Transformers
Latent Prototype Routing: Achieving Near-Perfect Load Balancing in Mixture-of-Experts
토론
이 논문과 연결된 포럼 글이 아직 없습니다.