Low-Rank Adapters Meet Neural Architecture Search for LLM Compression
원문 초록
The rapid expansion of Large Language Models (LLMs) has posed significant challenges regarding the computational resources required for fine-tuning and deployment. Recent advancements in low-rank adapters have demonstrated their efficacy in parameter-efficient fine-tuning (PEFT) of these models. This retrospective paper comprehensively discusses innovative approaches that synergize low-rank representations with Neural Architecture Search (NAS) techniques, particularly weight-sharing super-networks. Robust solutions for compressing and fine-tuning large pre-trained models are developed by integrating these methodologies. Our analysis highlights the potential of these combined strategies to democratize the use of LLMs, making them more accessible for deployment in resource-constrained environments. The resulting models exhibit reduced memory footprints and faster inference times, paving the way for more practical and scalable applications of LLMs. Models and code are available at https://github.com/IntelLabs/Hardware-Aware-Automated-Machine-Learning.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
LoSiA: Efficient High-Rank Fine-Tuning via Subnet Localization and Optimization
Detecting Referring Expressions in Visually Grounded Dialogue with Autoregressive Language Models
AnalogNAS-Bench: A NAS Benchmark for Analog In-Memory Computing
From Tiny Machine Learning to Tiny Deep Learning: A Survey
GuiLoMo: Allocating Expert Number and Rank for LoRA-MoE via Bilevel Optimization with GuidedSelection Vectors
AR-RAG: Autoregressive Retrieval Augmentation for Image Generation
토론
이 논문과 연결된 포럼 글이 아직 없습니다.