Efficient and Private: Memorisation under differentially private parameter-efficient fine-tuning in language models
원문 초록
Fine-tuning large language models (LLMs) for specific tasks introduces privacy risks, as models may inadvertently memorise and leak sensitive training data. While Differential Privacy (DP) offers a solution to mitigate these risks, it introduces significant computational and performance trade-offs, particularly with standard fine-tuning approaches. Previous work has primarily focused on full-parameter updates, which are computationally intensive and may not fully leverage DPs potential in large models. In this work, we address these shortcomings by investigating Parameter-Efficient Fine-Tuning (PEFT) methods under DP constraints. We show that PEFT methods achieve comparable performance to standard fine-tuning while requiring fewer parameters and significantly reducing privacy leakage. Furthermore, we incorporate a data poisoning experiment involving intentional mislabelling to assess model memorisation and directly measure privacy risks. Our findings indicate that PEFT methods not only provide a promising alternative but also serve as a complementary approach for privacy-preserving, resource-efficient fine-tuning of LLMs.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Gradient Inversion Attacks on Parameter-Efficient Fine-Tuning
FedVLMBench: Benchmarking Federated Fine-Tuning of Vision-Language Models
Devil's Hand: Data Poisoning Attacks to Locally Private Graph Learning Protocols
Assortment of Attention Heads: Accelerating Federated PEFT with Head Pruning and Strategic Client Selection
Privacy Preserving Conversion Modeling in Data Clean Room
Federated Adapter on Foundation Models: An Out-Of-Distribution Approach
토론
이 논문과 연결된 포럼 글이 아직 없습니다.