What Would Elsa Do? Freezing Layers During Transformer Fine-Tuning
원문 초록
Pretrained transformer-based language models have achieved state of the art across countless tasks in natural language processing. These models are highly expressive, comprising at least a hundred million parameters and a dozen layers. Recent evidence suggests that only a few of the final layers need to be fine-tuned for high quality on downstream tasks. Naturally, a subsequent research question is, "how many of the last layers do we need to fine-tune?" In this paper, we precisely answer this question. We examine two recent pretrained language models, BERT and RoBERTa, across standard tasks in textual entailment, semantic similarity, sentiment analysis, and linguistic acceptability. We vary the number of final layers that are fine-tuned, then study the resulting change in task-specific effectiveness. We show that only a fourth of the final layers need to be fine-tuned to achieve 90% of the original quality. Surprisingly, we also find that fine-tuning all layers does not always help.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
DIBERT: Dependency Injected Bidirectional Encoder Representations from Transformers
GATE: General Arabic Text Embedding for Enhanced Semantic Textual Similarity with Matryoshka Representation Learning and Hybrid Loss Training
CrosGrpsABS: Cross-Attention over Syntactic and Semantic Graphs for Aspect-Based Sentiment Analysis in a Low-Resource Language
Omni TM-AE: A Scalable and Interpretable Embedding Model Using the Full Tsetlin Machine State Space
Estimating Quality in Therapeutic Conversations: A Multi-Dimensional Natural Language Processing Framework
FA: Forced Prompt Learning of Vision-Language Models for Out-of-Distribution Detection
토론
이 논문과 연결된 포럼 글이 아직 없습니다.