Undivided Attention: Are Intermediate Layers Necessary for BERT?
원문 초록
In recent times, BERT-based models have been extremely successful in solving a variety of natural language processing (NLP) tasks such as reading comprehension, natural language inference, sentiment analysis, etc. All BERT-based architectures have a self-attention block followed by a block of intermediate layers as the basic building component. However, a strong justification for the inclusion of these intermediate layers remains missing in the literature. In this work we investigate the importance of intermediate layers on the overall network performance of downstream tasks. We show that reducing the number of intermediate layers and modifying the architecture for BERT-BASE results in minimal loss in fine-tuning accuracy for downstream tasks while decreasing the number of parameters and training time of the model. Additionally, we use centered kernel alignment and probing linear classifiers to gain insight into our architectural modifications and justify that removal of intermediate layers has little impact on the fine-tuned accuracy.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
DeFTX: Denoised Sparse Fine-Tuning for Zero-Shot Cross-Lingual Transfer
SALAD: Improving Robustness and Generalization through Contrastive Learning with Structure-Aware and LLM-Driven Augmented Data
LogiQA 2.0—An Improved Dataset for Logical Reasoning in Natural Language Understanding
DIBERT: Dependency Injected Bidirectional Encoder Representations from Transformers
Text-Embedded Bilinear Model for Fine-Grained Visual Recognition
Understanding and Tackling Label Errors in Individual-Level Nature Language Understanding
토론
이 논문과 연결된 포럼 글이 아직 없습니다.