Towards Better Few-Shot and Finetuning Performance with Forgetful Causal Language Models
원문 초록
Large language models (LLM) trained using the next-token-prediction objective, such as GPT3 and PaLM, have revolutionized natural language processing in recent years by showing impressive zero-shot and few-shot capabilities across a wide range of tasks. In this work, we propose a simple technique that significantly boosts the performance of LLMs without adding computational cost. Our key observation is that, by performing the next token prediction task with randomly selected past tokens masked out, we can improve the quality of the learned representations for downstream language understanding tasks. We hypothesize that randomly masking past tokens prevents over-attending to recent tokens and encourages attention to tokens in the distant past. We find that our method, Forgetful Causal Masking (FCM), significantly improves both few-shot and finetuning performance of PaLM. We further consider a simple extension, T-FCM, which introduces bidirectional context to causal language model without altering the sequence order, and further improves finetuning performance.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
DIBERT: Dependency Injected Bidirectional Encoder Representations from Transformers
Improving Large Language Model Safety with Contrastive Representation Learning
LRCTI: A Large Language Model-Based Framework for Multi-Step Evidence Retrieval and Reasoning in Cyber Threat Intelligence Credibility Verification
Audio Flamingo 3: Advancing Audio Intelligence with Fully Open Large Audio Language Models
SEED: A Structural Encoder for Embedding-Driven Decoding in Time Series Prediction with LLMs
GeoRecon: Graph-Level Representation Learning for 3D Molecules via Reconstruction-Based Pretraining
토론
이 논문과 연결된 포럼 글이 아직 없습니다.