Enhancing Latent Computation in Transformers with Latent Tokens
원문 초록
Augmenting large language models (LLMs) with auxiliary tokens has emerged as a promising strategy for enhancing model performance. In this work, we introduce a lightweight method termed latent tokens; these are dummy tokens that may be non-interpretable in natural language but steer the autoregressive decoding process of a Transformer-based LLM via the attention mechanism. The proposed latent tokens can be seamlessly integrated with a pre-trained Transformer, trained in a parameter-efficient manner, and applied flexibly at inference time, while adding minimal complexity overhead to the existing infrastructure of standard Transformers. We propose several hypotheses about the underlying mechanisms of latent tokens and design synthetic tasks accordingly to verify them. Numerical results confirm that the proposed method noticeably outperforms the baselines, particularly in the out-of-distribution generalization scenarios, highlighting its potential in improving the adaptability of LLMs.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Positional Encoding meets Persistent Homology on Graphs
Pruning Spurious Subgraphs for Graph Out-of-Distribtuion Generalization
The Avengers: A Simple Recipe for Uniting Smaller Language Models to Challenge Proprietary Giants
LORE: Lagrangian-Optimized Robust Embeddings for Visual Encoders
On the Out-of-Distribution Generalization of Self-Supervised Learning
From Local Details to Global Context: Advancing Vision-Language Models with Attention-Based Selection
토론
이 논문과 연결된 포럼 글이 아직 없습니다.