Beyond the Limits: A Survey of Techniques to Extend the Context Length in Large Language Models
원문 초록
Recently, large language models (LLMs) have shown remarkable capabilities including understanding context, engaging in logical reasoning, and generating responses. However, this is achieved at the expense of stringent computational and memory requirements, hindering their ability to effectively support long input sequences. This survey provides an inclusive review of the recent techniques and methods devised to extend the sequence length in LLMs, thereby enhancing their capacity for long-context understanding. In particular, we review and categorize a wide range of techniques including architectural modifications, such as modified positional encoding and altered attention mechanisms, which are designed to enhance the processing of longer sequences while avoiding a proportional increase in computational requirements. The diverse methodologies investigated in this study can be leveraged across different phases of LLMs, i.e., training, fine-tuning and inference. This enables LLMs to efficiently process extended sequences. The limitations of the current methodologies is discussed in the last section along with the suggestions for future research directions, underscoring the importance of sequence length in the continued advancement of LLMs.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
RuleArena: A Benchmark for Rule-Guided Reasoning with LLMs in Real-World Scenarios
Ref-Long: Benchmarking the Long-context Referencing Capability of Long-context Language Models
Cache Me If You Can: How Many KVs Do You Need for Effective Long-Context LMs?
SoundMind: RL-Incentivized Logic Reasoning for Audio-Language Models
EviNet: Evidential Reasoning Network for Resilient Graph Learning in the Open and Noisy Environments
DAM: Dynamic Attention Mask for Long-Context Large Language Model Inference Acceleration
토론
이 논문과 연결된 포럼 글이 아직 없습니다.