Equipping Transformer with Random-Access Reading for Long-Context Understanding
원문 초록
Long-context modeling presents a significant challenge for transformer-based large language models (LLMs) due to the quadratic complexity of the self-attention mechanism and issues with length extrapolation caused by pretraining exclusively on short inputs. Existing methods address computational complexity through techniques such as text chunking, the kernel approach, and structured attention, and tackle length extrapolation problems through positional encoding, continued pretraining, and data engineering. These approaches typically require $\textbf{sequential access}$ to the document, necessitating reading from the first to the last token. We contend that for goal-oriented reading of long documents, such sequential access is not necessary, and a proficiently trained model can learn to omit hundreds of less pertinent tokens. Inspired by human reading behaviors and existing empirical observations, we propose $\textbf{random access}$, a novel reading strategy that enables transformers to efficiently process long documents without examining every token. Experimental results from pretraining, fine-tuning, and inference phases validate the efficacy of our method.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Ref-Long: Benchmarking the Long-context Referencing Capability of Long-context Language Models
Cache Me If You Can: How Many KVs Do You Need for Effective Long-Context LMs?
cAST: Enhancing Code Retrieval-Augmented Generation with Structural Chunking via Abstract Syntax Tree
TableRAG: A Retrieval Augmented Generation Framework for Heterogeneous Document Reasoning
Real-Time Execution of Action Chunking Flow Policies
DAM: Dynamic Attention Mask for Long-Context Large Language Model Inference Acceleration
토론
이 논문과 연결된 포럼 글이 아직 없습니다.