Scaling Instruction-Tuned LLMs to Million-Token Contexts via Hierarchical Synthetic Data Generation
원문 초록
Large Language Models (LLMs) struggle with long-context reasoning, not only due to the quadratic scaling of computational complexity with sequence length but also because of the scarcity and expense of annotating long-context data. There has been barely any open-source work that systematically ablates long-context data, nor is there any openly available instruction tuning dataset with contexts surpassing 100K tokens. To bridge this gap, we introduce a novel post-training synthetic data generation strategy designed to efficiently extend the context window of LLMs while preserving their general task performance. Our approach scalably extends to arbitrarily long context lengths, unconstrained by the length of available real-world data, which effectively addresses the scarcity of raw long-context data. Through a step-by-step rotary position embedding (RoPE) scaling training strategy, we demonstrate that our model, with a context length of up to 1M tokens, performs well on the RULER benchmark and InfiniteBench and maintains robust performance on general language tasks.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
How Good Are Synthetic Requirements ? Evaluating LLM-Generated Datasets for AI4RE
SDialog: A Python Toolkit for Synthetic Dialogue Generation and Analysis
Cosmos-Drive-Dreams: Scalable Synthetic Driving Data Generation with World Foundation Models
Optimization-Free Universal Watermark Forgery with Regenerative Diffusion Models
BEAR: BGP Event Analysis and Reporting
Datasheets Aren't Enough: DataRubrics for Automated Quality Metrics and Accountability
토론
이 논문과 연결된 포럼 글이 아직 없습니다.