LLMs on the Line: Data Determines Loss-to-Loss Scaling Laws
원문 초록
Scaling laws guide the development of large language models (LLMs) by offering estimates for the optimal balance of model size, tokens, and compute. More recently, loss-to-loss scaling laws that relate losses across pretraining datasets and downstream tasks have emerged as a powerful tool for understanding and improving LLM performance. In this work, we investigate which factors most strongly influence loss-to-loss scaling. Our experiments reveal that the pretraining data and tokenizer determine the scaling trend. In contrast, model size, optimization hyperparameters, and even significant architectural differences, such as between transformer-based models like Llama and state-space models like Mamba, have limited impact. Consequently, practitioners should carefully curate suitable pretraining datasets for optimal downstream performance, while architectures and other settings can be freely optimized for training efficiency.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
MambaFusion: Height-Fidelity Dense Global Fusion for Multi-modal 3D Object Detection
Sparsified State-Space Models are Efficient Highway Networks
Hierarchical Mamba Meets Hyperbolic Geometry: A New Paradigm for Structured Language Embeddings
Hybrid-Emba3D: Geometry-Aware and Cross-Path Feature Hybrid Enhanced State Space Model for Point Cloud Classification
RD-UIE: Relation-Driven State Space Modeling for Underwater Image Enhancement
Vision Mamba in Remote Sensing: A Comprehensive Survey of Techniques, Applications and Outlook
토론
이 논문과 연결된 포럼 글이 아직 없습니다.