Revisiting Parallel Context Windows: A Frustratingly Simple Alternative and Chain-of-Thought Deterioration
원문 초록
We identify two crucial limitations in the evaluation of recent parallel-integrated method Parallel Context Windows (PCW), which extends the maximum context lengths of language models, e.g., 2048 for LLaMA, by harnessing window-wise attention and positional embedding techniques. We first show that a simple yet strong baseline, weighted sum ensemble, is missing for the in-context few-shot classification. Moreover, on more challenging Chain-of-Thought (CoT) reasoning (e.g., HotpotQA), PCW would present unexpected deterioration regarding question miscomprehension and false inference. Based on our findings, we suggest that the existing PCW design may not guarantee sufficient improvement and practicality in handling lengthy documents in real-world applications. More community efforts on enabling language models' long context understanding ability should be paid.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Ref-Long: Benchmarking the Long-context Referencing Capability of Long-context Language Models
Cache Me If You Can: How Many KVs Do You Need for Effective Long-Context LMs?
DAM: Dynamic Attention Mask for Long-Context Large Language Model Inference Acceleration
MesaNet: Sequence Modeling by Locally Optimal Test-Time Training
SpecExtend: A Drop-in Enhancement for Speculative Decoding of Long Sequences
Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compression
토론
이 논문과 연결된 포럼 글이 아직 없습니다.