PyTorchKR Papers

최신 논문

총 2편
태그·기간 등 상세 필터

카테고리 탐색

로그인 후 이 태그 구독

arXiv:2607.01538v1cs.CL

Can Language Models Actually Retrieve In-Context? Drowning in Documents at Million Token Scale

Language models (LMs) raise an intriguing alternative to vector-based retrieval: conditioning on an in-context corpus and directly generating a relevant answer. However, prior work has largely focused on proprietary systems or the smaller-scale reranking task, leaving corpus-scale in-context retrieval largely unexplored. In this work, we present the first systematic study of in-context retrieval on two scales practical retrievers demand: million-token corpora and length-generalization far beyond training-time sizes. We first introduce BlockSearch, a 0.6B LM retriever whose architectural and training modifications improve over prior LM baselines and length-generalize up to 10 times beyond its training regime. Nevertheless, retrieval still collapses under more extreme extrapolation. We trace this failure to an attention dilution effect: as the corpus grows, irrelevant documents dominate the softmax denominator, reducing the normalized mass on the gold document even when its pre-softmax score stays high. Motivated by this analysis, we introduce length-aware adjustments to the attention softmax and document-level sparse attention. With these modifications, at the million-token scale, our model matches dense retrieval on widely studied benchmarks (e.g, MS MARCO and NQ), while outperforming the concurrent model MSA despite being 7 times smaller. Furthermore, it significantly outperforms dense retrieval on tasks requiring entirely different notions of similarity, such as LIMIT, achieving a 3 times higher score. Together, our results position in-context retrieval a promising alternative to classical retrieval while emphasizing attention control under extreme context growth as a new challenge.

arXiv:2606.21649v2cs.CL

EvoEmbedding: 긴 문맥 검색 및 에이전트 메모리를 위한 진화 가능한 표현

EvoEmbedding: Evolvable Representations for Long-Context Retrieval and Agentic Memory

정적 임베딩 모델은 텍스트를 고립적으로 인코딩해 주변 문맥과 시간적 순서를 반영하지 못하지만, EvoEmbedding은 순차 입력을 처리하며 갱신되는 잠재 메모리와 원시 콘텐츠를 함께 사용해 동일 쿼리도 진화하는 문맥에 맞게 다른 대상을 검색할 수 있는 진화 가능한 임베딩을 생성한다. EvoTrain-180K로 학습한 이 모델은 긴 문맥 검색 벤치마크에서 Qwen3-Embedding-8B·KaLM-Embedding-Gemma3-12B 등 더 큰 전문 모델을 능가했으며, 단순 검색-증강 생성(RAG) 파이프라인으로도 전용 에이전트 메모리 시스템을 넘어선다.