A Contrastive Pre-training Approach to Learn Discriminative Autoencoder for Dense Retrieval
원문 초록
Dense retrieval (DR) has shown promising results in information retrieval. In essence, DR requires high-quality text representations to support effective search in the representation space. Recent studies have shown that pre-trained autoencoder-based language models with a weak decoder can provide high-quality text representations, boosting the effectiveness and few-shot ability of DR models. However, even a weak autoregressive decoder has the bypass effect on the encoder. More importantly, the discriminative ability of learned representations may be limited since each token is treated equally important in decoding the input texts. To address the above problems, in this paper, we propose a contrastive pre-training approach to learn a discriminative autoencoder with a lightweight multi-layer perception (MLP) decoder. The basic idea is to generate word distributions of input text in a non-autoregressive fashion and pull the word distributions of two masked versions of one text close while pushing away from others. We theoretically show that our contrastive strategy can suppress the common words and highlight the representative words in decoding, leading to discriminative representations. Empirical results show that our method can significantly outperform the state-of-the-art autoencoder-based language models and other pre-trained models for dense retrieval.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Seq vs Seq: An Open Suite of Paired Encoders and Decoders
JointRank: Rank Large Set with Single Pass
From Web Search towards Agentic Deep Research: Incentivizing Search with Reasoning Agents
DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agents
Context-Aware Search and Retrieval Over Erasure Channels
From Chaos to Automation: Enabling the Use of Unstructured Data for Robotic Process Automation
토론
이 논문과 연결된 포럼 글이 아직 없습니다.