arXiv:2202.01145
Relative Position Prediction as Pre-training for Text Encoders
원문 초록
Meaning is defined by the company it keeps. However, company is two-fold: It's based on the identity of tokens and also on their position (topology). We argue that a position-centric perspective is more general and useful. The classic MLM and CLM objectives in NLP are easily phrased as position predictions over the whole vocabulary. Adapting the relative position encoding paradigm in NLP to create relative labels for self-supervised learning, we seek to show superior pre-training judged by performance on downstream tasks.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Speech Quality Assessment Model Based on Mixture of Experts: System-Level Performance Enhancement and Utterance-Level Challenge Analysis
Position Dependent Prediction Combination For Intra-Frame Video Coding
Open-World Social Event Classification
EasySpider: A No-Code Visual System for Crawling the Web
Multidimensional Particle Filter for Long-Term Visual Teach and Repeat in Changing Environments
Car Object Counting and Position Estimation via Extension of the CLIP-EBC Framework
토론
이 논문과 연결된 포럼 글이 아직 없습니다.