BURT: BERT-inspired Universal Representation from Twin Structure
원문 초록
Pre-trained contextualized language models such as BERT have shown great effectiveness in a wide range of downstream Natural Language Processing (NLP) tasks. However, the effective representations offered by the models target at each token inside a sequence rather than each sequence and the fine-tuning step involves the input of both sequences at one time, leading to unsatisfying representations of various sequences with different granularities. Especially, as sentence-level representations taken as the full training context in these models, there comes inferior performance on lower-level linguistic units (phrases and words). In this work, we present BURT (BERT inspired Universal Representation from Twin Structure) that is capable of generating universal, fixed-size representations for input sequences of any granularity, i.e., words, phrases, and sentences, using a large scale of natural language inference and paraphrase data with multiple training objectives. Our proposed BURT adopts the Siamese network, learning sentence-level representations from natural language inference dataset and word/phrase-level representations from paraphrasing dataset, respectively. We evaluate BURT across different granularities of text similarity tasks, including STS tasks, SemEval2013 Task 5(a) and some commonly used word similarity tasks, where BURT substantially outperforms other representation models on sentence-level datasets and achieves significant improvements in word/phrase-level representation.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Parameter-Efficient Transformer Embeddings
A MISMATCHED Benchmark for Scientific Natural Language Inference
Contrastive Prompting Enhances Sentence Embeddings in LLMs through Inference-Time Steering
JTCSE: Joint Tensor-Modulus Constraints and Cross-Attention for Unsupervised Contrastive Learning of Sentence Embeddings
Domain Adaptation for Japanese Sentence Embeddings with Contrastive Learning based on Synthetic Sentence Generation
TextInPlace: Indoor Visual Place Recognition in Repetitive Structures with Scene Text Spotting and Verification
토론
이 논문과 연결된 포럼 글이 아직 없습니다.