Low Anisotropy Sense Retrofitting (LASeR) : Towards Isotropic and Sense Enriched Representations
원문 초록
Contextual word representation models have shown massive improvements on a multitude of NLP tasks, yet their word sense disambiguation capabilities remain poorly explained. To address this gap, we assess whether contextual word representations extracted from deep pretrained language models create distinguishable representations for different senses of a given word. We analyze the representation geometry and find that most layers of deep pretrained language models create highly anisotropic representations, pointing towards the existence of representation degeneration problem in contextual word representations. After accounting for anisotropy, our study further reveals that there is variability in sense learning capabilities across different language models. Finally, we propose LASeR, a 'Low Anisotropy Sense Retrofitting' approach that renders off-the-shelf representations isotropic and semantically more meaningful, resolving the representation degeneration problem as a post-processing step, and conducting sense-enrichment of contextualized representations extracted from deep neural language models.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
SANDWiCH: Semantical Analysis of Neighbours for Disambiguating Words in Context ad Hoc
Fietje: An open, efficient LLM for Dutch
LexMatcher: Dictionary-centric Data Collection for LLM-based Machine Translation
Do Pretrained Contextual Language Models Distinguish between Hebrew Homograph Analyses?
3AM: An Ambiguity-Aware Multi-Modal Machine Translation Dataset
PejorativITy: Disambiguating Pejorative Epithets to Improve Misogyny Detection in Italian Tweets
토론
이 논문과 연결된 포럼 글이 아직 없습니다.