Look Backward and Forward: Self-Knowledge Distillation with Bidirectional Decoder for Neural Machine Translation
원문 초록
Neural Machine Translation(NMT) models are usually trained via unidirectional decoder which corresponds to optimizing one-step-ahead prediction. However, this kind of unidirectional decoding framework may incline to focus on local structure rather than global coherence. To alleviate this problem, we propose a novel method, Self-Knowledge Distillation with Bidirectional Decoder for Neural Machine Translation(SBD-NMT). We deploy a backward decoder which can act as an effective regularization method to the forward decoder. By leveraging the backward decoder's information about the longer-term future, distilling knowledge learned in the backward decoder can encourage auto-regressive NMT models to plan ahead. Experiments show that our method is significantly better than the strong Transformer baselines on multiple machine translation data sets.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Tackling Data Heterogeneity in Federated Learning through Knowledge Distillation with Inequitable Aggregation
A Novel Lightweight Transformer with Edge-Aware Fusion for Remote Sensing Image Captioning
ConECT Dataset: Overcoming Data Scarcity in Context-Aware E-Commerce MT
TalkingMachines: Real-Time Audio-Driven FaceTime-Style Video via Autoregressive Diffusion Models
Distilled Gradual Pruning with Pruned Fine-tuning
Eyelid’s Intrinsic Motion-aware Feature Learning for Real-time Eyeblink Detection in the Wild
토론
이 논문과 연결된 포럼 글이 아직 없습니다.