A new approach to calculating BERTScore for automatic assessment of translation quality
원문 초록
The study of the applicability of the BERTScore metric was conducted to translation quality assessment at the sentence level for English -> Russian direction. Experiments were performed with a pre-trained Multilingual BERT as well as with a pair of Monolingual BERT models. To align monolingual embeddings, an orthogonal transformation based on anchor tokens was used. It was demonstrated that such transformation helps to prevent mismatching issue and shown that this approach gives better results than using embeddings of the Multilingual model. To improve the token matching process it is proposed to combine all incomplete WorkPiece tokens into meaningful words and use simple averaging of corresponding vectors and to calculate BERTScore based on anchor tokens only. Such modifications allowed us to achieve a better correlation of the model predictions with human judgments. In addition to evaluating machine translation, several versions of human translation were evaluated as well, the problems of this approach were listed.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Intrinsic vs. Extrinsic Evaluation of Czech Sentence Embeddings: Semantic Relevance Doesn't Help with MT Evaluation
ConECT Dataset: Overcoming Data Scarcity in Context-Aware E-Commerce MT
Building a Functional Machine Translation Corpus for Kpelle
Combining the Best of Both Worlds: A Method for Hybrid NMT and LLM Translation
Same evaluation, more tokens: On the effect of input length for machine translation evaluation using Large Language Models
CycleDistill: Bootstrapping Machine Translation using LLMs with Cyclical Distillation
토론
이 논문과 연결된 포럼 글이 아직 없습니다.