A Meta-Evaluation of Style and Attribute Transfer Metrics
원문 초록
LLMs make it easy to rewrite text in any style, be it more polite, persuasive, or more positive. We present a large-scale study of evaluation metrics for style and attribute transfer with a focus on content preservation; meaning content not attributed to the style shift is preserved. The de facto evaluation approach uses lexical or semantic similarity metrics often between source sentences and rewrites. While these metrics are not designed to distinguish between style or content differences, empirical meta-evaluation shows a reasonable correlation to human judgment. In fact, recent works find that LLMs prompted as evaluators are only comparable to semantic similarity metrics, even though intuitively, the LLM approach should better fit the task. To investigate this discrepancy, we benchmark 8 metrics for evaluating content preservation on existing datasets and additionally construct a new test set that better aligns with the meta-evaluation aim. Indeed, we then find that the empirical conclusion aligns with the intuition: content preservation metrics for style/attribute transfer must be conditional on the style shift. To support this, we propose a new efficient zero-shot evaluation method using the likelihood of the next token. We hope our meta-evaluation can foster more research on evaluating content preservation metrics, and also to ensure fair evaluation of methods for conducting style transfer.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Community Search in Time-dependent Road-social Attributed Networks
FA: Forced Prompt Learning of Vision-Language Models for Out-of-Distribution Detection
ImpliRet: Benchmarking the Implicit Fact Retrieval Challenge
Inv-Entropy: A Fully Probabilistic Framework for Uncertainty Quantification in Language Models
Trend-Aware Fashion Recommendation with Visual Segmentation and Semantic Similarity
KNN-Defense: Defense against 3D Adversarial Point Clouds using Nearest-Neighbor Search
토론
이 논문과 연결된 포럼 글이 아직 없습니다.