FigCaps-HF: A Figure-to-Caption Generative Framework and Benchmark with Human Feedback
원문 초록
Captions are crucial for understanding scientific visualizations and documents. Existing captioning methods for scientific figures rely on figure-caption pairs extracted from documents for training, many of which fall short with respect to metrics like helpfulness, explainability, and visual-descriptiveness [15] leading to generated captions being misaligned with reader preferences. To enable the generation of high-quality figure captions, we introduce FigCaps-HF a new framework for figure-caption generation that can incorporate domain expert feedback in generating captions optimized for reader preferences. Our framework comprises of 1) an automatic method for evaluating quality of figure-caption pairs, 2) a novel reinforcement learning with human feedback (RLHF) method to optimize a generative figure-to-caption model for reader preferences. We demonstrate the effectiveness of our simple learning framework by improving performance over standard fine-tuning across different types of models. In particular, when using BLIP as the base model, our RLHF framework achieves a mean gain of 35.7%, 16.9%, and 9% in ROUGE, BLEU, and Meteor, respectively. Finally, we release a large-scale benchmark dataset with human feedback on figure-caption pairs to enable further evaluation and development of RLHF techniques for this problem.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
DenseWorld-1M: Towards Detailed Dense Grounded Caption in the Real World
SonicVerse: Multi-Task Learning for Music Feature-Informed Captioning
FusionAudio-1.2M: Towards Fine-grained Audio Captioning with Multimodal Contextual Fusion
VCapsBench: A Large-scale Fine-grained Benchmark for Video Caption Quality Evaluation
LoVR: A Benchmark for Long Video Retrieval in Multimodal Contexts
VideoMultiAgents: A Multi-Agent Framework for Video Question Answering
토론
이 논문과 연결된 포럼 글이 아직 없습니다.