Are Large Language Models In-Context Personalized Summarizers? Get an iCOPERNICUS Test Done!
원문 초록
Large Language Models (LLMs) have succeeded considerably in In-Context-Learning (ICL) based summarization. However, saliency is subject to the users' specific preference histories. Hence, we need reliable In-Context Personalization Learning (ICPL) capabilities within such LLMs. For any arbitrary LLM to exhibit ICPL, it needs to have the ability to discern contrast in user profiles. A recent study proposed a measure for degree-of-personalization called EGISES for the first time. EGISES measures a model's responsiveness to user profile differences. However, it cannot test if a model utilizes all three types of cues provided in ICPL prompts: (i) example summaries, (ii) user's reading histories, and (iii) contrast in user profiles. To address this, we propose the iCOPERNICUS framework, a novel In-COntext PERsonalization learNIng sCrUtiny of Summarization capability in LLMs that uses EGISES as a comparative measure. As a case-study, we evaluate 17 state-of-the-art LLMs based on their reported ICL performances and observe that 15 models' ICPL degrades (min: 1.6%; max: 3.6%) when probed with richer prompts, thereby showing lack of true ICPL.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Assay2Mol: large language model-based drug design using BioAssay context
Universal pre-training by iterated random computation
Evolving Prompts In-Context: An Open-ended, Self-replicating Perspective
Conquering the Retina: Bringing Visual in-Context Learning to OCT
Evaluating Cell Type Inference in Vision Language Models Under Varying Visual Context
PRO-V: An Efficient Program Generation Multi-Agent System for Automatic RTL Verification
토론
이 논문과 연결된 포럼 글이 아직 없습니다.