A Study on Agreement in PICO Span Annotations
원문 초록
In evidence-based medicine, relevance of medical literature is determined by predefined relevance conditions. The conditions are defined based on PICO elements, namely, Patient, Intervention, Comparator, and Outcome. Hence, PICO annotations in medical literature are essential for automatic relevant document filtering. However, defining boundaries of text spans for PICO elements is not straightforward. In this paper, we study the agreement of PICO annotations made by multiple human annotators, including both experts and non-experts. Agreements are estimated by a standard span agreement (i.e., matching both labels and boundaries of text spans), and two types of relaxed span agreement (i.e., matching labels without guaranteeing matching boundaries of spans). Based on the analysis, we report two observations: (i) Boundaries of PICO span annotations by individual human annotators are very diverse. (ii) Despite the disagreement in span boundaries, general areas of the span annotations are broadly agreed by annotators. Our results suggest that applying a standard agreement alone may undermine the agreement of PICO spans, and adopting both a standard and a relaxed agreements is more suitable for PICO span evaluation.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
AlpaPICO: Extraction of PICO Frames from Clinical Trial Documents Using LLMs
FactPICO: Factuality Evaluation for Plain Language Summarization of Medical Evidence
PiCO: Peer Review in LLMs based on the Consistency Optimization
Bio-SIEVE: Exploring Instruction Tuning Large Language Models for Systematic Review Automation
Towards Effective Visual Representations for Partial-Label Learning
Simulating single-photon detector array sensors for depth imaging
토론
이 논문과 연결된 포럼 글이 아직 없습니다.