What is the ground truth? Reliability of multi-annotator data for audio tagging
원문 초록
Crowdsourcing has become a common approach for annotating large amounts of data. It has the advantage of harnessing a large workforce to produce large amounts of data in a short time, but comes with the disadvantage of employing non-expert annotators with different backgrounds. This raises the problem of data reliability, in addition to the general question of how to combine the opinions of multiple annotators in order to estimate the ground truth. This paper presents a study of the annotations and annotators' reliability for audio tagging. We adapt the use of Krippendorf's alpha and multi-annotator competence estimation (MACE) for a multi-labeled data scenario, and present how MACE can be used to estimate a candidate ground truth based on annotations from non-expert users with different levels of expertise and competence.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Hierarchical Label Propagation: A Model-Size-Dependent Performance Booster for AudioSet Tagging
Solla: Towards a Speech-Oriented LLM That Hears Acoustic Context
Exploring Performance-Complexity Trade-Offs in Sound Event Detection Models
LC-Protonets: Multi-Label Few-Shot Learning for World Music Audio Tagging
Integrating IP Broadcasting with Audio Tags: Workflow and Challenges
DASS: Distilled Audio State Space Models Are Stronger and More Duration-Scalable Learners
토론
이 논문과 연결된 포럼 글이 아직 없습니다.