Efficient Test Collection Construction via Active Learning
원문 초록
To create a new IR test collection at low cost, it is valuable to carefully select which documents merit human relevance judgments. Shared task campaigns such as NIST TREC pool document rankings from many participating systems (and often interactive runs as well) in order to identify the most likely relevant documents for human judging. However, if one's primary goal is merely to build a test collection, it would be useful to be able to do so without needing to run an entire shared task. Toward this end, we investigate multiple active learning strategies which, without reliance on system rankings: 1) select which documents human assessors should judge; and 2) automatically classify the relevance of additional unjudged documents. To assess our approach, we report experiments on five TREC collections with varying scarcity of relevant documents. We report labeling accuracy achieved, as well as rank correlation when evaluating participant systems based upon these labels vs.\ full pool judgments. Results show the effectiveness of our approach, and we further analyze how varying relevance scarcity across collections impacts our findings. To support reproducibility and follow-on work, we have shared our code online: https://github.com/mdmustafizurrahman/ICTIR_AL_TestCollection_2020/.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Near-Polynomially Competitive Active Logistic Regression
CriticLean: Critic-Guided Reinforcement Learning for Mathematical Formalization
Active Learning for Manifold Gaussian Process Regression
The Catechol Benchmark: Time-series Solvent Selection Data for Few-shot Machine Learning
Info-Coevolution: An Efficient Framework for Data Model Coevolution
ALINE: Joint Amortization for Bayesian Inference and Active Data Acquisition
토론
이 논문과 연결된 포럼 글이 아직 없습니다.