Limitations of Assessing Active Learning Performance at Runtime
원문 초록
Classification algorithms aim to predict an unknown label (e.g., a quality class) for a new instance (e.g., a product). Therefore, training samples (instances and labels) are used to deduct classification hypotheses. Often, it is relatively easy to capture instances but the acquisition of the corresponding labels remain difficult or expensive. Active learning algorithms select the most beneficial instances to be labeled to reduce cost. In research, this labeling procedure is simulated and therefore a ground truth is available. But during deployment, active learning is a one-shot problem and an evaluation set is not available. Hence, it is not possible to reliably estimate the performance of the classification system during learning and it is difficult to decide when the system fulfills the quality requirements (stopping criteria). In this article, we formalize the task and review existing strategies to assess the performance of an actively trained classifier during training. Furthermore, we identified three major challenges: 1)~to derive a performance distribution, 2)~to preserve representativeness of the labeled subset, and 3) to correct against sampling bias induced by an intelligent selection strategy. In a qualitative analysis, we evaluate different existing approaches and show that none of them reliably estimates active learning performance stating a major challenge for future research for such systems. All plots and experiments are provided in a Jupyter notebook that is available for download.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Wide Contextual Residual Network with Active Learning for Remote Sensing Image Classification
CriticLean: Critic-Guided Reinforcement Learning for Mathematical Formalization
Active Learning for Manifold Gaussian Process Regression
The Catechol Benchmark: Time-series Solvent Selection Data for Few-shot Machine Learning
Info-Coevolution: An Efficient Framework for Data Model Coevolution
ALINE: Joint Amortization for Bayesian Inference and Active Data Acquisition
토론
이 논문과 연결된 포럼 글이 아직 없습니다.