Minder: Faulty Machine Detection for Large-scale Distributed Model Training
원문 초록
Large-scale distributed model training requires simultaneous training on up to thousands of machines. Faulty machine detection is critical when an unexpected fault occurs in a machine. From our experience, a training task can encounter two faults per day on average, possibly leading to a halt for hours. To address the drawbacks of the time-consuming and labor-intensive manual scrutiny, we propose Minder, an automatic faulty machine detector for distributed training tasks. The key idea of Minder is to automatically and efficiently detect faulty distinctive monitoring metric patterns, which could last for a period before the entire training task comes to a halt. Minder has been deployed in our production environment for over one year, monitoring daily distributed training tasks where each involves up to thousands of machines. In our real-world fault detection scenarios, Minder can accurately and efficiently react to faults within 3.6 seconds on average, with a precision of 0.904 and F1-score of 0.893.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
Online Isolation Forest
Comparative Study of Generative Models for Early Detection of Failures in Medical Devices
Benchmarking Traditional Machine Learning and Deep Learning Models for Fault Detection in Power Transformers
Data-Driven Sensor Fault Diagnosis with Proven Guarantees using Incrementally Stable Recurrent Neural Networks
Unsupervised Anomaly Detection through Mass Repulsing Optimal Transport
Integrating Physics and Data-Driven Approaches: An Explainable and Uncertainty-Aware Hybrid Model for Wind Turbine Power Prediction
토론
이 논문과 연결된 포럼 글이 아직 없습니다.