Delving into Effective Gradient Matching for Dataset Condensation
원문 초록
As deep learning models and datasets rapidly scale up, network training is extremely time-consuming and resource-costly. Instead of training on the entire dataset, learning with a small synthetic dataset becomes an efficient solution. Extensive research has been explored in the direction of dataset condensation, among which gradient matching achieves state-of-the-art performance. The gradient matching method directly targets the training dynamics by matching the gradient when training on the original and synthetic datasets. However, there are limited deep investigations into the principle and effectiveness of this method. In this work, we delve into the gradient matching method from a comprehensive perspective and answer the critical questions of what, how, and where to match. We propose to match the multi-level gradients to involve both intra-class and inter-class gradient information. We demonstrate that the distance function should focus on the angle, considering the magnitude simultaneously to delay the overfitting. An overfitting-aware adaptive learning step strategy is also proposed to trim unnecessary optimization steps for algorithmic efficiency improvement. Ablation and comparison experiments demonstrate that our proposed methodology shows superior accuracy, efficiency, and generalization compared to prior work.
함께 살펴볼 논문
공통 태그를 근거로 연결했습니다. 실제 인용 관계나 논문의 우수성을 뜻하지 않습니다.
A Large-Scale Study on Video Action Dataset Condensation
Bi-Directional Multi-Scale Graph Dataset Condensation via Information Bottleneck
Decomposed Distribution Matching in Dataset Condensation
Towards Model-Agnostic Dataset Condensation by Heterogeneous Models
Mitigating Bias in Dataset Distillation
CondTSF: One-line Plugin of Dataset Condensation for Time Series Forecasting
토론
이 논문과 연결된 포럼 글이 아직 없습니다.