Cataloguing Hugging Face Models to Software Engineering Activities: Automation and Findings

원문 초록

Context: Open-source Pre-Trained Models (PTMs) provide extensive resources for various Machine Learning (ML) tasks, yet these resources lack a classification tailored to Software Engineering (SE). Objective: To address this gap, we derive a taxonomy encompassing 146 SE tasks and apply an SE-oriented classification to PTMs in a popular open-source ML repository, Hugging Face (HF). Method: Our repository mining study followed a five-phase pipeline: (i) identification of SE tasks from the literature; (ii) collection of PTM data from the HF API, including model card descriptions and metadata, and the abstracts of the associated arXiv papers; (iii) text processing to ensure consistency; (iv) a two-phase validation of SE relevance, involving humans and LLM assistance, supported by five pilot studies with human annotators and a generalization test; and (v) data analysis. This process yielded a curated catalogue of 5,027 SE PTMs. Results: We find that most SE PTMs target code generation and coding, emphasizing implementation over early or late development stages. In terms of ML tasks, text generation dominates within SE PTMs. Notably, the number of SE PTMs has increased markedly since 2023 Q2, while benchmarking remains limited: only 5.55\% of all SE PTMs in our catalogue provide evaluation results. Conclusions: Our catalogue reveals documentation and transparency gaps, highlights SE activity imbalances, and supports automated SE scenarios, such as the selection of suitable PTMs.