Cataloguing Hugging Face Models to Software Engineering Activities: Automation and Findings
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.
Tom: We're looking at the paper: "Cataloguing Hugging Face Models to Software Engineering Activities".
Jane: Here's the gist of the paper. Open-source Pre-Trained Models (PTMs) provide extensive resources for various Machine Learning (ML) tasks, yet these resources lack a classification tailored to Software Engineering (SE) needs to support the reliable identification and reuse of models for SE Pruning redundant reasoning tokens The objective is to address the gap in classification by deriving a taxonomy encompassing 147 SE tasks and applying an SE-oriented classification to PTMs in the popular open-source ML repository, Hugging Face (HF) The method involved a five-phase pipeline, including identification of SE tasks from literature, collection of PTM data from the HF API, text processing for consistency, two-phase validation involving humans and LLM assistance with five pilot studies, and data analysis This process yielded a curated catalogue of 2,205 SE PTMs How the training signal is built The taxonomy of 147 SE tasks was built on prior research and established guidelines, extending a foundation from Hou et al.
Tom: (2024) This taxonomy broadened the scope by refining definitions for SE activities and enriching the set of tasks using concepts from the Software Engineering Body of Knowledge (SWEBOK) The data collection phase involved accessing the HF API to retrieve model card descriptions, metadata, and arXiv paper abstracts Data processing included text normalization via tokenizing, lowercasing, and lemmatizing to standardize textual descriptions Where the gains come from The analysis revealed that most SE PTMs target code generation and coding, emphasizing implementation over early or late development stages Text generation dominates within SE PTMs in terms of ML tasks The number of SE PTMs has increased markedly since 2023 Q2, while evaluation remains limited, with only 9.6% reporting benchmark results, mostly scoring below 50% Pruning redundant reasoning tokens The catalogue reveals documentation and transparency gaps It highlights imbalances across SDLC phases and provides a foundation for automated SE scenarios, such as the sampling and selection of suitable PTMs Where the gains come from The study found that software implementation is the most dominant SE activity in terms of PTM coverage, with 2,704 models Code generation and coding are the most represented SE tasks Software implementation-related PTMs outnumber those for early and late SDLC stages Where the gains come from The analysis of base models showed that established models such as meta-llama/Llama-2-7b-hf have supported 9 PTMs since their introduction, while newer models like Spestly/Atlas-Flash-7BPreview serve as the base for 36 PTMs within a few months of release Most SE PTMs score below 50% on several common benchmarks, showing many tasks remain largely unsolved Where the gains come from Permissive licenses such as Apache 2.0 and MIT dominate, particularly in the software implementation phase This dominance aligns with prior works on HF, underscoring the preference for licenses that enable broad use, modification, and redistribution with minimal barriers The findings motivate the ongoing development of MLAssetSelection to assist in the sampling and selection of SE PTMs Pruning redundant reasoning tokens The study identifies documentation gaps, noting that 68.52% of PTMs have a non-empty model card, but only 7.32% mention an SE task Correct classification requires addressing duplicated model cards and contextual misinterpretations of SE tasks The catalogue supports informed selection based on project needs, such as license type or reported evaluation Where the gains come from The findings reveal that software design and requirements engineering are under-represented, highlighting a need for broader coverage across the SDLC The most prevalent ML task across SE-related PTMs is text generation, while code generation is the most commonly addressed SE task This snapshot highlights existing gaps in SE resources and provides valuable insights into the field’s evolving needs Pruning redundant reasoning tokens The framework supports incremental updates, allowing users to retrieve only the PTMs created after the cutoff date, ensuring efficient maintainability without redundant processing The value of this work lies not only in the static snapshot provided but in the updatability of the framework The catalogue is designed to be extensible and interoperable with other existing datasets Where the gains come from The study's findings have also motivated the ongoing development of MLAssetSelection, a web-application that operationalizes our cataloguing pipeline to assist in the sampling and selection of SE PTMs The work provides a foundation for researchers to delve deeper into the challenges uncovered in this study and to conduct empirical studies using the collected SE PTMs The replication package includes the complete pipeline described in Section 3, enabling full replicability of our results Pruning redundant reasoning tokens Potential threats to validity include internal validity risks from incomplete or ambiguous documentation, and external validity concerns regarding other registries The construct validity of the SE task taxonomy builds upon existing literature, which may evolve as the SE community introduces new standards The study mitigates subjective bias through inter-coder agreement practices and LLM validation Where the gains come from The research suggests that certain phases of the SDLC remain underrepresented, particularly requirements engineering and software design, where few PTMs explicitly target these activities The catalogue provides a foundation for researchers to delve deeper into the challenges uncovered in this study and to conduct empirical studies using the collected SE PTMs The work aims to extend this classification to other registries, such as GitHub Models and PyTorch Hub PyTorch Foundation The findings reveal important gaps and opportunities in the current SE PTM ecosystem Where the gains come from The study concludes by suggesting that a combination of semantic and metric-based selection criteria can be set to guide users to find PTMs best suited to their specific context This process demonstrates how our framework can support decision-making in real-world scenarios The findings have also motivated the ongoing development of MLAssetSelection, a web-application that operationalizes our cataloguing pipeline to assist in the sampling and selection of SE PTMs The work contributes to automating the sampling and selection of relevant PTMs for researchers and practitioners The study is fully replicable, with all data, scripts, and resources publicly available Pruning redundant reasoning tokens The work supports the adoption of ML-driven tools within SE workflows by automating the identification and selection of PTMs The research reveals that software design is related to a single ML task: text generation This imbalance highlights the urgent need for greater research and tooling support for early SDLC activities, such as requirements engineering and software design The study is structured to analyze PTMs across temporal, structural, and legal dimensions Where the gains come from The analysis of training datasets shows clear patterns of dataset reuse across SE tasks, with codeparrot/github-code-clean being among the most commonly reused datasets No PTMs targeting requirements engineering report associated training datasets Permissive licenses such as Apache 2.
Paper summary: Tom: So we’re talking about this paper, "Cataloguing Hugging Face Models to Software Engineering Activities: Automation and Findings." Basically, they found that open-source models on platforms like Hugging Face are a huge resource for Machine Learning, but they don't have a good way to sort them specifically for Software Engineering needs.
Jane: That sounds like a real problem because if you’re building software, you need tools that are actually built for coding and design, not just general language tasks. So what did this paper set out to do?
Tom: The paper aimed to fix that gap by creating a specific classification system. They derived a taxonomy of one hundred forty-seven different Software Engineering tasks and then applied that structure to look at all the Pre-Trained Models on Hugging Face <ref:2506.03013#pg1>.
Lu: It’s interesting because they didn't just make up a list; they built this taxonomy based on prior research and guidelines, extending work from Hou et al. two thousand twenty-four by using concepts from the Software Engineering Body of Knowledge, or SWEBOK <ref:2506.03013#pg2>.
Meng: So they took that list of one hundred forty-seven tasks and started mining data directly from the Hugging Face API to see which models fit where <ref:2506.03013#pg1>. That’s a big data collection effort, right?
Jane: Exactly. They used this five-phase pipeline—they identified SE tasks from literature, collected model card descriptions and arXiv paper abstracts from the HF API, processed that text for consistency, did some human and AI validation through five pilot studies with annotators, and then analyzed all that data.
Lalam: And they ended up with a curated catalogue of two thousand two hundred five Software Engineering Pre-Trained Models <ref:2506.03013#pg2>. That’s a pretty substantial starting point for anyone trying to find relevant models.
Tom: What did the analysis show about what these models are actually doing? I want to know if they are focused on the right things for an engineer.
Lu: The analysis showed that most of these Software Engineering Pre-Trained Models are really targeting code generation and coding, emphasizing implementation over early or late development stages. Text generation dominates within the SE Pre-Trained Models in terms of the ML tasks they perform.
Meng: That makes sense from a deployment standpoint. Implementation is where you spend most of your time writing working code, not just designing requirements upfront or figuring out the initial architecture.
Paper summary: Jane: And they pointed out that software implementation is the most dominant Software Engineering activity in terms of Pre-Trained Model coverage, with two thousand seven hundred and four models falling into that category <ref:2506.03013#pg2>. That’s a lot of focus on building things.
Tom: But there are some caveats here too. They found documentation and transparency gaps across the whole development lifecycle, which means we aren't seeing a balanced spread across all stages yet.
Jane: They also highlighted an imbalance where software design and requirements engineering are underrepresented in the models they found, which is a key finding for what’s missing in this ecosystem.
Lu: And on the model side, they looked at the base models and saw that established ones like meta llama two seven billion have supported nine of these Pre-Trained Models since their introduction. But newer ones like spestly atlas flash seven billion are serving as the base for thirty-six Pre-Trained Models within a few months of release.
Meng: That tells you something about how fast the ecosystem is moving and how quickly new models are being adapted to these specific tasks.
Lalam: And they noted that permissive licenses like apache two point zero and mit dominate, especially in the software implementation phase, which aligns with prior work on Hugging Face showing a preference for licenses that allow broad use.
Tom: But here’s the part that’s a bit concerning—a lot of these models score below fifty percent on several common benchmarks. So even though they found thousands of SE Pre-Trained Models, many tasks still seem largely unsolved by the models themselves.
Jane: That speaks to the evaluation side. It seems like while there are lots of potential resources, a lot of them haven't proven they actually work reliably for specific engineering tasks yet.
Lu: They also found that while most SE Pre-Trained Models score below fifty percent on several common benchmarks, the number of SE Pre-Trained Models has increased markedly since the second quarter of two thousand twenty-three <ref:2506.03013#pg2>.
Meng: That growth is impressive, but if only nine point six percent of them report benchmark results, it means we’re still mostly flying blind when trying to trust these models for production work right now.
Paper summary: Lalam: The paper also pointed out documentation gaps too; while sixty-eight point five two percent of Pre-Trained Models have a non-empty model card, only seven point three two percent mention an SE task. That means we still have a lot of confusion when trying to find the right fit because the labels aren't consistent.
Tom: So what’s the big picture takeaway from all this? What does this catalogue actually do for someone who doesn't read research papers every day?
Jane: This catalogue is designed to support automated Software Engineering scenarios, like sampling and selecting suitable Pre-Trained Models based on the specific needs of your project. It gives you a structured way to look at the landscape.
Lu: And they’ve built a framework that allows for incremental updates, meaning users can retrieve only the Pre-Trained Models created after a certain date, which helps with maintainability without having to reprocess everything every time.
Meng: I like the idea of updatability; if a model gets updated or you need to switch your resources, you don't have to re-run the whole pipeline from scratch. That’s something that matters when running things at scale.
Lalam: And this entire study has motivated the ongoing development of MLAssetSelection, which is a web application that operationalizes their cataloguing pipeline to help with sampling and selection. It turns this research into a practical tool for researchers and practitioners alike.
Tom: So, to sum up, the paper provides a foundation for using AI-driven tools within Software Engineering workflows by automating how you identify and select these Pre-Trained Models based on their specific SE relevance.
Jane: The main implication is that we can move beyond just general ML tasks and start finding resources specifically tuned for coding and implementation. It helps us see where the gaps are, like in requirements engineering or design, so we know what research needs to focus on next.
Lu: And it also gives researchers a way to extend this classification to other registries like GitHub Models and PyTorch Hub, which opens up the field of PTM resources beyond just Hugging Face.
Meng: From an engineering standpoint, it’s useful because it helps us avoid wasting time testing models that aren't even targeting the right SDLC phase for our work. It’s a filter for our search.
Lalam: And they suggest that a combination of semantic and metric-based selection criteria can guide users to find Pre-Trained Models best suited to their specific context, which is how you really make smart choices with these models.
Conclusion: Tom: So, this paper is about taking all those open-source models on Hugging Face and trying to organize them specifically for Software Engineering needs.
Jane: It’s really about building a system to classify these models according to the actual tasks engineers do, like coding or designing software.
Lu: The authors built a taxonomy of one hundred forty seven different Software Engineering tasks and applied that structure to the model data they collected.
Meng: They found that most of these models are focused on code generation and coding, which is pretty clear from the numbers they gathered.
Lalam: And because a lot of them are focused on implementation, you see that software implementation is the most dominant task in terms of model coverage.
Tom: It’s interesting how they mapped this out; it shows where the existing resources are actually concentrated in the SE space.
Jane: But there are still some huge gaps, like requirements engineering and software design not being very represented in these models.
Meng: That means if you're working on the earlier stages of development, you might be looking in a very sparse area right now.
Lu: And they pointed out that many of these models score below fifty percent on common benchmarks, meaning there’s still a lot unsolved work.
Tom: So what does this cataloguing actually change for the people using these open-source models?
Jane: It gives researchers and practitioners a structured way to find the right model for their specific project needs, whether it's by license or by what kind of task it targets.
Meng: It helps automate the selection process, which is useful because you don't want to spend hours manually sifting through thousands of models.
Lalam: And they even mentioned a framework that lets you update the catalogue incrementally, so it stays current without needing a total reprocess every time.
Tom: This work sets up a foundation for using AI tools in SE workflows by making the identification and selection process more systematic.
Jane: It’s motivating because it shows how we can move beyond just general language tasks and start finding resources tuned for specific engineering activities.
Lu: The authors are also looking at extending this classification to other model registries like GitHub Models and PyTorch Hub, which broadens the scope of what this cataloguing can handle.
Meng: It’s a good way to map out where the current SE PTM ecosystem is weak so we know where future research needs to go.
Tom: And they suggest using a mix of semantic understanding and metric testing to guide users toward models that fit their context best.
Jane: It moves the decision-making process from just picking a model based on its name to picking one based on what it actually does for you.
Lu: The study shows clear patterns in how data is reused across different SE tasks, which gives us hints about what training sets are most valuable.
Lalam: They also highlight that permissive licenses like Apache two point zero are really popular, especially in the implementation phase, which points to a preference for broad use.
Tom: So this catalogue isn't just a list; it's becoming a tool for making smarter choices when you're building something with AI models.
Jane: And if you want to see how this classification applies beyond just Hugging Face, we’ve got more on that coming up next.
Alexandra Gonz´alez, Xavier Franch, David Lo, Silverio Mart´ınez-Fern´andez
Universitat Politecnica de Catalunya · Singapore Management University
cs.SE, cs.LG
Submitted: 2025-06-03
Updated: 2026-11-09
Code: https://github.com/emsejournal/openscience
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Importance score: 83/100
The gist: Open-source Pre-Trained Models (PTMs) provide extensive resources for various Machine Learning (ML) tasks…
Key concepts
- SE Task Taxonomy
- This is a structured classification system containing 147 specific tasks related to Software Engineering activities. It was built by refining prior research and incorporating concepts from the Software Engineering Body of Knowledge (SWEBOK) to provide a standardized way to categorize ML models for SE use.
- PTM Coverage Analysis
- This involves analyzing which SE tasks are most represented across the collected PTMs. The study found that software implementation, specifically code generation and coding, is the most dominant activity covered by existing models, while early stages like requirements engineering are under-represented.
- Model Card Gaps
- Model cards provide documentation for PTMs. The analysis revealed a gap: although most PTMs have model cards, only a small percentage explicitly mention an SE task. This highlights a need to address duplicated cards and improve contextual understanding of the tasks.
- MLAssetSelection
- This is an ongoing web-application designed to operationalize the cataloguing pipeline. It helps users sample and select suitable PTMs for specific SE projects based on criteria like license type or reported evaluation scores.
Terminology
Summary
Open-source Pre-Trained Models (PTMs) provide extensive resources for various Machine Learning (ML) tasks, yet these resources lack a classification tailored to Software Engineering (SE) needs to support the reliable identification and reuse of models for SE
Pruning redundant reasoning tokens
The objective is to address the gap in classification by deriving a taxonomy encompassing 147 SE tasks and applying an SE-oriented classification to PTMs in the popular open-source ML repository, Hugging Face (HF) The method involved a five-phase pipeline, including identification of SE tasks from literature, collection of PTM data from the HF API, text processing for consistency, two-phase validation involving humans and LLM assistance with five pilot studies, and data analysis This process yielded a curated catalogue of 2,205 SE PTMs
How the training signal is built
The taxonomy of 147 SE tasks was built on prior research and established guidelines, extending a foundation from Hou et al. (2024) This taxonomy broadened the scope by refining definitions for SE activities and enriching the set of tasks using concepts from the Software Engineering Body of Knowledge (SWEBOK) The data collection phase involved accessing the HF API to retrieve model card descriptions, metadata, and arXiv paper abstracts Data processing included text normalization via tokenizing, lowercasing, and lemmatizing to standardize textual descriptions
Where the gains come from
The analysis revealed that most SE PTMs target code generation and coding, emphasizing implementation over early or late development stages Text generation dominates within SE PTMs in terms of ML tasks The number of SE PTMs has increased markedly since 2023 Q2, while evaluation remains limited, with only 9.6% reporting benchmark results, mostly scoring below 50%
The catalogue reveals documentation and transparency gaps It highlights imbalances across SDLC phases and provides a foundation for automated SE scenarios, such as the sampling and selection of suitable PTMs
The study found that software implementation is the most dominant SE activity in terms of PTM coverage, with 2,704 models Code generation and coding are the most represented SE tasks Software implementation-related PTMs outnumber those for early and late SDLC stages
The analysis of base models showed that established models such as meta-llama/Llama-2-7b-hf have supported 9 PTMs since their introduction, while newer models like Spestly/Atlas-Flash-7BPreview serve as the base for 36 PTMs within a few months of release Most SE PTMs score below 50% on several common benchmarks, showing many tasks remain largely unsolved
Permissive licenses such as Apache 2.0 and MIT dominate, particularly in the software implementation phase This dominance aligns with prior works on HF, underscoring the preference for licenses that enable broad use, modification, and redistribution with minimal barriers The findings motivate the ongoing development of MLAssetSelection to assist in the sampling and selection of SE PTMs
The study identifies documentation gaps, noting that 68.52% of PTMs have a non-empty model card, but only 7. p. 032% mention an SE task Correct classification requires addressing duplicated model cards and contextual misinterpretations of SE tasks The catalogue supports informed selection based on project needs, such as license type or reported evaluation
The findings reveal that software design and requirements engineering are under-represented, highlighting a need for broader coverage across the SDLC The most prevalent ML task across SE-related PTMs is text generation, while code generation is the most commonly addressed SE task This snapshot highlights existing gaps in SE resources and provides valuable insights into the field’s evolving needs
The framework supports incremental updates, allowing users to retrieve only the PTMs created after the cutoff date, ensuring efficient maintainability without redundant processing The value of this work lies not only in the static snapshot provided but in the updatability of the framework The catalogue is designed to be extensible and interoperable with other existing datasets
The study's findings have also motivated the ongoing development of MLAssetSelection, a web-application that operationalizes our cataloguing pipeline to assist in the sampling and selection of SE PTMs The work provides a foundation for researchers to delve deeper into the challenges uncovered in this study and to conduct empirical studies using the collected SE PTMs The replication package includes the complete pipeline described in Section 3, enabling full replicability of our results
Potential threats to validity include internal validity risks from incomplete or ambiguous documentation, and external validity concerns regarding other registries The construct validity of the SE task taxonomy builds upon existing literature, which may evolve as the SE community introduces new standards The study mitigates subjective bias through inter-coder agreement practices and LLM validation
The research suggests that certain phases of the SDLC remain underrepresented, particularly requirements engineering and software design, where few PTMs explicitly target these activities The catalogue provides a foundation for researchers to delve deeper into the challenges uncovered in this study and to conduct empirical studies using the collected SE PTMs The work aims to extend this classification to other registries, such as GitHub Models and PyTorch Hub PyTorch Foundation The findings reveal important gaps and opportunities in the current SE PTM ecosystem
The study concludes by suggesting that a combination of semantic and metric-based selection criteria can be set to guide users to find PTMs best suited to their specific context This process demonstrates how our framework can support decision-making in real-world scenarios The findings have also motivated the ongoing development of MLAssetSelection, a web-application that operationalizes our cataloguing pipeline to assist in the sampling and selection of SE PTMs The work contributes to automating the sampling and selection of relevant PTMs for researchers and practitioners The study is fully replicable, with all data, scripts, and resources publicly available
The work supports the adoption of ML-driven tools within SE workflows by automating the identification and selection of PTMs The research reveals that software design is related to a single ML task: text generation This imbalance highlights the urgent need for greater research and tooling support for early SDLC activities, such as requirements engineering and software design The study is structured to analyze PTMs across temporal, structural, and legal dimensions
The analysis of training datasets shows clear patterns of dataset reuse across SE tasks, with codeparrot/github-code-clean being among the most commonly reused datasets No PTMs targeting requirements engineering report associated training datasets Permissive licenses such as Apache 2.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Automated SE PTM Selection based on SDLC Phase: The framework supports
automated SE scenarios, such as the sampling and selection of suitable PTMs,
allowing users to select models tailored to specific development stages from requirements engineering through software maintenance. -
Enhanced Model Filtering for Contextual Accuracy: The system uses a two-phase validation involving an LLM and human annotators, refining the catalogue by resolving
contextual misinterpretations of SE tasks
such as distinguishing betweensystem/software requirement
and functional requirements in requirements engineering. -
Temporal and Usage Analysis for Proactive Resource Management: The analysis tracks
cumulative growth curves to describe the evolution of SE-related models over time,
enabling practitioners to identify emerging trends, such as the rapid increase in PTMs related to software implementation since 2023 Q2. -
Cross-Activity Reuse Identification: The system visualizes pairwise relationships between activities, showing that
software implementation and software maintenance
exhibit the strongest co-occurrence, allowing users to find models suitable for reuse across mid- and late-stage activities. -
License Compliance Filtering: Users can filter PTMs based on license type, facilitating the selection of models aligned with organizational policies by identifying which licenses are used predominantly in specific activities like software implementation (e.g., Apache 2.0).
Sources
Related papers
- ArkEval: Benchmarking and Evaluating Automated CodeRepair for ArkTS
- TestPrism: Rethinking Test Evaluation Beyond a Single Reference
- Mining Workflow Graphs for Black-Box Boundary Testing of Conversational LLM Agents
- Falsification-Based Verification of LLM-Generated Optimization Models: Sound Test Batteries and Their Detection Limits
- GitSkills: A Dataset of Agent Skills on GitHub
- For Coding Agents, the Repository Matters