A Framework for the Systematic Review of ML Assets in AI Registries

arXiv:2610.09551 · cs.LG · Submitted 2026-10-07 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: Today's paper: "A Framework for the Systematic Review of ML Assets in AI Registries".

Jane: The gist: This work proposes a framework for systematically retrieving ML assets from AI registries, inspired by systematic reviews of scientific literature,

Tom: First, who's behind it and why it matters.

Title and authors: Jane: So, to summarize the main point of "A Framework for the Systematic Review of ML Assets in AI Registries," it’s that current methods for picking machine learning assets are too informal and lack a solid structure.

Tom: Right. They argue that because these assets—models and datasets—are so varied, you can't just pick them based on intuition or some quick look at popularity metrics >

Lu: The paper proposes using the structure of a systematic literature review to systematically retrieve these ML assets, treating them as the main items of study >

Meng: It’s like they are creating a repeatable recipe for finding useful AI components instead of just hoping you stumble upon something good >

Jane: That’s right. They detail the phases: planning, conducting, and documenting—basically taking a standard research method and adapting it to the messy world of AI registries >

Tom: The key takeaway is that this framework aims to make asset selection transparent. Instead of saying "I picked this model because it looked good," you can show exactly how you applied the protocol >

Jane: Exactly. It’s about moving from ad hoc choices to an evidence-based, traceable process for finding ML assets that are actually reliable >

Lu: They focus on defining clear search strategies that handle the different ways these registries organize their data, which is a big challenge because they aren't standardized >

Tom: So they define how you identify the target registries—whether you’re looking at model-centric or dataset-centric ones—and how you get access to that data, like using APIs or downloading dumps >

Jane: And then they explain how to select assets based on multiple dimensions, not just one metric, but things like scope alignment and maintenance activity >

Lu: They also stress the importance of quality assessment, where you check how well the asset supports claims about its use to make sure it minimizes bias and maximizes validity >

Meng: I like that they focus on traceability. Being able to trace a model back to its training data lineage is something I really need when we’re debugging production issues >

Jane: And they integrate snowballing, which means tracing those dependencies backward through the ecosystem, which helps build a complete picture of what you’re actually using >

Tom: So the summary really boils down to building a rigorous, documented pipeline that lets anyone replicate your asset selection process >

Lu: It shifts the focus from just looking at single assets to analyzing them within their interconnected network of models and data >

Jane: And they even plan for an iterative process, recognizing that AI registries aren't static, so you have to re-run the search as things change >

Meng: So it’s not a finish line; it’s a continuous loop of refinement where you keep refreshing your selection criteria as new assets appear or old ones get updated >

Tom: That iterative nature is what makes this framework useful for dynamic systems, because static lists just become outdated quickly >

The paper's summary: Jane: Now we look at the specific improvements the authors suggest to this framework. They’re not just proposing a new idea, but they are refining how you execute each step of that systematic review process.

Tom: They are adding specific operational details to things like the search strategy and the quality assessment strategy to make them more concrete >

Lu: For example, in the search strategy section, they suggest defining how you handle missing information explicitly rather than just ignoring it, which is important when dealing with heterogeneous data >

Meng: That’s practical. In a real-world scenario, you know some registries will be missing metadata for certain asset types; the framework suggests flagging that gap >

Jane: And then they push for multi-dimensional selection criteria. Instead of picking one thing, you need to consider scope alignment, popularity, maintenance activity, traceability—all at once >

Tom: And they emphasize that quality assessment needs to focus on whether the claims about the asset can be independently verified by checking for bias and validity >

Lu: They also formalize the data extraction and synthesis strategies. This means deciding exactly which attributes you are extracting from model cards, for instance, and how you’ll harmonize them >

Meng: That harmonization is key. If one registry calls a parameter 'learning rate' and another calls it 'lr', they need a way to map those so the analysis actually works across different sources >

Jane: And they suggest using stratified sampling during synthesis to get a truly representative subset from that large corpus of assets >

Tom: The iterative process improvement is also key—they make sure that after validation, you keep re-executing the conduct review phase as registries evolve, and updating the documentation to capture those version snapshots >

Lu: That versioning aspect is what makes it robust for dynamic systems; you aren't just looking at one snapshot in time, but a history of how things changed >

Meng: For deployment pipelines, having those version snapshots means if we need to roll back or compare two versions of an asset, the traceability is already built into the documentation >

Tom: So these improvements are all about taking the abstract idea and giving you concrete mechanisms for how to actually build this retrieval system >

The paper's improvements: Jane: So, wrapping up on "A Framework for the Systematic Review of ML Assets in AI Registries," the main implication is that we can finally get a structured way to explore and select machine learning assets in AI registries.

Tom: It’s about operationalizing reproducibility at the level of ecosystem exploration. We are moving away from ad hoc choices toward a traceable process for finding useful components >

Lu: This framework sets a foundation for more systematic ML asset discovery by treating the entire registry landscape as a research corpus you can study methodically >

Meng: For us on the engineering side, this means we have a standardized way to explore potential assets rather than just relying on whoever has the most buzz or who seems closest to us right now >

Jane: It gives researchers and engineers a way to build trust in the assets they use by proving their selection process was systematic and evidence-based >

Tom: Yeah, it's about making sure that when we build something with an AI asset, we can actually explain *why* we picked it based on a clear protocol >

Lu: It’s a solid methodological contribution because it bridges the gap between traditional scientific review and the dynamic reality of AI ecosystems >

Meng: I think the versioned review snapshots are going to be really useful when we start thinking about long-term asset management, as things change over time >

Jane: That sounds like a good place to leave it. It’s a framework for systematic discovery, and it’s built on the idea of making that selection process transparent and reproducible >

Conclusion: Jane: So we’ve been looking at "A Framework for the Systematic Review of ML Assets in AI Registries," and essentially, they’ve taken the structure of a scientific review and tried to apply it to finding machine learning assets in those messy AI registries >

Tom: Yeah, the whole point is making asset selection transparent. It’s not just picking a model because it looks popular; you have to show your work on how you picked it using this protocol >

Lu: The framework adapts the phases of a systematic review—planning, conducting, documenting—to treat ML assets as the primary units of analysis in these registries >

Meng: It’s ambitious. They are trying to bring rigor to something that’s inherently dynamic and unstructured >

Jane: It gets into defining a search strategy for those registries, which they say has to account for how different the registries are—like model-centric versus dataset-centric ones >

Tom: Right. So you gotta tell the system if you're looking at Hugging Face or Kaggle first, because their metadata is totally different >

Lu: And then they talk about multi-dimensional selection criteria, so you’re not just picking based on one thing like popularity but also scope alignment and maintenance activity >

Meng: I like that they focus on traceability. Being able to trace a model back to its original training data lineage is something we need when we are debugging production issues >

Jane: And they insist on using backward snowballing, which means tracing those dependencies backward through the entire ecosystem to see where the asset came from >

Tom: The quality assessment part is interesting too. They suggest you evaluate how well the asset minimizes bias and maximizes internal and external validity, checking if claims can be independently verified >

Lu: And they build in iterative loops. Because AI registries change constantly, they suggest re-executing the conduct review phase as the registries evolve, updating the documentation to keep a version snapshot of what was found >

Meng: That versioning is crucial for us. If we deploy something and then it gets updated or deprecated a month later, we need to know which specific snapshot of that registry we were working with >

Jane: It shows how this can actually support evidence synthesis over these evolving AI registries, positioning asset retrieval as a versioned process >

Tom: So, to wrap up "A Framework for the Systematic Review of ML Assets in AI Registries," it's a way to operationalize reproducibility when you’re exploring the massive landscape of AI assets >

Lu: It gives researchers and engineers a foundation for systematic discovery by treating asset retrieval like a formal research process >

Meng: It’s about building trust in the components we use, which is necessary when we’re shipping models to production at scale >

Jane: So, if you’re looking at how to systematically map and understand the AI ecosystem instead of just picking things randomly, this paper offers a solid starting point >

Tom: Yeah. We'll be taking a quick break and then we’re going to look at some work on how memorization is actually encoded in the geometry of those diffusion models. Stay with us.

Alexandra Gonzólez, Quim Motger, Xavier Franch, Silverio Mart᧭ez-Fern궭ez

Universitat Politècnica de Catalunya - BarcelonaTech (UPC)

cs.LG

Submitted: 2026-10-07

Updated: 2026-10-07

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 83/100

The gist: The gist: This work proposes a framework for systematically retrieving ML assets from AI registries, inspired by systematic reviews of scientific literature, to make their selection transparent and

Key concepts

Systematic Review Framework
Adapts the standard phases of scientific literature reviews (planning, conducting, documenting) to treat ML assets as primary units. It provides a structured, step-by-step process for identifying and selecting relevant assets from complex AI registries.
ML Asset Retrieval
The core task of systematically searching and selecting pre-trained models, datasets, and benchmarks from various AI registries. This involves defining search strategies that account for the structural differences between different registries.
Iterative Process for Dynamic Registries
Recognizes that AI registries change frequently. The framework requires an iterative loop where the review is re-executed to capture updates, ensuring the retrieval process remains current and reflects the evolving ecosystem.

Summary

The gist: This work proposes a framework for systematically retrieving ML assets from AI registries, inspired by systematic reviews of scientific literature, to make their selection transparent and reproducible

Background and Problem

Modern software systems increasingly rely on Machine Learning (ML) assets such as pre-trained models, datasets, and benchmarks for building, evaluating, and integrating ML-based systems. Current practices for selecting these assets remain ad hoc and are rarely documented in a systematic way. This lack of standardization reflects a fundamental mismatch between ML assets and traditional scholarly publications.

The Mismatch Between Scientific Reviews and AI Registries

Traditional evidence synthesis methodologies, such as Systematic Literature Reviews (SLRs), are grounded in document-centric corpora, assuming structured and standardized data. In contrast, AI ecosystems are dynamic, heterogeneous, and governed by evolving and partially implicit dependency relationships. This mismatch is summarized in Table 1 which contrasts the aspects of systematic reviews of scientific papers and ML assets. The key differences include the primary unit of analysis, data sources, metadata quality, and relationships.

The Proposed Systematic Review Framework

The framework adapts the phases of systematic reviews—planning, conducting, and documenting—to Artificial Intelligence (AI) registries treating ML assets as first-class units of analysis. The process is structured into distinct steps:

  1. Identify Review Need.

  2. Specify Search Question, which requires translating a study’s high-level objective into a specification of the ML asset type and its intended role within the study.

  3. Develop Review Protocol, which must define elements such as search strategy, identification strategy, selection criteria, and quality assessment strategy.

  4. Validate Review Protocol by executing pilot searches over a representative subset to confirm the search strategy retrieves relevant ML assets.

  5. Conduct the Review by executing the search strategy, applying inclusion, exclusion, and quality assessment criteria, and employing snowballing.

  6. Select ML Assets by examining multiple complementary dimensions such as scope alignment, popularity, maintenance activity, traceability, and metadata completeness.

  7. Assess ML Asset Quality by evaluating the strength of information supporting use in terms of minimizing bias and maximizing internal and external validity.

  8. Extract Data by harmonizing representations of equivalent attributes and extracting relevant unstructured information from model cards and dataset cards.

  9. Synthesize Data by applying a stratified sampling approach to extract a representative subset from the large corpus.

  10. Document the Review by recording search strategies, selection decisions, and extracted data in a structured report.

Key Methodological Components

The framework incorporates several advanced retrieval and analysis techniques tailored for AI registries:

((

(Search Strategy): This specifies target registries while accounting for their structural heterogeneity and considering their primary focus, distinguishing between model-centric, dataset-centric, and mixed-purpose registries. The identification strategy determines access methods like public APIs or downloadable registry dumps. The selection criteria must be adapted to the target registries' structure, potentially handling missing information explicitly. The quality assessment strategy defines how ML assets will be evaluated for verifiability of reported claims and internal/external validity. The data extraction and synthesis strategy specifies which attributes and dependencies will be extracted from selected ML assets. The dissemination strategy covers reporting guidelines, versioning practices, and the planned schedule for conducting and re-executing the review.

Iterative Process for Dynamic Registries

Given the dynamic nature of AI registries, a one-off execution is insufficient, necessitating an iterative loop over the Conduct Review and Document Review phases. The Plan Review phase remains fixed after validation to ensure consistency across iterations. During the iteration, the Conduct Review phase is re-executed to refresh retrieval, filtering, and analysis as registries evolve. The Document Review phase is updated to record new versioned snapshots and preserve traceability of changes over time.

Conclusion

This framework illustrates how systematic review principles can be extended beyond scientific literature to support evidence synthesis over evolving AI registries. It positions ML asset retrieval as a versioned process, operationalizing reproducibility at the level of ecosystem exploration. The framework offers a foundation for more systematic ML asset discovery.

Future Work

Future work will focus on further formalizing the proposed methodology into a more operational framework, including mapping its steps against existing literature to identify established, partially addressed, and implicit practices. Researchers also aim to investigate how the framework can be empirically validated and envision the construction of a knowledge graph of the AI ecosystem where models, datasets, and their relationships are explicitly represented.

Data Availability

This paper presents a conceptual and methodological framework, stating that no datasets, code, or other research artifacts were collected or analyzed as part of this article.

References Summary

The references cited support the need for structured retrieval methods in SE, the analysis of AI ecosystems as interconnected networks, and the role of model cards and datasheets in documentation. The paper builds upon established practices from SLRs and SMSs while addressing the unique challenges posed by heterogeneous AI registries.

Key References

The references include foundational work on systematic literature reviews in SE, studies on mining Hugging Face repositories, and recent empirical studies involving LLMs in software engineering. The paper also references work on ecosystem analysis and knowledge graphs for MLSea. The inclusion of literature on model cards and documentation practices is significant."

--- Page 1 ---

A Framework for the Systematic Review of ML Assets in AI Registries

Assets in AI Registries

Alexandra González, Quim Motger, Xavier Franch, Silverio Martínez-Fernández

--- Page 2 ---

23:2 A Framework for the Systematic Review of ML Assets in AI Registries

[23] aim to improve transparency and standardize artifact documentation, they do not provide methodological guidance for systematically retrieving ML assets in AI registries. Furthermore, different registries expose heterogeneous schemes and traceability signals. For instance, registries such as Hugging Face provide popularity indicators (e.g., likes) and explicit traceability links to downstream models trained or fine-tuned on a given dataset. In contrast, others, such as Kaggle, rely on alternative metrics (e.g., upvotes) while offering limited support for tracing reuse. This leads to uneven levels of traceability across registries where relationships between ML assets may be explicitly documented or only implicitly inferred. When considered jointly, these heterogeneous signals form a complex network spanning links between datasets and models, model-to-model adaptations (e.g., quantization, fine-tuning) [35], and dataset-to-dataset lineages (e.g., subsets, augmentations) [20]. This network is critical as a model is unlikely to perform reliably in production if its deployment context diverges from its training data, or if these datasets reflect unwanted biases [13, 26].

Improvements for AI systems

  1. Bold Header: Systematic ML Asset Retrieval Framework

This framework enables structured ML asset discovery by adapting systematic review phases (planning, conducting, documenting) to AI registries, ensuring selection is transparent and reproducible.

  1. Bold Header: Heterogeneous Registry Search Strategy

The protocol must define a Search strategy that specifies target registries while accounting for their structural heterogeneity and considering their primary focus (e.g., model-centric vs. dataset-centric).

  1. Bold Header: Multi-Dimensional Asset Selection Criteria

Selection criteria should incorporate multiple dimensions, such as scope, traceability, and maintenance activity, allowing for tailored asset filtering based on the specific research goal and registry signals.

  1. Bold Header: Traceability via Snowballing

The process must employ backward snowballing to identify resources from which an ML asset originates (e.g., tracing the base model), ensuring a comprehensive understanding of dependency structures.

  1. Bold Header: Quality Assessment for Internal/External Validity

Quality assessment should focus on the extent to which bias is minimized and internal and external validity are maximized, evaluating whether claims can be independently verified.

  1. Bold Header: Versioned Review Snapshots

The documentation must include Version snapshots to preserve the state of AI registries at specific points in time, addressing the challenge that ML assets are continuously updated, modified, or deprecated.

Sources

Related papers