DAG Card is the new Model Card

The DAG Card proposal introduces Directed Acyclic Graph Cards as an extension of Model Cards, advocating for documenting Machine Learning pipelines rather than just models to capture the data-centric view crucial for real-world deployments.

The gist: We propose DAG Cards as a form of documentation encompassing the tenets of a data-centric point of view, arguing that Machine Learning pipelines are the most appropriate level of documentation for many practical use cases.

From models to pipelines

Data-centric AI sits at the conjunction of two theses: on one side, modelling has been increasingly commoditized, as deep learning’s inherent flexibility and pre-trained models provide good out-of-the-box performances; on the other, the Cambrian explosion of MLOps tools brought more attention on the fundamental importance of what happens before (data collection, preparation, labeling, quality [17]) and after (testing, serving, monitoring, drifting) the modelling stage. The paper moves from stand-alone scripts to the structured concept of DAG: while dependency management was present in earlier frameworks as well [5], Metaflow goes one step further, and provides the possibility of picking per task a particular combination of Python packages and computational resources. Data-centric AI understands that all tasks – not just modelling – play a crucial role in determining the final behavior of the system, when deployed.

Modern ML pipelines with Metaflow

Metaflow was developed at Netflix and nudges users into following data-centric best practices through its design. Metaflow provides three key features to support reproducible and robust ML pipelines:

1. workflow definition through DAG: by structuring the application as a DAG-based workflow, it replaces sets of ad hoc scripts and unreliable piecemeal execution.

2. Reproducible, isolated execution of each step: Metaflow packages together user-defined code and 3rd party dependencies in an immutable code package, which can be shipped and executed on various compute layers.

3. Abstraction of computation and scheduling: Metaflow comes with built-in integrations to popular cloud-based batch compute layers, such as AWS Batch or Kubernetes, and schedulers, like AWS Step Functions.

Moreover, Metaflow and data-centric AI share the core assumption that the final behavior of ML systems is due to the interaction of two equally important components: modelling and data. As such, Metaflow ships with additional features that make it an ideal tool to support automated generation of DAG Cards.

DAG Cards

The authors designed DAG Cards as a team of practitioners leveraging their extensive combined experience in ML for e-commerce, customer service and the entertainment industry. They highlight the following usability and philosophical principles:

1. low effort: cards require a small effort from the developer – as long as the code is wellcommented/decorated, changing the code automatically results in new, up-to-date documentation. Embracing the “documentation-as-code” idea has two intended consequences: on the organizational side, we don’t promote the hand-over of explaining a system from engineers to PMs, but instead require ML developers to be responsible for (at least some of) the explanatory artifacts; on the practical side, we remove the additional step – often neglected, postponed, forgotten – of updating some other website after pushing working code; every time this step is forgotten, documentation gets stale, less useful and in some cases even potentially harmful.

2. versioning: cards are artifacts themselves and can be generated after each run with pointers to the dataset, model weights, and all other artifacts in use; every card can be stored as an immutable, self-sufficient record of a particular run; as such, practitioners are able to quickly know how to debug specific states or even re-run the Flow entirely.

3. extensibility: cards can be extended to incorporate all sorts of information (including interactive elements) and, more generally, act as a one-stop repository for data stored in multiple tools (e.g. retrieve accuracy from an experiment tracking system).

A sample DAG card includes features such as Title and menu, DAG description rendered through Python introspection using obj. doc, Ownership showing the distribution of runs, Structure and parameters printing out the DAG for easy visual inspections of dependencies and tracking parameters, Model architecture and training info reporting on training loss/accuracy per epoch chart from a third-party tool [6], and Behavioral tests which encourage collaboration between engineering and product teams.

Conclusion

The authors argue that ML pipelines are the most appropriate level of analysis for ML systems, and they propose a first open-source implementation of DAG Cards within the official Metaflow codebase. They conclude that documenting pipelines, not just (commoditized) models, will be more and more important for all stakeholders involved in the next wave of ML systems.

The DAG Card implementation is released within the official Metaflow codebase to provide practical guidance and can be extended to other DAGs by open-sourcing its principles. The paper suggests that documentation-as-code requires developers to be responsible for explanatory artifacts, removing the neglected step of updating external websites after pushing code. DAG Cards serve as a first step toward documenting the entire ML pipeline for all stakeholders involved in future ML systems. The work is an open implementation designed to generate cards from code using Metaflow [4].

The paper is supported by acknowledgments thanking reviewers Andrea Polonioli, Luca Bigon and Patrick John Chia from Coveo, as well as Parker Barnes from Google. The references cover foundational work on Model Cards [1], data-centric AI [9], MLOps tooling landscape [10], and behavioral testing of models with checklists [16]. The paper is available on arXiv as 2110.13601v2. The paper was presented at the 35th Conference on Neural Information Processing Systems (NeurIPS 2021), Sydney, Australia. The paper includes references to Luigi [5], and various other relevant works in the field such as those by Molino and Ré [14]. The paper is available on GitHub under jacopotagliabue/dag-card-is-the-new-model-card. The paper is published in the arXiv repository with the identifier 2110.13601v2. The paper discusses behavioral testing of recommender systems with reclist [7]. The paper references experiment tracking with weights and biases [6]. The paper references the dataflow model [2].

The DAG Card is the new Model Card because it documents Machine Learning pipelines rather than just models to capture the data-centric view crucial for real-world deployments. This approach addresses the growing need to understand how ML systems work in production, moving documentation focus from static models to dynamic, reproducible workflows. The proposal leverages Metaflow's workflow definition capabilities to structure applications as DAGs, replacing ad hoc scripts with structured execution flows. By generating cards automatically from code through comments and introspection, the effort required for documentation is minimized while ensuring that documentation remains up-to-date with the code. This system emphasizes versioning by creating immutable artifacts for each run, allowing practitioners to debug specific states or re-run flows entirely. Ultimately, DAG Cards aim to make ML pipeline documentation accessible and useful for all stakeholders involved in deploying complex ML systems. The paper provides a first open-source implementation of DAG Cards within Metaflow, offering practical guidance for extending this concept to other tools. The focus on data quality, feature preparation, and choice of metrics influences the output of the system in the wild, necessitating documentation that covers these aspects. The paper is a significant step towards documenting ML pipelines instead of just models as ML systems become more complex and widely deployed.

1. Data-centric AI pipeline documentation via DAG Cards allows for low effort documentation where changing the code automatically results in new, up-to-date documentation, shifting developer responsibility for explanatory artifacts. This enables ML developers to be responsible for documenting the entire lifecycle, from data quality to behavioral testing.

2. Reproducible and self-sufficient run records via DAG Cards enable practitioners to quickly know how to debug specific states or even re-run the Flow entirely by storing code, results and execution metadata are automatically tracked and persisted in an immutable data store.

3. Extensible artifact repository functionality allows DAG Cards to incorporate external data, enabling them to act as a one-stop repository for data stored in multiple tools, such as retrieving accuracy from an experiment tracking system. This overcomes the limitation of static internal wiki pages which paint an incomplete picture.

4. Behavioral testing integration within the card structure encourages rigorous evaluation by including sections for Behavioral tests to highlight performance deficits on specific user subsets or check for regression errors, which is a key component of documenting how ML systems perform in the wild.

With the progressive commoditization of modeling capabilities, data-centric AI recognizes that what happens before and after training becomes crucial for real-world deployments. Following the intuition behind Model Cards, we propose DAG Cards as a form of documentation encompassing the tenets of a data-centric point of view. We argue that Machine Learning pipelines (rather than models) are the most appropriate level of documentation for many practical use cases, and we share with the community an open implementation to generate cards from code.