AIDA: Analysis of Interval Data in R

summary

The gist

AIDA is an R package that provides tools for the analysis of interval-valued data, including construction, visualization, robust estimation, and explainable outlier detection.

In short

AIDA is an R package for analyzing interval-valued data, offering tools for construction, visualization, robust estimation, and explainable outlier detection. It introduces methods like IMCD for robust location/covariance estimation and uses Shapley values to decompose outlier importance into variable contributions.

Key concepts

Symbolic Data Analysis (SDA)
SDA is a statistical field focused on analyzing complex data like lists or intervals, using structured representations instead of single numbers. Interval data is a key type of symbolic data, providing a structured view of underlying variability in applications like climate records.
intData Class
This S4 class represents interval-valued observations within the AIDA package. It stores interval information using center-range parameterization and includes latent variables to capture underlying microdata structure, allowing for detailed modeling of uncertainty.
IMCD Estimator
The Interval Minimum Covariance Determinant (IMCD) estimator is used for robust estimation of the barycenter and symbolic covariance matrix. It works by finding a subset of observations that minimizes the determinant of the covariance matrix, effectively ignoring outliers to get a stable estimate.
Shapley Values
Shapley values are used to explain why an observation is an outlier by decomposing its outlyingness measure into contributions from individual variables. This quantifies each variable's average impact on the total distance of the anomaly.

Terminology used across episodes

This episode discusses

The paper

AIDA: Analysis of Interval Data in R · Read on arXiv

Catarina P. Loureiro, M. Rosário Oliveira, Paula Brito

CEMAT and Department of Mathematics, Instituto Superior Técnico, Universidade de Lisboa · Faculdade de Economia e Gestão, Universidade do Porto and LIAAD-INESC TEC

AIDA is an R package that provides tools for the analysis of interval-valued data, including construction, visualization, robust estimation, and explainable outlier detection. The package is built around the intData class and is designed to support methodological research and applied workflows involving interval-valued data. AIDA implements methods for symbolic covariance estimation based on the Mallows distance, robust estimation via the Interval Minimum Covariance Determinant estimator, and explainable outlier detection using Shapley values. The package also includes visualization tools to support interpretation of the results and several interval-valued datasets.

Transcript

Introduction to the show: ident: Statistics. The day in statistics: methods, theory and the inference under them.

Tom: We're looking at the paper: "AIDA: Analysis of Interval Data in R".

Jane: Here's the gist of the paper. AIDA is an R package that provides tools for the analysis of interval-valued data, including construction, visualization, robust estimation, and explainable outlier detection.

Tom: Foundation of Symbolic Data Analysis Symbolic Data Analysis (SDA) is a growing field of statistics dedicated to the analysis of complex data such as lists, intervals or distributions, leveraging structured or aggregated data representations rather than conventional single-valued observations Interval-valued data, in particular, constitute a common and useful type of symbolic data providing a structured representation of an underlying variability pattern This ontic perspective is central to the analysis of interval data and motivates the development of methods that account for both the location and the internal dispersion of each observation Interval-valued data arise in a wide range of applications such as climate records, economic indicators, environmental monitoring, and aggregated consumer behavior Core Data Structure and Framework AIDA is built around the intData S4 class which represents interval-valued observations while retaining the information required to model their internal variability The package supports the construction of interval-valued datasets from raw microdata or aggregated summaries, descriptive statistics and symbolic covariance matrices, robust estimation of location and scatter, outlier detection using robust IntervalMahalanobis distances, and explainability through Shapley values and interaction indices The intData class adopts a center-range parameterization storing interval-valued observations through two complementary components: a data frame of centers and a data frame of ranges It incorporates information about the underlying microdata through a latent variable framework with slots for LatentCase, LatentDist, and LatentParam Robust Estimation and Outlier Detection For robust analysis, AIDA implements the Interval Minimum Covariance Determinant (IMCD) estimator together with robust Interval-Mahalanobis distances for outlier detection The IMCD algorithm extends the FastMCD algorithm to interval-valued data providing robust estimates of both the barycenter and the symbolic covariance matrix Outlier detection is based on a robust version of this distance, where an observation is flagged as an outlier if its distance exceeds a given cutoff value Cutoff values can be obtained using a nonparametric approach based on the concept of farness Explainability through Shapley Values Shapley values are used to decompose an outlyingness measure into variable-wise contributions, quantifying their importance to the anomalous behavior Using the squared Interval-Mahalanobis distance as the value function, the Shapley value of variable j measures its average contribution to the total distance The Shapley interaction index further extends Shapley values to quantify joint effects between variables, capturing their joint contribution to the total outlyingness Visualization and Datasets AIDA provides visualization tools such as bar plots for Shapley values and heatmaps for Shapley interaction indexes to support the interpretation of results The package includes four interval-valued datasets from different application domains, including creditcard, entrecampos air quality, intCars, and spotify tracks These datasets allow users to explore the intData class and its associated methods through various aggregation procedures The analysis on the entrecampos air quality dataset demonstrates the advantage of using robust distance for outlier detection compared to classical strategies AIDA is a package for the analysis of interval-valued data in R, supporting a complete workflow including data construction, descriptive analysis, distance-based measures, robust estimation, and explainable outlier detection A key contribution is the implementation of the IMCD estimator and the corresponding robust Interval-Mahalanobis distance providing a robust framework for identifying atypical observations These methods are complemented by Shapley value-based explainability tools including interaction indices which decompose the outlyingness measure into variable-specific contributions The package also provides visualization tools and example datasets to facilitate exploratory analysis, graphical diagnostics, and interpretation of results The package is currently in version 0.2.2 with further methods and features planned to extend its support for interval-valued data analysis Acknowledgements Generative AI (GPT-5) was used for language editing and assistance with R coding and troubleshooting, with the authors reviewing and editing the content as needed, and taking full responsibility for the published article This work was supported by FCT — Fundação para a Ciência e Tecnologia, I.P.

Jane: through grant UI/BD/153720/2021, and projects UID/04621/2025 and UID/50014/2025 References The references listed include works on Symbolic Data Analysis, Mallows distance, Minimum Covariance Determinant estimator, Shapley values, and various R packages for symbolic data analysis The references also include specific package documentation and related methodological papers such as those by Loureiro et al.

Tom: (2026a) and Oliveira et al.

Title and authors: Tom: So, we're looking at this paper called "AIDA: Analysis of Interval Data in R". It’s from Loureiro and Oliveira, and it’s basically an R package designed to handle interval-valued data.

Jane: Right. What that means for us is that instead of just looking at single numbers, we can analyze things like ranges or uncertainty directly. It sounds like a big step because traditional methods often struggle with this kind of structured variability.

Lu: Exactly. The core idea here is building an integrated framework, not just one isolated tool. They're using something called the intData S4 class to represent these interval-valued observations, which keeps track of both the center and the range of each observation.

Meng: So they're trying to get past those limitations where you only have a center point and no idea how much variation is around it. It sounds like a necessary structure for real-world data, right?

Lalam: The paper emphasizes that this framework links microdata and macrodata using a latent variable approach to give us that structured representation of variability, which is the foundation for everything else in AIDA.

Tom: Right, so the authors are building this from scratch to be comprehensive. They aren't just throwing some existing tools together; they're creating a whole workflow for data construction and analysis.

Jane: And what’s interesting is that it supports everything from building those interval datasets directly from raw microdata to robust estimation and even finding outliers. It covers the whole spectrum of symbolic data analysis.

Lu: They specifically mention supporting the construction of interval-valued datasets from raw microdata or aggregated summaries, descriptive statistics, and symbolic covariance matrices. This makes it very flexible for different starting points.

Meng: If you can go from raw microdata straight to a fully specified representation using something like the micro2intData function when LatentCase is set to General, that’s a huge win for getting data into the system efficiently.

Lalam: That process allows you to get both descriptive bounds and distributional information right away, which is crucial for setting up your analysis correctly.

Tom: And then they move on to estimation and detection. They implement robust estimation using the Interval Minimum Covariance Determinant estimator, which I think is key because it minimizes the determinant of the covariance matrix by finding a subset of observations that are least contaminated by outliers.

Jane: That's a specific technique, so what does that actually do for us when we’re dealing with messy data? It helps them get a more stable estimate of the location and scatter when there are some weird points in the data.

Lu: The IMCD algorithm extends the FastMCD algorithm to interval-valued data, which gives them robust estimates for both the barycenter—that's like the center—and that symbolic covariance matrix. It’s a direct improvement over standard methods for this kind of data.

Title and authors: Meng: From an engineering standpoint, that sounds promising because it means we don't have to manually clean up outliers before we can reliably estimate the underlying structure of the distribution. What breaks at scale is usually when you rely on assumptions that aren't true, and this seems to address that assumption directly.

Lalam: They also introduce robust outlier detection using robust Interval-Mahalanobis distances, which allows them to flag observations based on a distance cutoff value defined nonparametrically using something called farness.

Tom: So, if an observation falls outside that farness cutoff, it gets flagged as an outlier. That’s the practical application we need for finding anomalies in complex systems.

Jane: And they don't just flag them; they provide explainability tools too, using Shapley values to break down why something is considered outlying. This moves it beyond just saying "this point is weird" to telling us *which* variables contributed most to that weirdness.

Lu: The paper uses the squared Interval-Mahalanobis distance as the value function for the Shapley value of a variable, which tells you its average contribution to that total distance measure. That’s how you pinpoint the anomalous behavior variable by variable.

Meng: It’s cool that they also have an interaction index, which quantifies joint effects between variables, showing how variables work together when something is outlying. That gives us a fuller picture of the structure of the anomaly.

Lalam: So, instead of just seeing a single outlier flagged by a distance metric, you get a decomposition into variable-specific contributions and joint interactions. It helps us understand if it's one thing going wrong or several things happening together.

Tom: I like that decomposition part because it makes the results actionable for model tuning. If we know variable j is driving most of the outlyingness, we can focus our efforts there instead of looking everywhere at once.

Jane: So, to wrap up this section, AIDA provides a complete workflow: you build the data, you get a robust estimate with IMCD and Interval-Mahalanobis distance, and then you can use Shapley values to explain exactly what's causing the weirdness.

Lu: It’s an integrated framework that bridges the gap between microdata and macrodata representation through that latent variable linkage they developed. It shows how to handle interval data in a way that respects both its internal variability and its underlying structure.

Meng: From a deployment standpoint, if this framework can reliably estimate the symbolic covariance matrix without needing perfect initial assumptions, it makes running these kinds of models much more stable in production environments.

Lalam: The authors mention that they include four interval-valued datasets from different domains like credit cards and air quality, which gives us good examples to explore how this works in practice.

Tom: So we’ve seen the mechanics—the IMCD estimator, the Shapley values—and it sounds like a very solid package for anyone working with these kinds of complex observations.

Title and authors: Jane: It really shows that interval-valued data isn't just an academic curiosity; it's something that can be rigorously analyzed using established statistical techniques when you have the right tools.

Lu: The paper lays out how the Mallows distance is implemented through those centers and ranges, which is a concrete mathematical way to express that distance between two interval realizations.

Meng: That expression for the squared Mallows distance involves terms based on the diagonal matrix delta and another one called psi, which are derived from moments of the latent variable U. It grounds the abstract concept in actual expectations about those latent variables.

Lalam: And when they set those latent variables to be uniformly distributed between negative one and positive one, that simplifies the distance calculation significantly, showing a clear relationship between the assumed distribution and the resulting distance measure.

Tom: That simplification is pretty neat because it gives us a baseline for how fast that Mallows distance shrinks when we know something about the underlying data generation process.

Jane: So while this paper is focused on interval data, it really highlights how robust estimation and explainability tools can be tailored to handle the uncertainty inherent in symbolic representations of variability.

Lu: The authors are setting up future work around extending this framework, suggesting they plan to add more methods and features to keep pushing the boundaries of what’s possible with this approach.

Meng: I wonder if they'll focus on how computationally expensive those robust distance calculations become as the number of variables p gets really large in a real-world scenario. That’s where deployment often hits a wall.

Lalam: They did acknowledge that the paper is currently in version zero point two.two which tells us they are actively developing this package and planning extensions for future work based on their findings here in AIDA: Analysis of Interval Data in R <ref:2611.16723#pg1,AIDA: Analysis of Interval Data in R>.

Tom: So, we’ve covered the structure of the package, how it handles estimation with IMCD, and how it explains outliers using Shapley values. It's a very complete tool set for symbolic data analysis.

Jane: It really gives us a tangible way to move from raw data to an understandable conclusion about why something is anomalous in that interval-valued context.

Lu: This work provides a strong foundation, showing how to integrate those different concepts—robust estimation, symbolic covariance, and explainability—into one coherent R package.

Meng: It’s a good starting point for anyone who needs to move beyond simple point-based statistical analysis and into the realm of interval data modeling.

Lalam: The full title, "AIDA: Analysis of Interval Data in R", points to a comprehensive toolset for handling uncertainty and variability in structured data.

Tom: That’s what we have today on AIDA: Analysis of Interval Data in R. We’ll be back after the break with more new papers from arXiv.

The paper's summary: Tom: So, AIDA is an R package that takes interval data—data where you have both a center point and some kind of range or uncertainty—and gives you tools to build it, visualize it, estimate the center and spread robustly, and even figure out which variables are causing weird outliers.

Jane: It sounds like they're giving people a complete toolkit for handling that structured variability in data instead of just treating every observation as a single number.

Lu: They use this thing called the intData class, which keeps track of the centers and the ranges, and they link it to a latent variable framework that lets them build these interval datasets from raw microdata directly.

Meng: The idea that you can go straight from raw microdata to a fully specified representation using something like KDE estimation when LatentCase is set to General, that sounds like it could seriously speed up the data prep work for production models.

Lalam: And they don't just stop at the estimation; they have these robust outlier detection methods using Interval-Mahalanobis distances combined with a nonparametric cutoff called farness.

Tom: Which means you get a measure of how far an observation is from the main cluster, and that cutoff value isn't fixed; it adapts based on the data itself.

Jane: That makes sense for real-world data where you don't know exactly how much noise to expect beforehand. And then they use Shapley values to explain those outliers by showing you which variables are responsible for the distance measure.

Lu: They decompose that distance into variable contributions, so you can see exactly how important each feature is in creating that anomalous behavior. Plus, they have an interaction index to show how those variables work together when something goes wrong.

Meng: If I'm shipping models, knowing which specific features are driving the anomaly means I can debug the model or the data pipeline much faster than just seeing a flag on a row of data. It moves outlier detection from a black box to something you can actually investigate in production.

Tom: So, to summarize this part, they’re offering a full workflow: construction from microdata, robust location and covariance estimation with IMCD, and then explainable outlier detection using Shapley values and interaction indices.

Jane: It really shows how you can take the complexity of interval data—that uncertainty around every point—and give it a rigorous statistical analysis framework.

Lu: They’ve built a solid foundation here, showing how to integrate robust estimation with modern explainability techniques in one cohesive package for R users.

Meng: It’s not just about the math; it’s about having a tool that actually runs and gives you actionable insights when your data is messy.

Lalam: The paper is currently in version zero point two point two, meaning they are actively developing this, and they have plans to add more methods down the line for even deeper analysis.

Tom: What this means for people using these tools is that instead of guessing how much uncertainty you should account for when building your models, you get a statistically grounded way to estimate it and understand why things go wrong.

Jane: So if you’re working on anything involving ranges or inherent uncertainty in your observations, AIDA gives you a way to handle the structure of that data properly.

Lu: And this whole setup is really about getting past the limitations of single-point analysis by treating these observations as having both a center and an internal dispersion.

Meng: The real question for me is how much computational overhead that robust estimation, like IMCD, adds when you’re dealing with massive datasets that I'd typically be running on.

Tom: We’ll have to see if the speed of those robust methods keeps up with the complexity they are introducing here in AIDA.

The paper's improvements: Tom: We just talked about how AIDA handles data construction and robust estimation, but now we're going to look at the improvements the authors suggest for making that whole system even better.

Jane: So, what are they trying to fix in their current approach? Are there any gaps in how they handle things like outliers or how they estimate the center and spread of these interval observations?

Lu: The paper points out a few key areas where things could be tightened up. For instance, when it comes to robust estimation, the authors suggest using something called IMCD more directly to minimize that determinant of the covariance matrix even further.

Meng: Minimizing that determinant sounds like a way to make the location and scatter estimates even more resistant to those extreme outliers we talked about earlier. How does that translate to deployment?

Tom: It means if you have a few really weird data points, the resulting model parameters will be less skewed by those points. That’s good because it makes your baseline estimation more reliable before you even start running predictions at scale.

Jane: And for outlier detection itself, they propose using robust distances that rely on nonparametric cutoffs from something called farness to make that flagging process more flexible than a fixed threshold.

Lalam: So instead of setting a hard number for how "weird" an observation has to be, the method adapts its sensitivity based on the shape of the data itself. That sounds much more adaptable for different kinds of problems.

Lu: And they also really push hard on the explainability side by formalizing how Shapley values break down those outlyingness measures into specific variable contributions and interaction effects.

Tom: So, we move from just saying "this point is an outlier" to knowing exactly which inputs caused it and how they interacted with each other. That level of detail is what you need for debugging complex systems.

Jane: That decomposition sounds really powerful for understanding the underlying patterns, not just the anomalies in isolation. It helps you see the structure of the problem itself.

Meng: From an engineering standpoint, if we can get a more granular variable-level explanation, it helps us know exactly which parts of our input features are causing issues in our production environment. That cuts down on guesswork during troubleshooting.

Tom: And they also provide some more concrete examples and datasets to show how these methods work in practice across different domains like credit cards and air quality data.

Jane: So, the implication is that this isn't just theoretical math; it’s a practical package with clear ways to apply these robust estimation and explainability techniques right now.

Lu: They are essentially refining the existing framework by plugging in more mathematically rigorous estimators and linkage methods to create a more complete analysis pipeline.

Meng: I'm still focused on the computational cost, but if this refined method can achieve better robustness with less intensive computation, that’s what would make it viable for real-time monitoring.

Tom: We’ll keep an eye on those performance metrics when we look at the benchmarks later, but for now, this sounds like a really solid set of upgrades to the AIDA package.

Conclusion: Tom: So, we've covered how AIDA handles data construction and robust estimation, looking at things like IMCD for location and scatter, and how Shapley values help us explain those outliers in interval data.

Jane: It really boils down to giving us a complete workflow: you build the data from raw inputs, you get a stable estimate of the center and range using robust methods, and then you can actually understand *why* something is considered anomalous.

Lu: They’ve built this entire system around that intData class, linking it to a latent variable framework that lets them model both the center and the spread simultaneously.

Meng: And for production use, the idea of having a mechanism to decompose an outlier’s cause by variable contribution means we can target specific data sources or features for cleaning or fixing without just blindly removing everything.

Lalam: I'm confident this paper lays down a solid foundation because it connects the abstract concept of interval variability to concrete tools that work in R, providing a path forward for serious symbolic data analysis.

Tom: And the improvements they suggest? They focus on refining those robust estimators and making the explainability tools even more granular with better interaction indices.

Jane: It sounds like they're taking a very strong starting point and just polishing it to ensure it’s as stable and interpretable as possible for real-world science.

Lu: They are essentially focusing on increasing the mathematical rigor of those estimates, making sure that the symbolic covariance matrix they derive is as accurate as it can be under challenging conditions.

Meng: I'm still curious about the runtime cost of these more refined robust methods because if the calculation gets too slow, it just doesn't matter how accurate it is when you're running a high-volume inference pipeline.

Tom: Yeah, we need to keep an eye on those performance benchmarks when we look at how fast these advanced calculations run on bigger datasets.

Jane: So for someone listening who isn’t coding, the big picture here is that complex data with uncertainty can finally be analyzed using statistical methods that respect both the range and the center of the observation.

Lalam: I'm quite sure this work, AIDA: Analysis of Interval Data in R, offers a very practical way to handle structured uncertainty in any domain.

Lu: It’s a step toward building more sophisticated representation learning techniques because it shows how to properly model variability instead of ignoring it.

Meng: For deployment, having tools that can robustly estimate the structure means we don't have to spend all our time on pre-processing just to make the core modeling stable.

Tom: AIDA is a great package for anyone who needs to move beyond simple point-based statistics and into the realm of interval data modeling with a strong focus on explainability.

Jane: It’s definitely something worth checking out if you're dealing with any kind of data that has an inherent range or uncertainty attached to it.

More episodes

← Home