Fast Functional Clustering via Closed-Form Covariance Updates
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Statistics. The day in statistics: methods, theory and the inference under them.
Tom: We're looking at the paper: "Fast Functional Clustering via Closed-Form Covariance Updates".
Jane: Here's the gist of the paper. Functional clustering identifies coordinated molecular profiles by grouping features based on their expression patterns, and this paper develops a fast functional clustering procedure that combines linear mean profiles with first-order structured antedependence covariance to significantly accelerate joint analysis across multiple conditions Model Formulation for Joint Analysis The framework models distinct expression patterns using Gaussian mixture components where each feature has a single cluster membership across all conditions The observed-data likelihood is expressed as a product over features of the component log densities, which depend on condition-specific mean profiles and a shared covariance structure The mean profile for cluster k under condition l is modeled as a linear function of the standardized log niche coordinate with parameters that can vary by cluster and condition The covariance structure is modeled using a first-order structured antedependence model, SAD(1), which expresses dependence through conditional regressions on earlier ordered measurements The covariance blocks are assumed to be block-diagonal, capturing residual dependence within each condition while maintaining independence across conditions conditional on cluster membership Estimation Procedure and Parameter Updates The estimation process employs an expectation–maximization (EM) procedure where the M-step alternates between weighted least squares mean updates and direct conditional covariance updates The E-step computes posterior membership probabilities using the component log densities evaluated via innovation representation The M-step involves updating mixture proportions, mean profiles, and covariance parameters through alternating conditional updates based on weighted residual statistics For mean profiles, an unconstrained update is derived by minimizing a quadratic form involving the residual statistics Covariance parameter updates involve calculating weighted residual sums Ul, Vl, and Wl to derive new antedependence coefficients and innovation variances through closed-form expressions Computational Acceleration Results Closed-form SAD(1) conditional updates substantially accelerated covariance estimation by 91.5–201.4-fold in same-R comparisons when solving the conditional objectives Under matched complete mixture fits, this closed-form solver achieved a 1.87–2.37-fold speedup with identical cluster assignments The resulting implementation achieved median complete-workflow speedups of 156–2,913-fold relative to the original FunClu software across the benchmark configurations Model Recovery and Application Recovery simulations showed higher module classification accuracy and lower mean-curve error than K-means and a Gaussian mixture with shared full covariance when the mean structure was correctly specified SAD(1) provided an additional recovery gain under strong residual correlation, while curved-mean controls identified the importance of an appropriate mean family Applications to single-condition gut microbiota, paired leaf–root microbiota, and three-visit human gut multi-omics illustrate the framework across different numbers of conditions The analysis on gut multi-omics data selected K = 11 modules with a BIC of −2,817,008.40 Model Selection Criteria The model selection involves jointly selecting the number of clusters K and the degree of covariance parameter sharing S by minimizing the Bayesian information criterion (BIC) The total number of free parameters is defined as qK,S = (K − 1) + 2KL + qΣ(S), where qΣ(S) depends on the specification S The optimal pair is selected by minimizing BIC(K, S) = −2LbK,S + qK,S log p Conflict of Interest and Data Availability The authors declare that they have no competing interests The gut and paired leaf–root microbiota datasets were obtained from the publicly available idopNetwork package repository at https://github.com/cxzdsa2332/idopNetwork
5: The processed human gut multi-omics data are available from the source study’s figshare archive at https://doi.org/10.6084/m9.figshare.32043930
17: The repository link and persistent identifier for the FunClu implementation, simulation scripts, fitted results and figure source data generated in this study are XXX The author contributions statement details that Yu Wang designed the experiments and wrote the manuscript Closed-form SAD(1) estimation provides a substantial computational gain for functional clustering by replacing repeated numerical covariance optimization with direct updates, leading to significant speedups in both covariance solving and complete model fitting The recovery experiments further demonstrate the statistical benefit of modeling curve structure, showing that correctly specified means improve module recovery over simpler methods The resulting implementation offers a fast computational foundation for joint functional clustering across conditions References 1.
Tom: H.
Jane: Akaike.
Tom: A new look at the statistical model identification 2.
Paper summary: Jane: So to wrap up, we've talked about how the paper "Fast Functional Clustering via Closed-Form Covariance Updates" tackles the slowness of functional clustering by replacing slow numerical optimization with direct updates using a first-order structured antedependence model.
Tom: Yeah, they showed massive speedups in covariance estimation, hitting up to two thousand nine hundred thirteen times for complete workflow speedups on benchmark configurations. They also showed that their model structure helps improve module classification accuracy over simpler methods like K means and Gaussian mixture with shared full covariance when the mean structure is correctly specified.
Lu: It’s about moving from repeated searches to direct parameter updates within the expectation/conditional maximization framework, using an innovation representation and cached quadratic statistics to reduce those likelihood calculations.
Meng: From an engineering standpoint, that computational gain means you can run these analyses on much larger datasets or test more conditions without the fitting process grinding your pipeline to a halt.
Jane: And they gave us the BIC minimization criteria for selecting clusters and covariance sharing, showing how they balance model complexity against performance using that total number of free parameters q K S.
Tom: The title "Fast Functional Clustering via Closed-Form Covariance Updates" captures the core idea perfectly: it’s about speeding up the process by using a closed-form update for the covariance structure.
Lu: And they confirmed that this approach, when correctly specified, yields better recovery results compared to methods like K means and a Gaussian mixture with shared full covariance.
Jane: It seems like this is a solid tool for anyone dealing with complex longitudinal data where you need to compare many different states or conditions.
Conclusion: Tom: So, basically, this paper is about making functional clustering way faster by ditching slow numerical optimization for direct updates using a structured antedependence model for the covariance part.
Jane: I'm trying to get my head around that—so it’s taking something that used to take ages and replacing it with a closed-form equation, which is pretty big news for speed.
Lu: It replaces repeated numerical optimization with direct updates within the expectation maximization procedure, using innovation representation and cached quadratic statistics to reduce those likelihood calculations.
Meng: From an engineering standpoint, that computational gain means you can run these analyses on much larger datasets or test more conditions without the fitting process grinding your pipeline to a halt.
Tom: Yeah, so what does this actually mean? The title is "Fast Functional Clustering via Closed-Form Covariance Updates," and it points to solving the covariance problem directly instead of iteratively searching for it.
Jane: It means we can handle those complex joint analyses across multiple conditions much more efficiently, which is huge if you're dealing with time series data or repeated measurements.
Lu: The recovery experiments show that when you correctly specify the mean structure, this SAD one provides an additional recovery gain over simpler methods like K means and a Gaussian mixture with shared full covariance.
Tom: So it’s not just about speed; it’s about getting better results too, especially when the underlying data structure is complex.
Jane: Exactly. It shows that the way you model those relationships between features across different conditions actually matters for how well your clustering works.
Meng: I'm curious though, what does this closed-form update look like in practice? Is it just a neat trick or does it handle all the tricky edge cases we see when things go wrong at scale?
Tom: That’s a good question, Meng. We need to look closely at how robust these updates are when we move beyond the benchmark settings and start looking at real-world data.
Jane: Right, so while the speedups are impressive—we're talking thousands of times faster for some covariance solving steps—we still have to make sure this model structure holds up across different kinds of biological or social systems.
Lu: The BIC minimization criteria for selecting the number of clusters and the degree of covariance parameter sharing shows how they balance that complexity against performance using a total number of free parameters.
Tom: It’s a trade-off, right? You get this massive computational advantage, but you have to choose the right settings for K and S to keep the model meaningful.
Jane: And that points us toward where we might look next—finding those optimal balance points for selecting how many clusters and how much covariance structure to share.
Yu Wang, Jiaze Guo, Yao Chen, Tianxiang Hao, Rongling Wu
Institute of Statistics and Big Data, Renmin University of China · Beijing Key Laboratory of Topological Statistics and Applications for Complex Systems, Beijing Institute of Mathematical Sciences and Applications, Beijing Institute of Mathematical Sciences and Applications · School of Grassland Science, Beijing Forestry University · The Tsinghua Shenzhen International Graduate School, Tsinghua University · Yau Mathematical Sciences Center, Tsinghua University
stat.CO, stat.AP
Submitted: 2026-11-05
Updated: 2026-11-05
Code: https://github.com/cxzdsa2332/idopNetwork
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 98/100
The gist: Functional clustering identifies coordinated molecular profiles by grouping features based on their expression patterns…
Key concepts
- Functional Clustering
- A technique used to group features based on their coordinated expression patterns across different conditions. It aims to identify underlying molecular profiles by analyzing how features behave together in response to various experimental settings.
- Structured Antedependence Covariance (SAD(1))
- This mathematical model describes how the expression of one feature depends on the previous ordered measurements. It captures specific types of correlation between variables, modeling dependence through conditional regressions, which is more precise than assuming simple independence.
- Closed-Form Updates
- Instead of using iterative numerical optimization to find covariance parameters (which is slow), this method derives direct formulas for updating these parameters. This closed-form approach drastically accelerates the estimation process, leading to substantial computational speedups in fitting the model.
Terminology
Summary
Functional clustering identifies coordinated molecular profiles by grouping features based on their expression patterns, and this paper develops a fast functional clustering procedure that combines linear mean profiles with first-order structured antedependence covariance to significantly accelerate joint analysis across multiple conditions
Model Formulation for Joint Analysis
The framework models distinct expression patterns using Gaussian mixture components where each feature has a single cluster membership across all conditions The observed-data likelihood is expressed as a product over features of the component log densities, which depend on condition-specific mean profiles and a shared covariance structure The mean profile for cluster k under condition l is modeled as a linear function of the standardized log niche coordinate with parameters that can vary by cluster and condition The covariance structure is modeled using a first-order structured antedependence model, SAD(1), which expresses dependence through conditional regressions on earlier ordered measurements The covariance blocks are assumed to be block-diagonal, capturing residual dependence within each condition while maintaining independence across conditions conditional on cluster membership
Estimation Procedure and Parameter Updates
The estimation process employs an expectation–maximization (EM) procedure where the M-step alternates between weighted least squares mean updates and direct conditional covariance updates The E-step computes posterior membership probabilities using the component log densities evaluated via innovation representation The M-step involves updating mixture proportions, mean profiles, and covariance parameters through alternating conditional updates based on weighted residual statistics For mean profiles, an unconstrained update is derived by minimizing a quadratic form involving the residual statistics Covariance parameter updates involve calculating weighted residual sums Ul, Vl, and Wl to derive new antedependence coefficients and innovation variances through closed-form expressions
Computational Acceleration Results
Closed-form SAD(1) conditional updates substantially accelerated covariance estimation by 91.5–201.4-fold in same-R comparisons when solving the conditional objectives Under matched complete mixture fits, this closed-form solver achieved a 1. p. 4 Simulation experiments
Computational gains from closed-form SAD(1) updates
Closed-form SAD(1) updates substantially accelerated covariance
estimation and reduced complete fitting time. We quantified
these gains through matched-model fitting, conditional covariance
solving and original-software comparisons.Where the paper says it
Model Recovery and Application
Recovery simulations showed higher module classification accuracy and lower mean-curve error than K-means and a Gaussian mixture with shared full covariance when the mean structure was correctly specified SAD(1) provided an additional recovery gain under strong residual correlation, while curved-mean controls identified the importance of an appropriate mean family Applications to single-condition gut microbiota, paired leaf–root microbiota, and three-visit human gut multi-omics illustrate the framework across different numbers of conditions The analysis on gut multi-omics data selected K = 11 modules with a BIC of −2,817,008.40
Model Selection Criteria
The model selection involves jointly selecting the number of clusters K and the degree of covariance parameter sharing S by minimizing the Bayesian information criterion (BIC) The total number of free parameters is defined as qK,S = (K − 1) + 2KL + qΣ(S), where qΣ(S) depends on the specification S The optimal pair is selected by minimizing BIC(K, S) = −2LbK,S + qK,S log p
Conflict of Interest and Data Availability
The authors declare that they have no competing interests The gut and paired leaf–root microbiota datasets were obtained from the publicly available idopNetwork package repository at https://github.com/cxzdsa2332/idopNetwork [5] The processed human gut multi-omics data are available from the source study’s figshare archive at https://doi. p. 9 The source version,
retained features and samples, and preprocessing used in the present
analyses are documented in the Methods and Appendix C. The
processed human gut multi-omics data are available from the source
study’s figshare archive at https://doi.Where the paper says it
Conclusion
Closed-form SAD(1) estimation provides a substantial computational gain for functional clustering by replacing repeated numerical covariance optimization with direct updates, leading to significant speedups in both covariance solving and complete model fitting The recovery experiments further demonstrate the statistical benefit of modeling curve structure, showing that correctly specified means improve module recovery over simpler methods The resulting implementation offers a fast computational foundation for joint functional clustering across conditions
References
-
H. Akaike. A new look at the statistical model identification
-
X. Cao, A. Dong, G. Kang, X. Wang, L. Duan, H. Hou, T. Zhao, S Wu, X Liu, H Huang and R Wu Modeling spatial interaction networks of the gut microbiota
-
C Chen, L Jiang, G Fu M Wang Y Wang B Shen Z Liu Z Wang W Hou S A Berceli and R Wu An omnidirectional visualization model of personalized gene regulatory networks npj Systems Biology and Applications 5:38, 2019
-
A P Dempster, N M Laird, and D B Rubin Maximum likelihood from incomplete data via the EM algorithm Journal of the Royal Statistical Society Series B (Methodological), 39(1):1–22, 1977
-
A Dong, S Wu, J Che Y Wang and R Wu idopNetwork A network tool to dissect spatial community ecology Methods in Ecology and Evolution 14(9):2272–2283, 2023
-
B Efron Bootstrap methods: Another look at the jackknife The Annals of Statistics 7(1):1–26, 1979
-
C Fraley and A E Raftery Model-based clustering, discriminant analysis, and density estimation Journal of the American Statistical Association 97(458):611–631, 2002
-
K R Gabriel Ante-dependence analysis of an ordered set of variables The Annals of Mathematical Statistics 33(1):201–212, 1962
-
L Hubert and P Arabie Comparing partitions Journal of Classification 2(1):193–218, 1985
-
B R Kim, L Zhang, A Berg J Fan and R Wu A computational approach to the functional clustering of periodic gene-expression profiles Genetics 180(2):821–834, 2008
-
X L Meng and D B Rubin Maximum likelihood estimation via the ECM algorithm A general framework Biometrika 80(2):267–278, 1993
-
V N´u˜nez-Ant´on and D L Zimmerman Modeling nonstationary longitudinal data Biometrics 56(3):699–705, 2000
-
W Pan, J Che, and S Wu An improved method of Wu’s functional clustering Statistics Innovation 3:e005, 2026
-
G Schwarz Estimating the dimension of a model The Annals of Statistics 6(2):461–464, 1978
-
M R Wagner D S Lundberg T G del Rio S G Tringe J L Dangl and T Mitchell-Olds Host genotype and age shape the leaf and root microbiomes of a wild perennial plant Nature Communications 7:12151, 2016
-
Y Wang, M Xu, Z Wang, M Tao J Zhu L Wang R Li S A Berceli and R Wu How to cluster gene expression dynamics in response to environmental signals Briefings in Bioinformatics 13(2):162–174, 2012
-
M Zhang Source data and analysis code for the CPYE athletic adaptation study figshare, 2026 URL https://doi. p. 9
org/10.6084/m9.figshare.32043930 Processed source data and analysis codeWhere the paper says it
The source version, retained features and samples, and preprocessing used in the present analyses are documented in the Methods and Appendix C. The processed human gut multi-omics data are available from the source study’s figshare archive at https://doi.
-
M Zhang L Ge K Liu Y Liu G Zhao L Y Kwok Z Sun Z Sun and W Zhang Sex- and time-resolved physiological and gut ecosystem responses to a food-grade microbiome-targeted intervention in collegiate endurance athletes Advanced Science page e77664, 2026
-
W Zhao W Hou R C Littell and R Wu Structured antedependence models for functional mapping of multiple longitudinal traits Statistical Applications in Genetics and Molecular Biology 4(1):Article 33, 2005
Figure Descriptions
Figure 1 Computational cost under the same mixture model Median fitting time versus feature count p and samples per condition n, respectively with 95% dataset-level bootstrap intervals (5,000 resamples) Each setting has 30 independent datasets and three common numerical starts per method The reference has L = 3, p = 600, nl = 40, K = 6 and S4 covariance only the indicated dimension changes Timing includes preparation and all starts excluding common initialization Paired fitting times at the reference setting with 30 points per comparator C denotes closed updates with cached statistics N blockwise numerical covariance optimization J joint numerical covariance optimization R closed updates through direct raw-curve Python kernels The dashed and dotted lines denote y = x and y = 2x Vertical axes in (a,b) and both axes in (c) are logarithmic
Figure 2 Workflow time and mean recovery relative to the upstream FunClu software Complete three-start workflow time including common K-means and model-specific initialization on a logarithmic axis Process startup source loading and input reading are excluded Label-matched mean-curve RMSE over clusters conditions and sample positions Single and Joint tasks use (p, nl, K) = (300, 20, 3) with one and two conditions Larger uses (600, 40, 6) with two conditions Linear and Power denote the generating mean family There are 30 independent datasets per baseline configuration and ten per larger configuration Boxes span the quartiles center lines mark medians whiskers reach the most extreme observations within 1.5 interquartile ranges and circles show all observations beyond the whiskers
Figure 3 Module and mean-curve recovery under linear generating means Top ARI on independently generated test curves Bottom label-matched mean-curve RMSE Columns indicate the generating SAD(1) coefficient Each point summarizes 30 independent training datasets with 2,000 independent test curves per dataset Shaded bands are pointwise 95% bootstrap intervals for the mean based on 5,000 dataset-level resamples All methods use K = 4 and five shared initial partitions GMM denotes unrestricted mean vectors with a shared full covariance matrix FunClu estimated conditionspecific linear mean coefficients and S4 SAD(1) covariance shared across modules The independent-residual variant used the same mean and EM implementation fixed phi = 0 and re-estimated each residual variance from weighted residual squares
Figure 4 Abundance-preserving analysis of the idopNetwork gut microbiota example Dark solid lines show the seven selected component profiles pale hollow circles show all 960 observed feature-by-sample values from the 48 retained features and 20 samples Parentheses give feature counts The BIC panel compares K = 1,..., 20 gray crosses mark fitted candidates with any module containing fewer than two assigned features The red marker identifies K = 7 the BIC minimum among admissible candidates
Figure 5 Joint leaf and root microbiota analysis after within-organ feature standardization and positive shifting Each module is displayed as an adjacent pair of panels with leaves on the left and roots on the right A common module heading and member count appear above each pair Blue and red solid lines show leaf and root component profiles respectively Pale lines show every assigned OTU’s fitted power function Each organ contributes 88 samples The final panel shows the fixed-S4 ∆BIC scan over K = 1,..., 40 selecting K = 11
Figure 6 Niche-associated profiles of the eleven selected modules Dark lines show the back-transformed component profiles at Week 0 Week 3 and Week 6 shown as solid lines in red blue and green respectively Parentheses give feature counts (3,796 in total) Curves span each visit’s observed log niche range with 149 participants per visit Pale lines show every assigned member’s fitted power function sampled at 160 grid points Member coefficients are obtained by ordinary least squares on the modeled response and standardized niche coordinate both member and component curves are then exponentiated for display The vertical axis is therefore the back-transformed response not raw abundance Module panels share axis scales
Figure 8 Covariance solving for the same conditional objective in R Both implementations use fixed identical power-law mean curves and posterior weights Single and joint tasks have (p, nl, K) = (300, 20, 3) with one and two conditions respectively larger tasks have (600, 40, 6) with two conditions Linear and Power identify the generating mean family Each configuration uses ten independent datasets The closed kernel includes weighted residual-statistic construction
Table Summaries
Table 3 Complete-fitting speedups under the common model Each row has 30 independent paired datasets Entries are median per-dataset comparator/C time ratios C denotes closed updates with cached statistics N blockwise numerical covariance optimization J joint numerical covariance optimization and R closed updates through direct raw-curve Python kernels Configuration labels correspond to Fig. 7
Table 4 Workflow costs relative to upstream FunClu Times are medians in seconds including common K-means and model-specific initialization and all three fits Speedups are medians of paired ratios so need not equal ratios of the displayed median times
Figure Details
Figure 7 Distributions of paired computational gains (a) Comparator/closed-and-cached fitting-time ratios across size configurations (b) Ratios under correctly specified S1–S4 constraints at reference dimensions Every box contains 30 independent paired-dataset ratios Boxes span the quartiles center lines mark medians whiskers reach the most extreme observations within 1.5 interquartile ranges and circles show all observations beyond the whiskers The dashed line denotes equal fitting time
Figure 8 Covariance solving for the same conditional objective in R Both implementations use fixed identical power-law mean curves and posterior weights Single and joint tasks have (p, nl, K) = (300, 20, 3) with one and two conditions respectively larger tasks have (600, 40, 6) with two conditions Linear and Power identify the generating mean family Each configuration uses ten independent datasets Paired numerical and closed solving times for all 60 targets with task encoded by color and generating family by symbol Both axes are logarithmic The closed kernel includes weighted residual-statistic construction
Figure 9 Recovery under curved generating means at p = 600 Panels show independent-test ARI (top) and mean-curve RMSE (bottom) at three residual correlations KM GMM FC-iid FC-SAD denote K-means the shared full covariance GMM independent-residual FunClu and SAD(1) FunClu The same separation and average marginal noise variance are used as in the linear design
Figure 10 Estimated mean curves for a fixed example The first formal replicate true module 1 p = 600 and phi = 0.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Fast functional clustering via closed-form covariance updates allows AI systems to perform joint analysis across multiple experimental conditions substantially faster, achieving 91.5–201.4 times greater speedups in same-R comparisons compared to numerical optimization of identical conditional objectives.
-
The system can recover module classification accuracy and lower mean-curve error than K-means and a Gaussian mixture with shared full covariance when the mean structure is correctly specified, utilizing the structure provided by SAD(1) to gain an additional recovery benefit under strong residual correlation.
-
AI systems can estimate residual dependence using first-order structured antedependence models (SAD(1)) with four parameter-sharing specifications (S1–S4), allowing them to constrain within-condition dependence without introducing cross-condition residual covariance.
-
The system can select the optimal number of clusters and the degree of covariance parameter sharing by minimizing the Bayesian information criterion (BIC) across all candidate pairs, balancing model fit against parameter cost.
-
The AI system can generate high-speed, closed-form conditional updates for mean profiles using weighted residual statistics and innovation representation, replacing repeated numerical searches with direct parameter updates within an expectation/conditional maximization framework.
Abstract
Functional clustering identifies coordinated molecular profiles, but repeated numerical covariance optimization can make mixture-model fitting computationally demanding. We develop a fast functional clustering procedure for joint analysis across conditions, combining linear mean profiles on a standardized log niche coordinate with first-order structured antedependence (SAD(1)) covariance. Weighted residual statistics yield closed-form conditional covariance updates under four parameter-sharing specifications, while an innovation representation simplifies likelihood evaluation. In same-R comparisons with numerical optimization of identical conditional objectives, the closed-form solver was 91.5--201.4 times as fast, with objective agreement in all 60 targets. Under matched complete mixture fits, it achieved a 1.87--2.37-fold speedup with identical cluster assignments. Recovery simulations showed higher module classification accuracy and lower mean-curve error than K-means and a Gaussian mixture with shared full covariance when the mean structure was correctly specified. SAD(1) provided an additional recovery gain under strong residual correlation, whereas curved-mean controls identified the importance of an appropriate mean family. Applications to single-condition gut microbiota, paired leaf--root microbiota and three-visit human gut multi-omics illustrate the framework across different numbers of conditions. Closed-form covariance estimation makes joint functional clustering substantially faster while retaining the fitted model's likelihood and assignments.
Related papers
- Sticky Manifold Piecewise Deterministic Monte Carlo for Bayesian Model Averaging
- A perturbation theory for Feynman-Kac models and their particle interpretations
- A fast non-reversible sampler for Bayesian mixture models
- Inventory and Assortment Optimization under General Choice: A Unified Approximation Framework
- Least squares variational inference
- A Computable Measure of Suboptimality for Entropy-Regularised Variational Objectives