When Participant Disjoint Validation Is Not Enough to Establish Generalization in Parkinson's Gait Classification

arXiv:2610.16220 · cs.LG · Submitted 2026-10-12 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: Today's paper: "When Participant Disjoint Validation Is Not Enough to Establish Generalization in Parkinson's Gait Classification".

Jane: The gist: Participant-disjoint pooled validation did not isolate disease-related discrimination from site-associated variation in Parkinson’s gait classification,

Tom: First, who's behind it and why it matters.

Title and authors: Jane: Now that we’ve heard what the "When Participant Disjoint Validation Is Not Enough to Establish Generalization in Parkinson's Gait Classification" paper found, let’s break down what it actually means for us. The main point is that participant separation doesn't automatically guarantee you see true disease signals when site structure is involved.

Tom: Exactly. They looked at this WearGait-PD dataset with one hundred thirty-three participants, and they had a tricky setup: two sites only had one group, but VA-Seattle was the only one with both Parkinson's patients and controls <ref:2610.16220#pg1,VA-Seattle was the only>.

Lu: That structural dependence between site location and diagnosis was very strong, which is what the paper points to when it says site and diagnosis were strongly associated with a chi squared of ninety-eight point two four.

Jane: So, the naive question for us is: if we just look at the pooled performance number from that study—that AUC of zero point nine five five—are we safe assuming that signal is purely about Parkinson's disease <ref:2610.16220#pg1>?

Tom: The answer they’re giving you is no, because they showed that site alone achieved an AUC of zero point nine five five with a standard deviation of zero point zero zero three when using repeated participant-level nested cross-validation, which actually beats the gait-feature model’s AUC of zero point eight eight two with a standard deviation of zero point zero one four, according to page one of "When Participant Disjoint Validation Is Not Enough to Establish Generalization in Parkinson's Gait Classification" <ref:2610.16220#pg1,When Participant Disjoint Validation Is Not Enough to Establish Generalization in Parkinson>.

Lu: But then they immediately pivot and show that when they added gait features on top of the site and age and sex variables, there was no measurable improvement in pooled discrimination at all. That’s a key observation about feature importance.

Jane: So what’s the implication here for anyone building a predictive model? It suggests that relying only on participant separation doesn't isolate the disease discrimination from where you got the data.

Tom: Precisely, and they show that when they did site-controlled evaluations, like testing VA-Seattle same-site, the AUC dropped to zero point six six eight with a wide confidence interval of zero point four seven seven to zero point eight three six on page two of the paper.

Lu: That drop is telling because it shows that the site as a surrogate for diagnosis wasn't sufficient to isolate disease discrimination in those specific tests, which means you can’t just use site performance as proof of generalization yet.

Jane: So, if you’re a researcher trying to claim that your model works across different clinics, this paper warns you that the site structure might be causing the apparent performance boost.

Tom: It highlights why we need to look deeper into how features relate to acquisition context before we trust pooled results in non-factorial datasets.

Lu: Because they did a structural analysis showing that global label permutation removed the site–diagnosis association and centered performance near chance, but site-conditioned permutation kept a high reference AUC distribution because ninety-eight labels from the single-class sites were structurally fixed on page three of "When Participant Disjoint Validation Is Not Enough to Establish Generalization in Parkinson's Gait Classification".

Jane: That structural sensitivity analysis is really important because it illustrates why participant separation alone may be insufficient in non-factorial multisite datasets.

Tom: So, the takeaway here is that we need to treat acquisition site structure as a variable that needs explicit modeling rather than just ignoring it when we look at pooled numbers.

Lu: We need to be more careful about how we select features, because they might just be proxies for the site itself in some of these analyses.

Jane: And this paper is pushing us toward a validation strategy that includes within-site estimates wherever all target classes are represented and rigorous source-to-target transport tests.

Tom: It’s about moving beyond just looking at one big pooled number and understanding the underlying mechanics of how that number was built.

Lu: Because they’re also stressing the need to integrate non-clinical baselines, like site only or age and sex, because adding gait features to those didn't improve mean discrimination.

Jane: So, if you’re interested in this work, keep an eye on how these structural analyses influence future model design rather than just accepting the pooled AUC as gospel.

The paper's summary: Tom: We’ve established that participant-disjoint pooled validation doesn't isolate disease discrimination from site structure, and now we’re going to summarize the core findings of "When Participant Disjoint Validation Is Not Enough to Establish Generalization in Parkinson's Gait Classification."

Jane: In simple terms, the paper is saying that just keeping participants separate doesn't guarantee you are learning a truly generalizable rule for diagnosing Parkinson's disease. The numbers they reported are misleading because they’re being influenced by where the data came from.

Lu: They used one hundred fifty-nine features from one hundred thirty-three participants, with sixty-five PD and sixty-eight controls, and the key takeaway is that site alone achieved an AUC of zero point nine five five ± zero point zero zero three in their repeated participant-level nested cross-validation on page one of the paper <ref:2610.16220#pg1,AUC of 0.955 ± 0.003>.

Tom: That high number is what makes people think it’s a solid result, but they immediately show that when they tried to add gait features to site, age, and sex variables together, there was no measurable improvement in pooled discrimination under the logistic regression pipelines.

Jane: So what does that tell us about the relationship between site and diagnosis versus the actual gait information? It suggests that if you add those features on top of site, age, and sex—the discrimination doesn't get any better.

Lu: That really implies that adding gait features didn't introduce new disease-related information beyond what was already captured by the acquisition site structure.

Tom: So the paper is essentially saying that the gait representation they used wasn't capturing anything truly novel for classification when site context was already accounted for.

Jane: And then they show that site-controlled evaluations, like testing VA-Seattle same-site, gave them lower and less precise estimates compared to what they might expect.

Lu: That lower performance on the same-site tests confirms that the site as a surrogate for diagnosis wasn't sufficient to isolate disease discrimination in those specific contexts.

Jane: So the overall summary is that you can’t just trust pooled results without also looking at how site structure plays a role in the classification.

Tom: It’s about being skeptical of any result that seems too good to be true, especially when it comes to generalization across institutions.

Lu: And they set up the problem by showing that participant separation alone is insufficient in non-factorial multisite datasets because of this inherent site–diagnosis structure on page two of "When Participant Disjoint Validation Is Not Enough to Establish Generalization in Parkinson's Gait Classification".

Jane: So, if you’re just listening, the main message is that you need to be very careful about how those pooled AUC numbers are being derived because they can be influenced by site-diagnosis structure.

Tom: It’s a warning not to confuse correlation with causation when it comes to machine learning performance metrics across different institutions.

Lu: And they suggest that for multisite studies, you need to examine site–label structure and include non-clinical baselines in your evaluation framework.

Jane: It’s a call for more transparent testing before making broad claims about clinical generalization, and that's what this paper is really advocating for.

The paper's improvements: Tom: Now let’s look at the specific improvements the authors suggest to fix these issues, because they aren't just pointing out problems; they are offering a path forward.

Lu: They suggest that we need to move toward site-aware feature engineering, explicitly accounting for site-specific acquisition context instead of just relying on pooled performance alone.

Jane: That sounds like a necessary change in how we build the system: instead of just feeding the raw features, you have to feed it context about where those features came from.

Tom: They argue that this allows the system to better generalize across institutions because it helps distinguish between true disease signals and site shortcuts, preventing optimistic estimates from participant-disjoint cross-validation.

Lu: And they suggest implementing multi-stage validation pipelines instead of just one pooled AUC metric, following the paper's recommendation to reserve claims of clinical generalization for genuinely independent external validation.

Jane: So that means testing within-site estimates wherever all target classes are represented and rigorously assessing source-to-target transport test performance.

Tom: That’s a lot of work, but it means you can’t just rely on one metric; you need to check multiple metrics before making any clinical claims.

Lu: They also suggest using structural sensitivity analysis for feature selection, like site conditioned permutation analyses to see if the gait representation actually encodes acquisition context.

Jane: That’s a smart way to ensure that the features you pick aren't just proxies for the site itself, which is something they demonstrated in their analysis.

Tom: They also suggest integrating non-clinical baselines, like "site only" and "age and sex" because those baselines showed that adding gait features didn't improve mean discrimination on page three of the paper.

Lu: So, if you’re building an AI system from scratch, you need to start by establishing what baseline performance looks like before adding complex features.

Jane: It’s about building a more transparent validation framework that includes these different checks for structural dependence on where the data was collected.

Tom: And finally, they stress uncertainty quantification using participant-level bootstrap intervals because the small cohort didn't support a precise same-site performance claim on page two of the paper.

Lu: So, reporting uncertainty is crucial because it shows that we can’t just say something is perfectly certain when the data doesn't support it.

Jane: It sounds like they want us to adopt a validation strategy that includes these different checks for structural dependence on where the data was collected before making broad claims.

Conclusion: Tom: So, to wrap up on "When Participant Disjoint Validation Is Not Enough to Establish Generalization in Parkinson's Gait Classification," the principal finding is that participant-disjoint pooled validation did not isolate disease-related generalization from acquisition-site structure.

Jane: In short, it means site alone reproduced and exceeded the apparent gait-classification signal, even though they tried to test it with other methods.

Lu: The overall implication is that you need careful examination of site–label dependence before making broad claims about generalization across institutions.

Meng: For deployment, we need to be cautious and ensure we examine site–label structure rather than just blindly trusting the pooled AUC number.

Tom: We have to make sure we test within-site estimates where all target classes are represented and use structural sensitivity analysis for feature selection as they suggested.

Lu: They also stress that integrating non-clinical baselines, like "site only" or age and sex, because adding gait features didn't improve mean discrimination on page three of the paper.

Jane: The final message is that the available non-factorial sampling design cannot separate disease-related gait information from site–diagnosis structure strongly enough to support the clinical interpretation implied by pooled AUC.

Tom: So, we need to be really rigorous in how we build our models and reserve claims of clinical generalization for genuinely independent external validation.

Lu: That’s the path forward for building more robust AI systems that account for where the data was collected and what it means when you see those results on page two of "When Participant Disjoint Validation Is Not Enough to Establish Generalization in Parkinson's Gait Classification".

Jane: We learned a lot about how site structure can play a role in classification, and that’s something every listener should know.

Md. Sifata, Sania Aktera, Akif Islama, Md. Ekramul Hamida

Department of Computer Science and Engineering, University of Rajshahi

cs.LG

Submitted: 2026-10-12

Updated: 2026-10-12

Comments: 9 pages, 5 figures. Includes supplementary analyses integrated into the appendices

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 74/100

The gist: The gist: Participant-disjoint pooled validation did not isolate disease-related discrimination from site-associated variation in Parkinson’s gait classification, indicating that site alone could

Key concepts

Participant-disjoint pooled validation: This method involved testing a classification model using data from participants who were not grouped by their study site. The goal was to see if the model's performance generalized across different physical locations, such as hospitals or clinics.
Site-associated variation: This refers to differences in gait classification results that are caused by where the data was collected, rather than the actual disease status of the participant. The study investigated whether these location differences could be distinguished from true disease discrimination.
Non-factorial cohort structure: The study's participants were not organized into distinct groups based on multiple factors simultaneously (like site and diagnosis). Instead, some sites only had one group, making it difficult to isolate the effects of the site versus the disease itself.
Structural sensitivity analysis: This analysis examined how much of the data structure—specifically, which labels were tied to which sites—remained fixed when testing different assumptions. The results showed that because many labels were fixed by their site, separating disease signal from site structure was challenging.

Terminology

Generalization — 1× in this paper · explained in 13 episodes

Summary

The gist: Participant-disjoint pooled validation did not isolate disease-related discrimination from site-associated variation in Parkinson’s gait classification, indicating that site alone could reproduce pooled performance

Study Design and Cohort Structure

The study examined Parkinson’s disease classification using 159 pressure-walkway summary features from 133 WearGait-PD participants, including 65 participants with Parkinson’s disease and 68 controls. The cohort was non-factorial: two sites contained only one diagnostic group, whereas VA-Seattle was the only site containing both groups. The official identifier mapping assigned NLS participants to JHOC, HC participants to Bayview, and WPD/WHC participants to VA-Seattle. Site and diagnosis were strongly associated (χ 2 = 98.24, d f = 2, p = 4.65 × 10−22; Cramer’s V = 0.859).

Model Evaluation and Performance Metrics

The primary metric used was the Receiver operating characteristic AUC because no clinical operating threshold or intended-use prevalence was defined. The evaluation framework included five pooled diagnosis models: site only; age and sex; site, age, and sex; all 159 gait features; and site, age, sex, and gait features. In repeated participant-level nested cross-validation, site alone achieved an area under the receiver operating characteristic curve (AUC) of 0.955 ± 0.003. When adding gait features to site, age, and sex, no measurable improvement in pooled discrimination was produced.

Site-Specific and Transport Analysis

Site-controlled evaluations produced lower and imprecise estimates; for instance, VA-Seattle same-site AUC was 0.668 (95% bootstrap confidence interval 0.477–0.836). The JHOC + Bayview-to-VA-Seattle transport stress test achieved 0.576 (0.375–0.770). These results suggest that the site as a surrogate for diagnosis in pooled analyses was not sufficient to isolate disease discrimination.

Structural Analysis and Permutations

Site-conditioned permutation analyses showed that the gait representation retained substantial acquisition-site structure. Global label permutation removed the site–diagnosis association and centred performance near chance, whereas site-conditioned permutation retained a high reference AUC distribution because 98 labels from the single-class sites remained structurally fixed. This structural sensitivity analysis illustrated why participant separation alone may be insufficient in non-factorial multisite datasets.

Conclusion and Implications

The principal finding is that participant-disjoint pooled validation did not isolate disease-related generalization from acquisition-site structure, as site alone reproduced and exceeded the apparent gait-classification signal. The results encourage careful examination of site–label dependence and support for pooled results through within-site or independent validation before broader claims are made about generalization across institutions. Multisite prediction studies should examine site–label structure, include non-clinical baselines, and reserve claims of clinical generalization for genuinely independent external validation. The available non-factorial sampling design cannot separate disease-related gait information from site–diagnosis structure strongly enough to support the clinical interpretation implied by pooled AUC. The WearGait-PD investigators used common equipment models, software, protocols, staff training, and quality-control procedures across sites.

Ethics statement

The original WearGait-PD study obtained written informed consent, including consent for release of de-identified data. Data and code availability

WearGait-PD is available through Synapse accession syn52540892 and DOI 10.7303/syn52540892, subject to repository access terms Funding

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors. Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper CRediT authorship contribution statement

Md. Sifat: Conceptualization, Methodology, Software, Formal analysis, Investigation, Writing – original draft, Writing – review & editing. Sania Akter: Data curation, Investigation, Validation, Writing – review & editing. Akif Islam: Validation, Resources, Writing – review & editing. Md. Ekramul Hamid: Supervision, Project administration Declaration of generative AI and AI-assisted technologies in the manuscript preparation process

During the preparation of this work, the authors used OpenAI ChatGPT for language editing, document organization, and assistance in structuring reproducible analysis materials Acknowledgements

The authors thank the WearGait-PD investigators for making the de-identified dataset and its documentation available to the research community References

[1] W. Poewe, K. Seppi, C. M. Tanner, G. M. Halliday, P. Brundin, J. Volkmann, A.-E. Schrag, A. E. Lang, Parkinson disease [2] A. P. J., E., R., E., I., L., L., Gait parameters of parkinson’s disease compared with healthy controls: A systematic review and meta-analysis [3] L. di Biase, A. Di Santo, M. L. Caminiti, A. De Liso, S. A. Shah, L. Ricci, V., Di Lazzaro, Gait analysis in parkinson’s disease: An overview of the most accurate markers for diagnosis and symptoms monitoring [4] O., B., K., Concurrent validity evidence for pressure-sensing walkways measuring spatiotemporal features of gait: A systematic review and meta-analysis [5] R. C. Lynall, L. A., P., J. P., Reliability and validity of the protokinetics movement analysis software in measuring center of pressure during walking [6] M. Parati, E., B., De Maria, et al., Reliability of spatiotemporal gait parameters during dual-task walking on a pressure-sensitive walkway [7] R. Z. U., S., D., Y. Guan, A. J., J. Q. Shi, L. Rochester, Selecting clinically relevant gait characteristics for classification of early parkinson’s disease: A comprehensive machine learning approach [8] T. Nanayakkara, H., M. K. K., M. B., Herath, N., Madusanka, M., Yi, B.-i. Lee, Multi-domain cop feature analysis of functional mobility for parkinson’s disease detection using wearable pressure insoles [9] J. R. Zech, M. A., Badgeley, M., Liu, A. B., Costa, J. J., Titano, E. K., Oermann, Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: A cross-sectional study [10] J. Dockès, G. Varoquaux, J.-B., Poline, Preventing dataset shift from breaking machine-learning biomarkers [11] R. Geirhos, J.-H., C., M., R. Zemel, W. Brendel, M. Bethge, F. A. Wichmann, Shortcut learning in deep neural networks [12] G. C., N. L., Talbot, On over-fitting in model selection and subsequent selection bias in performance evaluation [13] S. Varma, R., Bias in error estimation when using cross-validation for model selection [14] S. Kapoor, A. Narayanan, Leakage and the reproducibility crisis in machine-learning-based science [15] M. B. A., S., N., R. Ranganath, L., Foschini, M. Ghassemi, Reproducibility in machine learning for health research: Still a ways to go [16] G. S., K. G. M., Moons, P. Dhiman, et al., TRIPOD+AI statement: Updated guidance for reporting clinical prediction models that use regression or machine learning methods [17] K. G. M., J. A. A., T., Kaul, et al., PROBAST+AI: An updated quality, risk of bias, and applicability assessment tool for prediction models using regression or artificial intelligence methods [18] K. Lekadir, A. F., A. R. Porras, et al., FUTURE-AI: International consensus guideline for trustworthy and deployable artificial intelligence in healthcare [19] F. Cabitza, G., Jurman, F., Molinari, R., Bellazzi, Why almost all ML models for medicine are wrong—and what we need for evidence-based medical AI [20] S. Shukla, P. Naliyatthaliyazchayil, J. W.

Improvements for AI systems

textbfIncrease Model Robustness via Site-Aware Feature Engineering: The improved AI system can better generalize across institutions by explicitly accounting for site-specific acquisition context rather than relying on pooled performance alone, as Site alone reproduced and exceeded the pooled gait-classification performance. This allows the system to distinguish between true disease signals and site shortcuts, preventing optimistic estimates from participant-disjoint cross-validation.
textbfImplement Multi-Stage Validation Pipelines: The improved AI system must incorporate multiple validation metrics rather than relying on a single pooled AUC, following the paper's recommendation to reserve claims of clinical generalization for genuinely independent external validation. This involves testing within-site estimates wherever all target classes are represented and rigorously assessing source-to-target transport test performance.
textbfEmploy Structural Sensitivity Analysis for Feature Selection: The improved system should use site conditioned permutation analyses to determine if a gait representation encodes acquisition context, as the paper demonstrated that Gait features predicted acquisition site within both diagnostic strata. This ensures that selected gait features are not merely proxies for the site itself.
textbf Integrate Non-Clinical Baselines: The AI system must include site only and age and sex baselines in its evaluation framework to establish a baseline, as the paper notes that these baselines were used to show that Adding gait to site and demographics did not improve mean discrimination. This prevents over-fitting to site structure.
textbf Contextualize Results with Uncertainty Quantification: The system must report uncertainty using participant-level bootstrap intervals rather than just pooled means, as the paper showed that The small cohort therefore did not support a precise same-site performance claim, emphasizing that uncertainty reporting is crucial for medical machine learning.

Sources

Batch Effects In Brain Foundation Model Embeddings