Site Is Decodable Before Pretraining: Negative Controls for Probing Frozen Brain-MRI Foundation Models

Frozen brain-MRI foundation-model (FM) embeddings are increasingly used as off-the-shelf representations on the assumption that they capture anatomy. This paper audits what they actually encode and finds that acquisition site is a large, intrinsic component of the representation. Across two independent cohorts (ABIDE-I, ABIDE-II), three frozen 3-D encoders (brain-pretrained, CT-pretrained, and randomly initialized), and every network depth, site is linearly decodable at roughly 0.9 balanced accuracy at deep layers, exceeding the decodability of every clinical or demographic variable (sex, age, autism diagnosis) at every layer. The effect is intrinsic rather than learned: a randomly initialized encoder is already a ∼0.9 site classifier on both cohorts and across three architecture families (Swin, ViT, ResNet), and site is decodable at ∼0.95 directly from the raw downsampled image with no encoder, so the fingerprint reflects low-level image statistics that any encoder preserves rather than a product of pretraining. Residualizing measured population covariates leaves site decodability essentially unchanged, indicating an acquisition- rather than population-driven effect. A nonlinear probe matches the linear one, so the fingerprint is fully linearly accessible. The site subspace is removable post hoc by iterative null-space projection or ComBat (site decodability 0.94 → 0.07/0.00), and is a site-attribution concern for shared or federated embeddings; but for dense segmentation this removal is not free, because site and anatomy occupy an entangled linear subspace (a matched-rank random-direction projection is Dice-neutral, whereas removing the site subspace is destructive). The paper recommends site-audited use of frozen brain-MRI FMs and releases an open audit toolkit.

Specifically, on ABIDE-I, across all three encoders and all five layers, linear site decodability ranges from 0.68 to 0.96, with 90% intervals within ±0.03 to ±0.05, while the best clinical/demographic decodability is at most ∼0.41. The site-dominance ratio is 2–6×. Site decodability rises with depth (L0 ≈ 0.70 → L4 ≈ 0.95), monotonically for the brain-pretrained and random-init encoders and near-monotonically for the CT-pretrained one (which plateaus after L3). A nonlinear MLP probe matches the linear probe (difference < 0.02 at every cell), so the fingerprint is fully linearly accessible. On the independent 15-site ABIDE-II cohort, in a matched 6-way comparison, deep-layer site decodability is 0.79–0.88 (at L4: brain-pretrained 0.87 [0.81, 0.93], CT-pretrained 0.79 [0.69, 0.87], random 0.88 [0.82, 0.93]), and the MLP probe again matches the linear one, reproducing the ∼0.9 fingerprint. Clinical decodability is higher than on ABIDE-I (peaking around 0.5 mid-network) but stays below site, so the site-dominance ratio is smaller yet consistent, ∼1.6–3×.

The fingerprint is acquisition-driven, not population: residualizing the embeddings against the measured population covariates (age, sex, diagnosis) leaves deep-layer site decodability essentially unchanged (95–99.8% retained: brain-pretrained 0.94 → 0.94, CT-pretrained 0.91 → 0.90, random 0.98 → 0.93). The measured population variables explain almost none of it.

The fingerprint is intrinsic, not learned: on ABIDE-I the randomly initialized encoder reaches site decodability 0.956 [0.92, 0.99] at L4, at least as high as the brain-pretrained (0.948) and CT-pretrained (0.926) encoders. The same ordering holds on ABIDE-II (6-way L4: random 0.88, brain-pretrained 0.87, CT-pretrained 0.79). A paired comparison over 200 matched holdouts confirms the random encoder is non-inferior to both pretrained encoders (random−pretrained mean difference +0.007 and +0.018; one-sided paired sign test p < 10−17), and the encoders are statistically equivalent within a margin of 0.05 (two one-sided tests, p < 10−50). The effect also generalizes beyond SwinUNETR: a random-initialized plain ViT and a random-initialized 3-D ResNet reach high deep-layer site decodability (∼0.90 and ∼0.96). The effect is stable across random weight-initialization seeds: over three independent seeds per architecture, deep-layer (L4) site decodability is 0.973 ± 0.006 (Swin), 0.916 ± 0.030 (ViT), and 0.987 ± 0.019 (ResNet). Indeed the fingerprint precedes the network entirely: site is decodable at 0.95 balanced accuracy from the heavily downsampled raw voxels (123, no encoder), whereas clinical decodability from raw voxels is weak (sex 0.33, age R2 0.07).

The depth profile differs between cohorts: on ABIDE-I, age R2 peaks mid-network (L2 ≈ 0.28–0.37) and falls to ∼0 at the L4 bottleneck, and sex/diagnosis are weak throughout, so the layer conventionally frozen for transfer is the least informative for clinical readouts. On ABIDE-II, clinical decodability is higher and also peaks mid-network (L1–L2). Site decodability on ABIDE-I rises with depth and is highest at the bottleneck, whereas on ABIDE-II it saturates early and is flat or mildly declining from L2 onward. What is common to both cohorts is the ordering: site exceeds the best clinical decodability at every layer.

The result is not an artifact of input resolution: repeating the full audit at 1.0 mm and 1603 leaves deep-layer site decodability at 0.93–0.95 (against 0.93–0.96 at 1.7 mm), moves the clinical peak only from 0.41 to 0.43, and preserves the site-dominance ratio at 2.0–5.6× (against 2.2–5.9×), with site exceeding clinical at every layer of every encoder. The nonlinear probe again matches the linear one (MLP minus linear at most 0.03 across all fifteen cells), and the intrinsic result survives: the randomly initialized encoder reaches 0.95 at both L3 and L4.

For leakage and mitigation: frozen embeddings shared across sites expose their acquisition origin at ∼0.9–0.96 balanced accuracy, a site-attribution concern for federated or shared-embedding pipelines. INLP and ComBat both remove it (bottleneck site decodability 0.94 → 0.066 and → 0.000 respectively). For a global readout, which mitigation is preferable is task-dependent: INLP edges ComBat when the target is separable from site (sex, LOSO balanced accuracy 0.626 vs. 0.616; raw 0.580), while covariate-preserving ComBat is better when the target is itself site-confounded (autism diagnosis). Neither mitigation yields a statistically significant cross-site gain on these embeddings, because the pooled representation encodes the clinical targets weakly to begin with.

For dense segmentation, the mitigation is not free: projecting the site subspace out of the frozen features mid-forward is Dice-neutral at the bottleneck (mean ∆ ≈ 2×10−5) but destructive across all scales (mean cross-site Dice 0.84 → 0.18, ∆ = −0.66). A matched-rank control settles the mechanism: projecting out the same number of random orthonormal directions per scale is Dice-neutral (0.84 → 0.83), so the collapse is specific to the site subspace, not a generic consequence of reducing rank. Site and anatomy therefore occupy an entangled linear subspace in the spatial features. Which structures survive the projection is itself informative: large peripheral tissue classes are comparatively spared (cerebral white matter falls only from 0.892 to 0.788, ∆ = −0.103; cerebral cortex from 0.830 to 0.651, ∆ = −0.179), while small, deep, subcortical structures are effectively erased (thalamus falls from 0.884 to 0.040 and putamen from 0.885 to 0.031, ∆ = −0.844 and −0.854, with hippocampus, pallidum, amygdala and accumbens all reduced to Dice below 0.01). The gap between the best- and worst-affected structure is a factor of roughly eight in ∆. A decoder trained on as few as two sites already segments held-out sites at ∼0.87 Dice, so the frozen features are site-robust for dense tasks even though their pooled form is a site fingerprint.

The paper draws four concrete recommendations: (1) Do not treat a bottleneck embedding as anatomy — the layer most commonly frozen and reused for transfer is simultaneously the most site-saturated and the least clinically informative. (2) Audit before you pool — report chance-corrected site decodability alongside the target metric, together with a random-init or raw-input baseline. (3) Scrub before you share — releasing or federating frozen representations discloses acquisition provenance. (4) Match the mitigation to the head — removal is not uniformly safe; a harmonization step that is harmless for a pooled classifier can be destructive for a dense predictor.

The most serious caveat is that “site” bundles scanner, acquisition protocol, and population; because ABIDE sites differ in age and cohort composition, decodable site could in principle encode population rather than acquisition. Residualizing the measured population covariates leaves site decodability essentially unchanged, indicating the bulk is acquisition, but unmeasured population factors (e.g., head motion, unrecorded demographics) remain a caveat. A traveling-subject design would be the definitive test and is not available in ABIDE. The paper accordingly describes the finding as a site fingerprint rather than a scanner one.

Improvements to AI Systems:

1. Add a mandatory site-audit layer to any medical-imaging foundation model (FM) pipeline. Before deploying a frozen brain-MRI encoder for downstream tasks, the system automatically computes chance-corrected site decodability (linear probe) at every network depth, plus a random-init and raw-input baseline. If site decodability exceeds clinical decodability by >2× at the bottleneck, the system flags the embedding as site-saturated and refuses to use it for pooled classification without mitigation.

2. Implement an intrinsic-fingerprint detector that works without pretraining. The system can identify whether an encoder's representations are dominated by acquisition site before any training, by running a linear probe on a randomly initialized version of the same architecture. If the random-init probe already achieves >0.9 balanced accuracy for site, the system knows the fingerprint is a property of the architecture and input statistics, not learned semantics—so it can preemptively apply harmonization.

3. Build a depth-aware feature-selection mechanism. Instead of blindly using the bottleneck layer (L4) for transfer, the system automatically selects the layer that maximizes target clinical decodability while minimizing site decodability. On ABIDE-I, this means choosing mid-network layers (L1–L2) for age/sex/diagnosis tasks, avoiding the site-saturated and clinically-empty bottleneck.

4. Integrate a post-hoc site-scrubbing module with task-adaptive mitigation. The system offers two removal strategies—iterative null-space projection (INLP) and ComBat—and automatically selects between them based on the downstream task: INLP when the target is separable from site (e.g., sex classification), ComBat when the target is site-confounded (e.g., autism diagnosis). It also computes a destructiveness index for dense tasks, warning the user if site removal would collapse segmentation performance (e.g., Dice drop >0.5).

5. Add a raw-input baseline for all clinical readouts. The system always compares FM embeddings against a heavily downsampled raw-voxel baseline (e.g., 123). If the raw baseline achieves comparable or better clinical decodability than the FM (as seen for site at 0.95), the system reports that the FM adds no value for that task and recommends simpler, more interpretable inputs.

6. Create a site-attribution risk score for shared/federated embeddings. Before releasing or federating frozen representations, the system computes a leakage score (site decodability from the embedding). If the score exceeds 0.85, it automatically triggers either (a) scrubbing via INLP/ComBat, or (b) a warning that the embedding discloses acquisition provenance, enabling compliance with privacy regulations.

7. Develop an entangled-subspace guard for dense segmentation. For tasks like tissue segmentation, the system detects whether site and anatomy share a linear subspace (by comparing the effect of site-subspace projection vs. matched-rank random projection). If site removal is destructive (Dice drop >0.5), it automatically switches to a site-robust training strategy—e.g., training a decoder on 2+ sites (which achieves 0.87 Dice on held-out sites) instead of scrubbing the features.

8. Enable cross-cohort generalization checks. The system can validate whether a site fingerprint is consistent across independent cohorts (e.g., ABIDE-I vs. ABIDE-II). If site decodability patterns differ (e.g., saturating early vs. rising with depth), the system adapts its layer-selection and mitigation strategy per cohort, rather than assuming a universal optimal configuration.

What the improved AI system can do:

• Avoid silent failures in medical imaging pipelines by detecting when a foundation model is actually a site-classifier in disguise.

• Choose the right layer and mitigation for each task, improving clinical readout accuracy (e.g., sex classification from 0.58 to 0.63) without sacrificing dense segmentation quality.

• Protect patient privacy in federated learning by automatically scrubbing site-identifiable information from shared embeddings before transmission.

• Save compute and complexity by identifying when raw downsampled inputs suffice, bypassing heavy FMs for tasks where they add no signal.

• Provide explainable audits to clinicians and regulators, showing exactly how much site information leaks at each layer and which mitigation was applied, with quantitative trade-offs (e.g., site decodability reduced from 0.94 to 0.07, but segmentation Dice for thalamus dropped from 0.88 to 0.04).

Frozen foundation-model (FM) embeddings are increasingly used as off-the-shelf brain-MRI representations, on the assumption that they capture anatomy. We audit what they actually encode and find that acquisition site is a large, intrinsic component of the representation. Across two independent cohorts (ABIDE-I, ABIDE-II), three frozen 3-D encoders (brain-pretrained, CT-pretrained, and randomly initialized), and every network depth, site is linearly decodable at roughly 0.9 balanced accuracy at deep layers, exceeding the decodability of every clinical or demographic variable (sex, age, autism diagnosis) at every layer. The effect is intrinsic rather than learned: a randomly initialized encoder is already a 0.9 site classifier on both cohorts and across three architecture families (Swin, ViT, ResNet), and site is decodable at 0.95 directly from the raw downsampled image with no encoder, so the fingerprint reflects low-level image statistics that any encoder preserves rather than a product of pretraining. Residualizing measured population covariates leaves site decodability essentially unchanged, indicating an acquisition- rather than population-driven effect. A nonlinear probe matches the linear one, so the fingerprint is fully linearly accessible. The site subspace is removable post hoc by iterative null-space projection or ComBat (site decodability 0.94 -> 0.07/0.00), and is a site-attribution concern for shared or federated embeddings; but for dense segmentation this removal is not free, because site and anatomy occupy an entangled linear subspace (a matched-rank random-direction projection is Dice-neutral, whereas removing the site subspace is destructive). We recommend site-audited use of frozen brain-MRI FMs and release an open audit toolkit.