Analysis choices decide whether untrained networks beat trained ones at V1: a multiverse analysis of model-brain RSA
summary
The gist
Untrained convolutional networks have been reported to rival trained networks as models of visual cortex and we had reported an untrained-network advantage over backpropagation at V1 in THINGS fMRI…
In short
The study investigated if untrained convolutional networks can outperform trained ones as models of visual cortex by testing different analysis choices. The core finding is that this advantage is not universal; it depends entirely on the specific metrics used, such as distance measures and normalization techniques. The advantage disappeared under certain conditions, suggesting the result is highly sensitive to how the data are analyzed.
Key concepts
- Untrained Convolutional Networks
- These are neural networks with no prior training data applied to them. The researchers tested if these networks could mimic the function of a trained visual cortex model, showing whether an untrained structure could perform better than one that has been learned from real visual data.
- Analysis Multiverse
- This refers to testing many different combinations of analytical choices—like how units are normalized, what distance measure is used, and how voxels are normalized. The goal was to see if the observed advantage of the untrained network holds true regardless of which specific analysis method is chosen.
- V1
- V1 (Visual Area 1) is a specific region in the visual cortex that processes basic visual information. The researchers tested whether their model-brain comparison held up specifically at this location, examining how the untrained network performed against trained models there.
Terminology used across episodes
This episode discusses
- Analysis choices decide whether untrained networks beat trained ones at V1: a multiverse analysis of model-brain RSA · Paper Radio
- Untrained CNNs Exceed Backpropagation in V1 Alignment at High Evaluation Resolution: A Systematic RSA Comparison of Four Learning Rules Against Human fMRI · Paper Radio
The paper
Analysis choices decide whether untrained networks beat trained ones at V1: a multiverse analysis of model-brain RSA · Read on arXiv
Nils Leutenegger
Untrained convolutional networks have been reported to rival trained networks as models of visual cortex, and we had reported an untrained-network advantage over backpropagation at V1 in THINGS fMRI, together with a dependence on evaluation resolution. Under a pre-specified ridge readout, the untrained network scored below the trained one at every pooling grid (median r difference-0.043 at g = 4, 0 of 5 seeds positive), and also in MSE skill and per-fold r. In exploratory analyses, across 36 combinations of unit normalisation, distance measure and voxel normalisation (28 distinct analyses), the sign of the V1 contrast depended on the analysis (intervals above 0 in 5 and below 0 in 9 distinct analyses), and in our networks the resolution dependence of the original convention appeared in only one cell. Raw correlation-distance representational similarity analysis (RSA) covaried with the alignment of model and voxel patterns with the unit-mean profile (Spearman 0.97 across our networks, 0.79 across six ImageNet architectures); the advantage was absent from the component orthogonal to the mean profile, and the parallel component alone did not resolve it. On the unweighted angular component of centred patterns, trained networks were ahead at V1 (our network and 5 of 6 ImageNet models) and at LOC (all seven); the norm-weighted angular term was not resolved. A luminance-and-contrast bank scored above every network under ridge (point estimates), and adding network features under a shared penalty did not improve prediction. Intervals are stimulus-bootstrap intervals for three subjects, and 6 pre-stated predictions were not met. We recommend reporting sensitivity to centring over the evaluated stimuli on both sides, reporting components separately, and including a small multiverse and simple baselines.
Transcript
Introduction to the show: ident: Genomics Radio. Generated commentary on the latest computational biology and genomics papers.
Ines: We're looking at the paper: "Analysis choices decide whether untrained networks beat trained ones at V1".
Marcus: Here's the gist of the paper. Untrained convolutional networks have been reported to rival trained networks as models of visual cortex and we had reported an untrained-network advantage over backpropagation at V1 in THINGS fMRI, together with a dependence on evaluation resolution Under a pre-specified ridge readout, the untrained network scored below the trained one at every pooling grid (median r difference −0.043 at g = 4, 0 of 5 seeds positive), and also in MSE skill and per-fold r Pruning redundant reasoning tokens The paper investigates whether untrained convolutional networks can outperform trained ones as models of visual cortex by examining the dependence of this advantage on various analysis choices, including distance measures, unit normalisation, and voxel normalisation.
Ines: The core finding is that the advantage of the untrained network at V1 depends entirely on the specific analysis employed How the training signal is built The authors tested four main ways to examine their earlier result, including a pre-specified analysis using a fitted ridge readout on the full feature map.
Marcus: They also explored an "analysis multiverse" involving 36 combinations of unit normalisation, distance measure, and voxel normalisation to test which conclusions hold across defensible choices.
Ines: Where the gains come from The advantage of the untrained network at V1 was found not to be robust across all analyses; it disappeared under correlation distance when model units were standardised, and it was absent from the component orthogonal to the mean profile of the units.
Paper summary: Ines: So this paper, "Analysis choices decide whether untrained networks beat trained ones at V1: a multiverse analysis of model-brain RSA," is really about how fragile those initial findings were that showed untrained networks could beat trained ones in the visual cortex models.
Marcus: The authors are showing that they can't just rely on one specific way to compare these neural network models against the training signal. They tested thirty-six different ways to look at the data, changing things like how you normalize units or what distance measure you use.
Yuki: What this means for us is that when we try to build these complex computational models of the visual cortex, we can't just pick one statistical tool and assume it tells us the whole story about performance.
Ines: They found that the advantage of the untrained network at V1 disappeared under correlation distance when they standardized those model units, and it vanished from components orthogonal to the mean profile of those units.
Marcus: Even when raw correlation distance representation similarity analysis closely tracked mean-profile alignment, that advantage still disappeared from those mean profile components, and parallel components didn't resolve it either.
Yuki: So essentially, if you don't standardize your approach and check all these different angles of comparison, you can't reliably claim one model beats another in this context.
Ines: They are suggesting we need to report sensitivity to centring over the stimuli on both sides, and report components separately with simple baselines so people can actually compare these findings themselves.
Marcus: The limitation they state is that the data comes from three subjects in a single dataset, and those intervals described different sources of variation, which means we need to be careful about generalizing those results broadly.
Yuki: So while this work shows how analysis choices affect model comparison at V1, the authors are being very clear that we still have limitations regarding sample size and the complexity of biological variation.
Ines: It really highlights just how much our conclusions in these fields depend on our specific analytical framework rather than just the underlying biology.
Conclusion: Ines: So, we're looking at this paper now called "Analysis choices decide whether untrained networks beat trained ones at V1: a multiverse analysis of model-brain RSA." It really hammers home how fragile those initial findings were that suggested untrained networks could outperform trained ones in models of the visual cortex.
Marcus: The authors are showing that you can’t just pick one specific way to compare these neural network models against the training signal. They tested thirty-six different ways to look at the data, changing things like how they normalize the units or what distance measure they use.
Yuki: What this means for us is that when we build these complex computational models of systems like the visual cortex, we can’t just pick one statistical tool and assume it tells us the whole story about performance.
Ines: They found that the advantage of the untrained network at V1 disappeared under correlation distance when they standardized those model units. It also vanished from components orthogonal to the mean profile of those units.
Marcus: Even when raw correlation distance representation similarity analysis closely tracked mean-profile alignment, that advantage still disappeared from those mean profile components, and parallel components didn't resolve it either.
Yuki: So essentially, if you don't standardize your approach and check all these different angles of comparison, you can’t reliably claim one model beats another in this context.
Ines: They are suggesting we need to report sensitivity to centring over the stimuli on both sides, and report components separately with simple baselines so people can actually compare these findings themselves in a meaningful way.
Marcus: The limitation they state is that the data comes from three subjects in a single dataset, and those intervals described different sources of variation, which means we need to be careful about generalizing those results broadly.
Yuki: So while this work shows how analysis choices affect model comparison at V1, the authors are being very clear that we still have limitations regarding sample size and the complexity of biological variation.
Ines: It really highlights just how much our conclusions in these fields depend on our specific analytical framework rather than just the underlying biology.
More episodes
- 2610.33881-Sensorimotor behavior from an untrained brain-and-nerve-cord connectome
- 2610.40344-Morphological Features of Murine Pulmonary Arterial Networks for Diagnosing Vascular Disease
- Daily Summary for 2026-11-19
- 2609.10193-Are You Learning Biological Signal or Shortcuts? Auditing and Mitigating Bias in Protein-Protein Interaction Datasets
- 2610.38236-Generative discovery of chemical reaction networks
- 2507.11176-An Interpretable AI framework Quantifying Traditional Chinese Medicine Principles Towards Enhancing and Integrating with Modern Biomedicine
- The day in conversation for 2026-11-19
- Daily Summary for 2026-11-19
- 2504.03590-Towards a Unified Framework for Determining Conformational Ensembles of Disordered Proteins
- 2603.15217-A multiscale discrete-to-continuum framework for structured population models