Analysis choices decide whether untrained networks beat trained ones at V1: a multiverse analysis of model-brain RSA
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Genomics Radio. Generated commentary on the latest computational biology and genomics papers.
Ines: We're looking at the paper: "Analysis choices decide whether untrained networks beat trained ones at V1".
Marcus: Here's the gist of the paper. Untrained convolutional networks have been reported to rival trained networks as models of visual cortex and we had reported an untrained-network advantage over backpropagation at V1 in THINGS fMRI, together with a dependence on evaluation resolution Under a pre-specified ridge readout, the untrained network scored below the trained one at every pooling grid (median r difference −0.043 at g = 4, 0 of 5 seeds positive), and also in MSE skill and per-fold r Pruning redundant reasoning tokens The paper investigates whether untrained convolutional networks can outperform trained ones as models of visual cortex by examining the dependence of this advantage on various analysis choices, including distance measures, unit normalisation, and voxel normalisation.
Ines: The core finding is that the advantage of the untrained network at V1 depends entirely on the specific analysis employed How the training signal is built The authors tested four main ways to examine their earlier result, including a pre-specified analysis using a fitted ridge readout on the full feature map.
Marcus: They also explored an "analysis multiverse" involving 36 combinations of unit normalisation, distance measure, and voxel normalisation to test which conclusions hold across defensible choices.
Ines: Where the gains come from The advantage of the untrained network at V1 was found not to be robust across all analyses; it disappeared under correlation distance when model units were standardised, and it was absent from the component orthogonal to the mean profile of the units.
Paper summary: Ines: So this paper, "Analysis choices decide whether untrained networks beat trained ones at V1: a multiverse analysis of model-brain RSA," is really about how fragile those initial findings were that showed untrained networks could beat trained ones in the visual cortex models.
Marcus: The authors are showing that they can't just rely on one specific way to compare these neural network models against the training signal. They tested thirty-six different ways to look at the data, changing things like how you normalize units or what distance measure you use.
Yuki: What this means for us is that when we try to build these complex computational models of the visual cortex, we can't just pick one statistical tool and assume it tells us the whole story about performance.
Ines: They found that the advantage of the untrained network at V1 disappeared under correlation distance when they standardized those model units, and it vanished from components orthogonal to the mean profile of those units.
Marcus: Even when raw correlation distance representation similarity analysis closely tracked mean-profile alignment, that advantage still disappeared from those mean profile components, and parallel components didn't resolve it either.
Yuki: So essentially, if you don't standardize your approach and check all these different angles of comparison, you can't reliably claim one model beats another in this context.
Ines: They are suggesting we need to report sensitivity to centring over the stimuli on both sides, and report components separately with simple baselines so people can actually compare these findings themselves.
Marcus: The limitation they state is that the data comes from three subjects in a single dataset, and those intervals described different sources of variation, which means we need to be careful about generalizing those results broadly.
Yuki: So while this work shows how analysis choices affect model comparison at V1, the authors are being very clear that we still have limitations regarding sample size and the complexity of biological variation.
Ines: It really highlights just how much our conclusions in these fields depend on our specific analytical framework rather than just the underlying biology.
Conclusion: Ines: So, we're looking at this paper now called "Analysis choices decide whether untrained networks beat trained ones at V1: a multiverse analysis of model-brain RSA." It really hammers home how fragile those initial findings were that suggested untrained networks could outperform trained ones in models of the visual cortex.
Marcus: The authors are showing that you can’t just pick one specific way to compare these neural network models against the training signal. They tested thirty-six different ways to look at the data, changing things like how they normalize the units or what distance measure they use.
Yuki: What this means for us is that when we build these complex computational models of systems like the visual cortex, we can’t just pick one statistical tool and assume it tells us the whole story about performance.
Ines: They found that the advantage of the untrained network at V1 disappeared under correlation distance when they standardized those model units. It also vanished from components orthogonal to the mean profile of those units.
Marcus: Even when raw correlation distance representation similarity analysis closely tracked mean-profile alignment, that advantage still disappeared from those mean profile components, and parallel components didn't resolve it either.
Yuki: So essentially, if you don't standardize your approach and check all these different angles of comparison, you can’t reliably claim one model beats another in this context.
Ines: They are suggesting we need to report sensitivity to centring over the stimuli on both sides, and report components separately with simple baselines so people can actually compare these findings themselves in a meaningful way.
Marcus: The limitation they state is that the data comes from three subjects in a single dataset, and those intervals described different sources of variation, which means we need to be careful about generalizing those results broadly.
Yuki: So while this work shows how analysis choices affect model comparison at V1, the authors are being very clear that we still have limitations regarding sample size and the complexity of biological variation.
Ines: It really highlights just how much our conclusions in these fields depend on our specific analytical framework rather than just the underlying biology.
Nils Leutenegger
q-bio.NC, cs.LG
Submitted: 2026-11-11
Updated: 2026-11-11
Comments: 31 pages, 6 figures; code and data pipeline: https://github.com/nilsleut/ridge-readout
Code: https://github.com/nilsleut/ridge-readout
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 81/100
The gist: Untrained convolutional networks have been reported to rival trained networks as models of visual cortex and we had reported an untrained-network advantage over backpropagation at V1 in THINGS fMRI…
Key concepts
- Untrained Convolutional Networks
- These are neural networks with no prior training data applied to them. The researchers tested if these networks could mimic the function of a trained visual cortex model, showing whether an untrained structure could perform better than one that has been learned from real visual data.
- Analysis Multiverse
- This refers to testing many different combinations of analytical choices—like how units are normalized, what distance measure is used, and how voxels are normalized. The goal was to see if the observed advantage of the untrained network holds true regardless of which specific analysis method is chosen.
- V1
- V1 (Visual Area 1) is a specific region in the visual cortex that processes basic visual information. The researchers tested whether their model-brain comparison held up specifically at this location, examining how the untrained network performed against trained models there.
Terminology
Summary
Untrained convolutional networks have been reported to rival trained networks as models of visual cortex and we had reported an untrained-network advantage over backpropagation at V1 in THINGS fMRI, together with a dependence on evaluation resolution Under a pre-specified ridge readout, the untrained network scored below the trained one at every pooling grid (median r difference −0. p. 0 , 2025], and we had reported an untrained-network advantage over
backpropagation at V1 in THINGS fMRI, together with a dependence on
evaluation resolution [Leutenegger, 2026a]. Under a pre-specified ridge
readout, the untrained network scored below the trained one at every
pooling grid (median r difference −0.Where the paper says it
Pruning redundant reasoning tokens
The paper investigates whether untrained convolutional networks can outperform trained ones as models of visual cortex by examining the dependence of this advantage on various analysis choices, including distance measures, unit normalisation, and voxel normalisation. The core finding is that the advantage of the untrained network at V1 depends entirely on the specific analysis employed
How the training signal is built
The authors tested four main ways to examine their earlier result, including a pre-specified analysis using a fitted ridge readout on the full feature map. They also explored an analysis multiverse
involving 36 combinations of unit normalisation, distance measure, and voxel normalisation to test which conclusions hold across defensible choices.
Where the gains come from
The advantage of the untrained network at V1 was found not to be robust across all analyses; it disappeared under correlation distance when model units were standardised, and it was absent from the component orthogonal to the mean profile of the units. Furthermore, while raw correlation-distance RSA closely tracked mean-profile alignment, the advantage was absent from the component orthogonal to the mean profile and from parallel components alone did not resolve it. p. 2 and is absent from the angular component and from the component orthogonal to the mean profile, whereas raw correlation-distance RSA closely tracks
the mean-profile alignment of model and voxels. The ranking at LOC favours
the trained networks wherever it is resolved.Where the paper says it
Sensitivity and limitations
The results demonstrate that whether untrained networks beat trained ones at V1 depends on the analysis, as shown by how the sign of the V1 contrast changed across different configurations. The authors recommend reporting sensitivity to centring over the evaluated stimuli on both sides, reporting components separately, and including simple baselines to make such comparisons interpretable. p. 0 Intervals are
stimulus-bootstrap intervals for three subjects, and 6 pre-stated predictions were not met. We recommend reporting sensitivity to centring over
the evaluated stimuli on both sides, reporting components separately, and
including a small multiverse and simple baselines. For RSA,
we resampled stimuli (1000 bootstrap resamples, seed 20261002) and computed
differences between conditions from the same resamples; intervals are 95% percentile intervals. They describe variation over the 720 stimuli, for these three
subjects and these networks; training seeds are averaged within each resample
and are not part of the intervals.Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Develop an AI system that can perform robust visual representation analysis by explicitly decomposing representational similarity into norm and angular components to determine which component carries a specific advantage over trained networks at V1.
-
Implement a decision-making framework that incorporates multiple analysis choices, such as different distance measures (correlation, Euclidean) and feature normalisations (raw, centered), to assess the robustness of any observed model-brain advantage.
-
Create a system capable of predicting whether an untrained network will outperform a trained one at V1 by testing specific hypotheses derived from the multiverse analysis, such as whether the advantage is carried by the component along the mean profile of the units.
-
Build a mechanism to assess evaluation resolution dependence for model-brain similarity, allowing for quantification of how changes in stimulus resolution affect both raw and angular components of representation.
-
Design a pipeline that reports sensitivity to centring over evaluated stimuli on both sides, ensuring that ranking claims are justified by reporting the influence of unit mean differences on the results.
Abstract
Untrained convolutional networks have been reported to rival trained networks as models of visual cortex, and we had reported an untrained-network advantage over backpropagation at V1 in THINGS fMRI, together with a dependence on evaluation resolution. Under a pre-specified ridge readout, the untrained network scored below the trained one at every pooling grid (median r difference-0.043 at g = 4, 0 of 5 seeds positive), and also in MSE skill and per-fold r. In exploratory analyses, across 36 combinations of unit normalisation, distance measure and voxel normalisation (28 distinct analyses), the sign of the V1 contrast depended on the analysis (intervals above 0 in 5 and below 0 in 9 distinct analyses), and in our networks the resolution dependence of the original convention appeared in only one cell. Raw correlation-distance representational similarity analysis (RSA) covaried with the alignment of model and voxel patterns with the unit-mean profile (Spearman 0.97 across our networks, 0.79 across six ImageNet architectures); the advantage was absent from the component orthogonal to the mean profile, and the parallel component alone did not resolve it. On the unweighted angular component of centred patterns, trained networks were ahead at V1 (our network and 5 of 6 ImageNet models) and at LOC (all seven); the norm-weighted angular term was not resolved. A luminance-and-contrast bank scored above every network under ridge (point estimates), and adding network features under a shared penalty did not improve prediction. Intervals are stimulus-bootstrap intervals for three subjects, and 6 pre-stated predictions were not met. We recommend reporting sensitivity to centring over the evaluated stimuli on both sides, reporting components separately, and including a small multiverse and simple baselines.
Sources
Related papers
- Self-Supervised Learning of Local Predictive Directions as a Neural Computation
- Synchronized Hearts, Desynchronized Brains: A Nonlinear Dynamics Perspective for Function and Failure
- Message interaction dynamics covary with brain volume across mammalian connectomes
- Planning as Dynamics Relaxation: Hippocampal Recurrent Network Realizes Optimal Goal-Directed Navigation
- The Platonic brain bridge hypothesis: brain networks as architectural priors for multimodal large language models
- BrainPFN: Amortized Brain-Behaviour Prediction via Prior-Data Fitted Networks