Analysis choices decide whether untrained networks beat trained ones at V1: a multiverse analysis of model-brain RSA

arXiv:2611.15533 · q-bio.NC, cs.LG · Submitted 2026-11-11 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Genomics Radio. Generated commentary on the latest computational biology and genomics papers.

Ines: We're looking at the paper: "Analysis choices decide whether untrained networks beat trained ones at V1".

Marcus: Here's the gist of the paper. Untrained convolutional networks have been reported to rival trained networks as models of visual cortex and we had reported an untrained-network advantage over backpropagation at V1 in THINGS fMRI, together with a dependence on evaluation resolution Under a pre-specified ridge readout, the untrained network scored below the trained one at every pooling grid (median r difference −0.043 at g = 4, 0 of 5 seeds positive), and also in MSE skill and per-fold r Pruning redundant reasoning tokens The paper investigates whether untrained convolutional networks can outperform trained ones as models of visual cortex by examining the dependence of this advantage on various analysis choices, including distance measures, unit normalisation, and voxel normalisation.

Ines: The core finding is that the advantage of the untrained network at V1 depends entirely on the specific analysis employed How the training signal is built The authors tested four main ways to examine their earlier result, including a pre-specified analysis using a fitted ridge readout on the full feature map.

Marcus: They also explored an "analysis multiverse" involving 36 combinations of unit normalisation, distance measure, and voxel normalisation to test which conclusions hold across defensible choices.

Ines: Where the gains come from The advantage of the untrained network at V1 was found not to be robust across all analyses; it disappeared under correlation distance when model units were standardised, and it was absent from the component orthogonal to the mean profile of the units.

Paper summary: Ines: So this paper, "Analysis choices decide whether untrained networks beat trained ones at V1: a multiverse analysis of model-brain RSA," is really about how fragile those initial findings were that showed untrained networks could beat trained ones in the visual cortex models.

Marcus: The authors are showing that they can't just rely on one specific way to compare these neural network models against the training signal. They tested thirty-six different ways to look at the data, changing things like how you normalize units or what distance measure you use.

Yuki: What this means for us is that when we try to build these complex computational models of the visual cortex, we can't just pick one statistical tool and assume it tells us the whole story about performance.

Ines: They found that the advantage of the untrained network at V1 disappeared under correlation distance when they standardized those model units, and it vanished from components orthogonal to the mean profile of those units.

Marcus: Even when raw correlation distance representation similarity analysis closely tracked mean-profile alignment, that advantage still disappeared from those mean profile components, and parallel components didn't resolve it either.

Yuki: So essentially, if you don't standardize your approach and check all these different angles of comparison, you can't reliably claim one model beats another in this context.

Ines: They are suggesting we need to report sensitivity to centring over the stimuli on both sides, and report components separately with simple baselines so people can actually compare these findings themselves.

Marcus: The limitation they state is that the data comes from three subjects in a single dataset, and those intervals described different sources of variation, which means we need to be careful about generalizing those results broadly.

Yuki: So while this work shows how analysis choices affect model comparison at V1, the authors are being very clear that we still have limitations regarding sample size and the complexity of biological variation.

Ines: It really highlights just how much our conclusions in these fields depend on our specific analytical framework rather than just the underlying biology.

Conclusion: Ines: So, we're looking at this paper now called "Analysis choices decide whether untrained networks beat trained ones at V1: a multiverse analysis of model-brain RSA." It really hammers home how fragile those initial findings were that suggested untrained networks could outperform trained ones in models of the visual cortex.

Marcus: The authors are showing that you can’t just pick one specific way to compare these neural network models against the training signal. They tested thirty-six different ways to look at the data, changing things like how they normalize the units or what distance measure they use.

Yuki: What this means for us is that when we build these complex computational models of systems like the visual cortex, we can’t just pick one statistical tool and assume it tells us the whole story about performance.

Ines: They found that the advantage of the untrained network at V1 disappeared under correlation distance when they standardized those model units. It also vanished from components orthogonal to the mean profile of those units.

Marcus: Even when raw correlation distance representation similarity analysis closely tracked mean-profile alignment, that advantage still disappeared from those mean profile components, and parallel components didn't resolve it either.

Yuki: So essentially, if you don't standardize your approach and check all these different angles of comparison, you can’t reliably claim one model beats another in this context.

Ines: They are suggesting we need to report sensitivity to centring over the stimuli on both sides, and report components separately with simple baselines so people can actually compare these findings themselves in a meaningful way.

Marcus: The limitation they state is that the data comes from three subjects in a single dataset, and those intervals described different sources of variation, which means we need to be careful about generalizing those results broadly.

Yuki: So while this work shows how analysis choices affect model comparison at V1, the authors are being very clear that we still have limitations regarding sample size and the complexity of biological variation.

Ines: It really highlights just how much our conclusions in these fields depend on our specific analytical framework rather than just the underlying biology.

Nils Leutenegger

q-bio.NC, cs.LG

Submitted: 2026-11-11

Updated: 2026-11-11

Comments: 31 pages, 6 figures; code and data pipeline: https://github.com/nilsleut/ridge-readout

Code: https://github.com/nilsleut/ridge-readout

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 81/100

The gist: Untrained convolutional networks have been reported to rival trained networks as models of visual cortex and we had reported an untrained-network advantage over backpropagation at V1 in THINGS fMRI…

Key concepts

Untrained Convolutional Networks
These are neural networks with no prior training data applied to them. The researchers tested if these networks could mimic the function of a trained visual cortex model, showing whether an untrained structure could perform better than one that has been learned from real visual data.
Analysis Multiverse
This refers to testing many different combinations of analytical choices—like how units are normalized, what distance measure is used, and how voxels are normalized. The goal was to see if the observed advantage of the untrained network holds true regardless of which specific analysis method is chosen.
V1
V1 (Visual Area 1) is a specific region in the visual cortex that processes basic visual information. The researchers tested whether their model-brain comparison held up specifically at this location, examining how the untrained network performed against trained models there.

Terminology

Summary

Untrained convolutional networks have been reported to rival trained networks as models of visual cortex and we had reported an untrained-network advantage over backpropagation at V1 in THINGS fMRI, together with a dependence on evaluation resolution Under a pre-specified ridge readout, the untrained network scored below the trained one at every pooling grid (median r difference −0. p. 0

Where the paper says it

, 2025], and we had reported an untrained-network advantage over backpropagation at V1 in THINGS fMRI, together with a dependence on evaluation resolution [Leutenegger, 2026a]. Under a pre-specified ridge readout, the untrained network scored below the trained one at every pooling grid (median r difference −0.

043 at g = 4, 0 of 5 seeds positive), and also in MSE skill and per-fold r

Pruning redundant reasoning tokens

The paper investigates whether untrained convolutional networks can outperform trained ones as models of visual cortex by examining the dependence of this advantage on various analysis choices, including distance measures, unit normalisation, and voxel normalisation. The core finding is that the advantage of the untrained network at V1 depends entirely on the specific analysis employed

How the training signal is built

The authors tested four main ways to examine their earlier result, including a pre-specified analysis using a fitted ridge readout on the full feature map. They also explored an analysis multiverse involving 36 combinations of unit normalisation, distance measure, and voxel normalisation to test which conclusions hold across defensible choices.

Where the gains come from

The advantage of the untrained network at V1 was found not to be robust across all analyses; it disappeared under correlation distance when model units were standardised, and it was absent from the component orthogonal to the mean profile of the units. Furthermore, while raw correlation-distance RSA closely tracked mean-profile alignment, the advantage was absent from the component orthogonal to the mean profile and from parallel components alone did not resolve it. p. 2

Where the paper says it

and is absent from the angular component and from the component orthogonal to the mean profile, whereas raw correlation-distance RSA closely tracks the mean-profile alignment of model and voxels. The ranking at LOC favours the trained networks wherever it is resolved.

Sensitivity and limitations

The results demonstrate that whether untrained networks beat trained ones at V1 depends on the analysis, as shown by how the sign of the V1 contrast changed across different configurations. The authors recommend reporting sensitivity to centring over the evaluated stimuli on both sides, reporting components separately, and including simple baselines to make such comparisons interpretable. p. 0

Where the paper says it

Intervals are stimulus-bootstrap intervals for three subjects, and 6 pre-stated predictions were not met. We recommend reporting sensitivity to centring over the evaluated stimuli on both sides, reporting components separately, and including a small multiverse and simple baselines.

The data are limited by using three subjects from a single dataset and the fact that intervals described different sources of variation. p. 8
Where the paper says it

For RSA, we resampled stimuli (1000 bootstrap resamples, seed 20261002) and computed differences between conditions from the same resamples; intervals are 95% percentile intervals. They describe variation over the 720 stimuli, for these three subjects and these networks; training seeds are averaged within each resample and are not part of the intervals.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Develop an AI system that can perform robust visual representation analysis by explicitly decomposing representational similarity into norm and angular components to determine which component carries a specific advantage over trained networks at V1.

  2. Implement a decision-making framework that incorporates multiple analysis choices, such as different distance measures (correlation, Euclidean) and feature normalisations (raw, centered), to assess the robustness of any observed model-brain advantage.

  3. Create a system capable of predicting whether an untrained network will outperform a trained one at V1 by testing specific hypotheses derived from the multiverse analysis, such as whether the advantage is carried by the component along the mean profile of the units.

  4. Build a mechanism to assess evaluation resolution dependence for model-brain similarity, allowing for quantification of how changes in stimulus resolution affect both raw and angular components of representation.

  5. Design a pipeline that reports sensitivity to centring over evaluated stimuli on both sides, ensuring that ranking claims are justified by reporting the influence of unit mean differences on the results.

Abstract

Untrained convolutional networks have been reported to rival trained networks as models of visual cortex, and we had reported an untrained-network advantage over backpropagation at V1 in THINGS fMRI, together with a dependence on evaluation resolution. Under a pre-specified ridge readout, the untrained network scored below the trained one at every pooling grid (median r difference-0.043 at g = 4, 0 of 5 seeds positive), and also in MSE skill and per-fold r. In exploratory analyses, across 36 combinations of unit normalisation, distance measure and voxel normalisation (28 distinct analyses), the sign of the V1 contrast depended on the analysis (intervals above 0 in 5 and below 0 in 9 distinct analyses), and in our networks the resolution dependence of the original convention appeared in only one cell. Raw correlation-distance representational similarity analysis (RSA) covaried with the alignment of model and voxel patterns with the unit-mean profile (Spearman 0.97 across our networks, 0.79 across six ImageNet architectures); the advantage was absent from the component orthogonal to the mean profile, and the parallel component alone did not resolve it. On the unweighted angular component of centred patterns, trained networks were ahead at V1 (our network and 5 of 6 ImageNet models) and at LOC (all seven); the norm-weighted angular term was not resolved. A luminance-and-contrast bank scored above every network under ridge (point estimates), and adding network features under a shared penalty did not improve prediction. Intervals are stimulus-bootstrap intervals for three subjects, and 6 pre-stated predictions were not met. We recommend reporting sensitivity to centring over the evaluated stimuli on both sides, reporting components separately, and including a small multiverse and simple baselines.

Sources

Related papers