Analysis choices decide whether untrained networks beat trained ones at V1: a multiverse analysis of model-brain RSA

summary

The gist

Untrained convolutional networks have been reported to rival trained networks as models of visual cortex and we had reported an untrained-network advantage over backpropagation at V1 in THINGS fMRI…

In short

The study investigated if untrained convolutional networks can outperform trained ones as models of visual cortex by testing different analysis choices. The core finding is that this advantage is not universal; it depends entirely on the specific metrics used, such as distance measures and normalization techniques. The advantage disappeared under certain conditions, suggesting the result is highly sensitive to how the data are analyzed.

Key concepts

Untrained Convolutional Networks
These are neural networks with no prior training data applied to them. The researchers tested if these networks could mimic the function of a trained visual cortex model, showing whether an untrained structure could perform better than one that has been learned from real visual data.
Analysis Multiverse
This refers to testing many different combinations of analytical choices—like how units are normalized, what distance measure is used, and how voxels are normalized. The goal was to see if the observed advantage of the untrained network holds true regardless of which specific analysis method is chosen.
V1
V1 (Visual Area 1) is a specific region in the visual cortex that processes basic visual information. The researchers tested whether their model-brain comparison held up specifically at this location, examining how the untrained network performed against trained models there.

Terminology used across episodes

This episode discusses

The paper

Analysis choices decide whether untrained networks beat trained ones at V1: a multiverse analysis of model-brain RSA · Read on arXiv

Nils Leutenegger

Untrained convolutional networks have been reported to rival trained networks as models of visual cortex, and we had reported an untrained-network advantage over backpropagation at V1 in THINGS fMRI, together with a dependence on evaluation resolution. Under a pre-specified ridge readout, the untrained network scored below the trained one at every pooling grid (median r difference-0.043 at g = 4, 0 of 5 seeds positive), and also in MSE skill and per-fold r. In exploratory analyses, across 36 combinations of unit normalisation, distance measure and voxel normalisation (28 distinct analyses), the sign of the V1 contrast depended on the analysis (intervals above 0 in 5 and below 0 in 9 distinct analyses), and in our networks the resolution dependence of the original convention appeared in only one cell. Raw correlation-distance representational similarity analysis (RSA) covaried with the alignment of model and voxel patterns with the unit-mean profile (Spearman 0.97 across our networks, 0.79 across six ImageNet architectures); the advantage was absent from the component orthogonal to the mean profile, and the parallel component alone did not resolve it. On the unweighted angular component of centred patterns, trained networks were ahead at V1 (our network and 5 of 6 ImageNet models) and at LOC (all seven); the norm-weighted angular term was not resolved. A luminance-and-contrast bank scored above every network under ridge (point estimates), and adding network features under a shared penalty did not improve prediction. Intervals are stimulus-bootstrap intervals for three subjects, and 6 pre-stated predictions were not met. We recommend reporting sensitivity to centring over the evaluated stimuli on both sides, reporting components separately, and including a small multiverse and simple baselines.

Transcript

Introduction to the show: ident: Genomics Radio. Generated commentary on the latest computational biology and genomics papers.

Ines: We're looking at the paper: "Analysis choices decide whether untrained networks beat trained ones at V1".

Marcus: Here's the gist of the paper. Untrained convolutional networks have been reported to rival trained networks as models of visual cortex and we had reported an untrained-network advantage over backpropagation at V1 in THINGS fMRI, together with a dependence on evaluation resolution Under a pre-specified ridge readout, the untrained network scored below the trained one at every pooling grid (median r difference −0.043 at g = 4, 0 of 5 seeds positive), and also in MSE skill and per-fold r Pruning redundant reasoning tokens The paper investigates whether untrained convolutional networks can outperform trained ones as models of visual cortex by examining the dependence of this advantage on various analysis choices, including distance measures, unit normalisation, and voxel normalisation.

Ines: The core finding is that the advantage of the untrained network at V1 depends entirely on the specific analysis employed How the training signal is built The authors tested four main ways to examine their earlier result, including a pre-specified analysis using a fitted ridge readout on the full feature map.

Marcus: They also explored an "analysis multiverse" involving 36 combinations of unit normalisation, distance measure, and voxel normalisation to test which conclusions hold across defensible choices.

Ines: Where the gains come from The advantage of the untrained network at V1 was found not to be robust across all analyses; it disappeared under correlation distance when model units were standardised, and it was absent from the component orthogonal to the mean profile of the units.

Paper summary: Ines: So this paper, "Analysis choices decide whether untrained networks beat trained ones at V1: a multiverse analysis of model-brain RSA," is really about how fragile those initial findings were that showed untrained networks could beat trained ones in the visual cortex models.

Marcus: The authors are showing that they can't just rely on one specific way to compare these neural network models against the training signal. They tested thirty-six different ways to look at the data, changing things like how you normalize units or what distance measure you use.

Yuki: What this means for us is that when we try to build these complex computational models of the visual cortex, we can't just pick one statistical tool and assume it tells us the whole story about performance.

Ines: They found that the advantage of the untrained network at V1 disappeared under correlation distance when they standardized those model units, and it vanished from components orthogonal to the mean profile of those units.

Marcus: Even when raw correlation distance representation similarity analysis closely tracked mean-profile alignment, that advantage still disappeared from those mean profile components, and parallel components didn't resolve it either.

Yuki: So essentially, if you don't standardize your approach and check all these different angles of comparison, you can't reliably claim one model beats another in this context.

Ines: They are suggesting we need to report sensitivity to centring over the stimuli on both sides, and report components separately with simple baselines so people can actually compare these findings themselves.

Marcus: The limitation they state is that the data comes from three subjects in a single dataset, and those intervals described different sources of variation, which means we need to be careful about generalizing those results broadly.

Yuki: So while this work shows how analysis choices affect model comparison at V1, the authors are being very clear that we still have limitations regarding sample size and the complexity of biological variation.

Ines: It really highlights just how much our conclusions in these fields depend on our specific analytical framework rather than just the underlying biology.

Conclusion: Ines: So, we're looking at this paper now called "Analysis choices decide whether untrained networks beat trained ones at V1: a multiverse analysis of model-brain RSA." It really hammers home how fragile those initial findings were that suggested untrained networks could outperform trained ones in models of the visual cortex.

Marcus: The authors are showing that you can’t just pick one specific way to compare these neural network models against the training signal. They tested thirty-six different ways to look at the data, changing things like how they normalize the units or what distance measure they use.

Yuki: What this means for us is that when we build these complex computational models of systems like the visual cortex, we can’t just pick one statistical tool and assume it tells us the whole story about performance.

Ines: They found that the advantage of the untrained network at V1 disappeared under correlation distance when they standardized those model units. It also vanished from components orthogonal to the mean profile of those units.

Marcus: Even when raw correlation distance representation similarity analysis closely tracked mean-profile alignment, that advantage still disappeared from those mean profile components, and parallel components didn't resolve it either.

Yuki: So essentially, if you don't standardize your approach and check all these different angles of comparison, you can’t reliably claim one model beats another in this context.

Ines: They are suggesting we need to report sensitivity to centring over the stimuli on both sides, and report components separately with simple baselines so people can actually compare these findings themselves in a meaningful way.

Marcus: The limitation they state is that the data comes from three subjects in a single dataset, and those intervals described different sources of variation, which means we need to be careful about generalizing those results broadly.

Yuki: So while this work shows how analysis choices affect model comparison at V1, the authors are being very clear that we still have limitations regarding sample size and the complexity of biological variation.

Ines: It really highlights just how much our conclusions in these fields depend on our specific analytical framework rather than just the underlying biology.

More episodes

← Home