No Universal Winner in Model Selection: Regression versus Representational Similarity Analysis
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Statistics. The day in statistics: methods, theory and the inference under them.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "No Universal Winner in Model Selection".
Jane: Here's the gist of the paper. In cognitive psychology and neuroscience, adjudicating between competing theoretical models is a common methodological challenge.
Tom: The relative performance of regression, RSA, and PCM depends on the form of the mapping between features and responses, as well as on the structure of the feature space.
Jane: Model Comparison Framework The study systematically evaluated regression, RSA, and Pattern Component Modeling (PCM) across distinct data-generating schemes including first-order linear mappings and geometry-to-first-order transformations with either linear or nonlinear sigmoid readouts.
Tom: The authors established two data-generating schemes: a first-order direct-mapping scheme and a geometry-to-first-order scheme incorporating either linear or nonlinear (sigmoid) transformations.
Title and authors: Tom: So we're looking at this paper from arXiv titled "No Universal Winner in Model Selection: Regression versus Representational Similarity Analysis," and it’s about how we choose between different ways of modeling data when you’re trying to figure out what’s going on in the brain or behavior.
Jane: That sounds really common, because researchers often use these first-order direct mapping approaches like linear regression, and then they use second-order abstraction methods like Representational Similarity Analysis, or RSA. The big question this paper tackles is whether the data itself, or how those features are structured, actually dictates which method works better for selecting the right model.
Lu: Exactly. The authors systematically tested regression, RSA, and Pattern Component Modeling across different ways data can be generated—things like simple linear mappings versus more complex geometry-to-first-order transformations with either linear or nonlinear sigmoid readouts.
Meng: But what’s interesting is that they didn't just run them once; they manipulated things like sample size, noise levels, feature dimensionality, and multicollinearity to see how sensitive each method was to those kinds of shifts.
Lalam: The core finding is that the relative performance really depends on the underlying generative mechanism of your data. For instance, under linear generative assumptions, regression and Pattern Component Modeling showed higher model selection accuracy than RSA.
Tom: That makes sense for me because when you have a direct, linear mapping from features to responses, like in first-order linear generation, regression just seems to provide clearer discrimination than RSA when you look at adjusted R squared and Cohen’s d.
Jane: But what happens when the data isn't perfectly linear? The authors found that under nonlinear but order-preserving transformations, rank-based RSA actually showed an advantage over both regression and Pattern Component Modeling.
Lu: That’s a key insight because it suggests that when the mapping is nonlinear but still keeps the ordinal structure of things intact, using rank-based second-order comparison can be more informative than sticking to linear regression or Pearson based RSA.
Meng: I’m curious about how feature space structure plays into this. The paper found that increasing feature dimensionality under a linear mapping generally strengthened the advantage for regression over RSA.
Tom: And multicollinearity also mattered in different ways, which is interesting because it shows the methods aren't all equally robust across those shifts.
Title and authors: Jane: They looked at how multicollinearity affected things differently depending on the data-generating scheme; under first-order generation, increasing collinearity among irrelevant features selectively impaired RSA while regression stayed relatively stable.
Lu: And they did a bit of empirical validation, and they found that RSA yielded lower model selection accuracy and produced more incorrect conclusions than linear regression when applied to the same datasets in their simulations.
Meng: That’s a big warning for anyone shipping models to production; if you rely too much on RSA without checking these conditions, you might get misleading results.
Tom: But they also found a way to improve RSA's performance under certain conditions by applying Principal Component Analysis, or PCA, to the predictor matrices.
Jane: That’s an interesting technical detail because applying PCA improved RSA’s ability to differentiate the two models, especially when sample sizes were small.
Lu: The paper suggests that PCA mainly changes how the feature representation is set up for RSA and that this reduces how much raw collinearity structure affects performance in first-order schemes.
Meng: So, if I’m working on a system where features are highly correlated, it seems like PCA could be a useful sensitivity analysis tool to see if the method differences depend strongly on the original feature representation rather than just improving RSA's ability across all conditions.
Tom: It sounds like the paper is arguing that we shouldn't just pick one method and assume it’s always best, but that its usefulness really depends on the structure of your data.
Jane: And they wrapped up by suggesting a few practical things for researchers moving forward, summarizing what we discussed about linear versus nonlinear mappings and the role of feature space structure.
Lu: The overall implication is that regression remains highly informative when you have direct feature-to-response mappings that are theoretically feasible and empirically sound under linear conditions.
Meng: For my work at scale, if you have a clear first-order prediction, regression is solid; but if the mapping is more complex, like the sigmoid readout in multivoxel data, things get trickier.
Tom: And when we look at those multivoxel fMRI extensions specifically, they found that Pattern Component Modeling and regression showed the strongest performance when the mapping was approximately linear.
Title and authors: Jane: But under that sigmoid readout condition with multivoxel data, rank-based RSA often became more competitive than regression and Pearson based RSA at larger sample sizes.
Lu: That reinforces the idea that even in those nonlinear settings, if you can use a rank-based comparison, it can be more informative than standard linear regression.
Meng: I just have to ask what the practical trade-off is there between using the simpler regression when things are linear and using the rank-based RSA when things aren't?
Tom: It really boils down to whether you’re prioritizing a direct first-order test or a more abstract relational structure.
Jane: So, to wrap up this paper on "No Universal Winner in Model Selection: Regression versus Representational Similarity Analysis," the main point is that there isn't one single best method for everything.
Lu: We see that regression and Pattern Component Modeling win under linear assumptions, while rank-based RSA shines when the data has a nonlinear but order-preserving structure.
Meng: And they found that PCA can actually help make RSA more reliable when predictors are highly correlated, which is something I'll keep in mind for scaling up my feature spaces.
Lalam: So, to summarize the main points of this paper, the relative performance of regression versus RSA changes based entirely on whether you’re dealing with linear or nonlinear data generation mechanisms.
Tom: It’s a reminder that we need to be careful about which tools we use and what assumptions we make when we try to adjudicate between these competing theoretical models.
Jane: The real implication for the listener is just this: don't assume one method is universally superior; you have to look at your data structure first.
Lu: We’re looking forward to seeing if this pattern of performance across linear and nonlinear transformations holds up when we test it with more complex types of non-monotonic functions in future work.
Meng: I’m hoping that the insights about PCA helping RSA won't just be a one-off trick but actually helps us design better feature spaces for production systems.
Lalam: This paper, "No Universal Winner in Model Selection: Regression versus Representational Similarity Analysis," shows us that we need to use these tools complementarily instead of viewing them as mutually exclusive competitors.
The paper's summary: Tom: So we're looking at this paper from arXiv titled "No Universal Winner in Model Selection: Regression versus Representational Similarity Analysis," and it’s about how we choose between different ways of modeling data when you’re trying to figure out what’s going on in the brain or behavior.
Jane: That sounds really common, because researchers often use these first-order direct mapping approaches like linear regression, and then they use second-order abstraction methods like Representational Similarity Analysis, or RSA. The big question this paper tackles is whether the data itself, or how those features are structured, actually dictates which method works better for selecting the right model.
Lu: Exactly. The authors systematically tested regression, RSA, and Pattern Component Modeling across different ways data can be generated—things like simple linear mappings versus more complex geometry-to-first-order transformations with either linear or nonlinear sigmoid readouts.
Meng: But what’s interesting is that they didn't just run them once; they manipulated things like sample size, noise levels, feature dimensionality, and multicollinearity to see how sensitive each method was to those kinds of shifts.
Lalam: The core finding is that the relative performance really depends on the underlying generative mechanism of your data. For instance, under linear generative assumptions, regression and Pattern Component Modeling showed higher model selection accuracy than RSA.
Tom: That makes sense for me because when you have a direct, linear mapping from features to responses, like in first-order linear generation, regression just seems to provide clearer discrimination than RSA when you look at adjusted R squared and Cohen’s d.
Jane: But what happens when the data isn't perfectly linear? The authors found that under nonlinear but order-preserving transformations, rank-based RSA actually showed an advantage over both regression and Pattern Component Modeling.
Lu: That’s a key insight because it suggests that when the mapping is nonlinear but still keeps the ordinal structure of things intact, using rank-based second-order comparison can be more informative than sticking to linear regression or Pearson based RSA.
Meng: I’m curious about how feature space structure plays into this. The paper found that increasing feature dimensionality under a linear mapping generally strengthened the advantage for regression over RSA.
Tom: And multicollinearity also mattered in different ways, which is interesting because it shows the methods aren't all equally robust across those shifts.
Jane: They looked at how multicollinearity affected things differently depending on the data-generating scheme; under first-order generation, increasing collinearity among irrelevant features selectively impaired RSA while regression stayed relatively stable.
Lu: And they did a bit of empirical validation, and they found that RSA yielded lower model selection accuracy and produced more incorrect conclusions than linear regression when applied to the same datasets in their simulations.
Meng: That’s a big warning for anyone shipping models to production; if you rely too much on RSA without checking these conditions, you might get misleading results.
Tom: But they also found a way to improve RSA's performance under certain conditions by applying Principal Component Analysis, or PCA, to the predictor matrices.
Jane: That’s an interesting technical detail because applying PCA improved RSA’s ability to differentiate the two models, especially when sample sizes were small.
Lu: The paper suggests that PCA mainly changes how the feature representation is set up for RSA and that this reduces how much raw collinearity structure affects performance in first-order schemes.
Meng: So, if I’m working on a system where features are highly correlated, it seems like PCA could be a useful sensitivity analysis tool to see if the method differences depend strongly on the original feature representation rather than just improving RSA's ability across all conditions.
Tom: It sounds like the paper is arguing that we shouldn't just pick one method and assume it’s always best, but that its usefulness really depends on the structure of your data.
Jane: And they wrapped up by suggesting a few practical things for researchers moving forward, summarizing what we discussed about linear versus nonlinear mappings and the role of feature space structure.
Lu: The overall implication is that regression remains highly informative when you have direct feature-to-response mappings that are theoretically feasible and empirically sound under linear conditions.
Meng: For my work at scale, if you have a clear first-order prediction, regression is solid; but if the mapping is more complex, like the sigmoid readout in multivoxel data, things get trickier.
Tom: And when we look at those multivoxel fMRI extensions specifically, they found that Pattern Component Modeling and regression showed the strongest performance when the mapping was approximately linear.
Jane: But under that sigmoid readout condition with multivoxel data, rank-based RSA often became more competitive than regression and Pearson based RSA at larger sample sizes.
Lu: That reinforces the idea that even in those nonlinear settings, if you can use a rank-based comparison, it can be more informative than standard linear regression.
Meng: I just have to ask what the practical trade-off is there between using the simpler regression when things are linear and using the rank-based RSA when things aren't?
Tom: It really boils down to whether you’re prioritizing a direct first-order test or a more abstract relational structure.
The paper's improvements: Tom: So we're looking at how this paper suggests we can actually fix those model selection issues when you're dealing with regression, RSA, and Pattern Component Modeling.
Jane: It basically says that instead of just picking one method blindly, you need to know what kind of data generation you have first—whether it’s linear or nonlinear—and then pick the tool that fits that structure.
Lu: The paper proposes using PCA as a sensitivity analysis tool, which means you can use Principal Component Analysis on the feature space to see if the method differences depend strongly on the original feature representation rather than just uniformly improving RSA performance across all conditions.
Meng: That makes sense for me because when I'm shipping models, I need to know if a performance boost is real or just an artifact of how I chose my initial features.
Tom: It suggests that PCA can actually help make RSA more reliable when predictors are highly correlated, which is something I'll keep in mind for scaling up my feature spaces.
Jane: What this changes for us is that instead of treating these models like they're all equal competitors, we have a structured way to choose which one to trust based on the data generation process.
Lu: The paper also points out that when you look at those multivoxel fMRI data under a sigmoid readout condition, the relative ordering of methods can shift, with rank-based RSA often becoming more competitive than regression and Pearson based RSA at larger sample sizes.
Meng: So if I'm deploying something in a setting where we have that kind of nonlinear readout, I might need to look beyond just the simple linear regression scores.
Tom: Exactly. It’s about moving past just looking at the raw R squared values and instead understanding the underlying structure of how those responses are actually being mapped from the features.
Jane: So, if you're working on a complex system where you can't assume linearity, this paper gives you a way to test your assumptions using PCA to see if your model choice is robust or just lucky with the initial feature setup.
Lu: It’s about being rigorous in how you set up those input features so that the output of the analysis isn't just dependent on some arbitrary starting point.
Meng: I guess it means more upfront work to prep the data structure, but if that prevents us from deploying a broken model later, then it might be worth the cost.
Tom: It’s about making sure our choice of AI architecture or statistical approach isn't just working for one specific data shape and failing everywhere else.
Jane: So, to wrap up this paper on "No Universal Winner in Model Selection," it shows us we need to use these tools complementarily instead of viewing them as mutually exclusive competitors.
Lu: We see that regression and Pattern Component Modeling win under linear assumptions, while rank-based RSA shines when the data has a nonlinear but order-preserving structure.
Meng: And they found that PCA can actually help make RSA more reliable when predictors are highly correlated, which is something I'll keep in mind for scaling up my feature spaces.
Tom: It’s a reminder that we need to be careful about which tools we use and what assumptions we make when we try to adjudicate between these competing theoretical models.
Conclusion: Tom: So we’re wrapping up this discussion on "No Universal Winner in Model Selection: Regression versus Representational Similarity Analysis." We just covered how the performance of regression, RSA, and Pattern Component Modeling totally depends on whether your data generation is linear or nonlinear.
Jane: That’s right. The main thing the paper shows is that there isn't one single best method for every situation; you have to match the tool to the data structure first.
Lu: And they pointed out how applying PCA as a sensitivity analysis tool can help us see if our model choice depends on just a lucky initial feature setup or if it’s actually robust.
Meng: So, if I'm shipping models, this means I need to be more careful about the feature space we feed into the system before we trust any of these comparisons.
Lalam: I'm sure of my answer here: the paper proves that regression and Pattern Component Modeling are strong when things are linear, but rank-based RSA can win when there’s a nonlinear but order-preserving structure.
Tom: It’s a reminder that we need to be careful about which tools we use and what assumptions we make when we try to adjudicate between these competing theoretical models.
Jane: Exactly. The implication is that for researchers, the choice of method isn't just about picking the fanciest algorithm; it's about understanding the math behind how your data actually behaves.
Lu: It suggests that future work should probably focus on testing whether this pattern of performance holds up when we introduce more complex non-monotonic functions into those transformations.
Meng: I think for production, it’s all about knowing the costs and what breaks at scale, so if we can use PCA to stabilize RSA, that’s a win for deployment stability.
Tom: Yeah, I agree. We need to stop treating these methods like they are all equally good baseline models across every single scenario.
Jane: So remember, the next time you're choosing between regression and RSA, look at how your data was generated before you pick a winner.
Chuanji Gaoa, Gang Chend, Svetlana V. Shinkareva, Rutvik H. Desaie
School of Psychology, Nanjing Normal University · Adolescent Education and Intelligence Support Lab of Nanjing Normal University Laboratory of Philosophy and Social Sciences at Universities in Jiangsu Province, Nanjing, China · The Joint Research Institute for Brain Science and Mental Health, Nanjing Normal University and Taizhou Fifth People's Hospital, China · Scientific and Statistical Computing Core, National Institute of Mental Health
stat.ME, stat.CO
Submitted: 2025-11-01
Updated: 2026-11-09
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 81/100
The gist: In cognitive psychology and neuroscience, adjudicating between competing theoretical models is a common methodological challenge.
Key concepts
- Regression
- A statistical method used to predict a response variable based on input features. It performs strongly when the relationship between features and responses is a direct, linear mapping, providing clear discrimination under first-order linear conditions.
- Representational Similarity Analysis (RSA)
- A technique that measures the similarity between different patterns of neural activity or feature representations. It is particularly valuable when direct feature-to-response mappings are unavailable or when the scientific goal is to understand relational structure rather than simple prediction.
- Pattern Component Modeling (PCM)
- A flexible approach used to model representational structure by decomposing data into fundamental components. PCM and regression show the strongest performance in multivoxel fMRI settings when the underlying feature mapping is approximately linear.
- Feature Space Structure
- Refers to the properties of the input data, such as dimensionality, noise levels, and multicollinearity among features. The relative success of methods like RSA versus regression is highly dependent on these structural properties.
Terminology
Summary
In cognitive psychology and neuroscience, adjudicating between competing theoretical models is a common methodological challenge. p. 1 Abstract
In cognitive psychology and neuroscience, adjudicating between competing theoretical
models is a common methodological challenge. Researchers often rely on either firstorder direct mapping approaches (e. Finally, analyses of empirical datasets
were consistent with the simulation results under approximately linear conditions, with
regression showing clearer model discrimination than RSA. Overall, these findings
suggest that the relative performance of regression, RSA, and PCM depends on the
form of the mapping between features and responses, as well as on the structure of the
feature space.Where the paper says it
Where the paper says it
Model Comparison Framework
The study systematically evaluated regression, RSA, and Pattern Component Modeling (PCM) across distinct data-generating schemes including first-order linear mappings and geometry-to-first-order transformations with either linear or nonlinear sigmoid readouts. The authors established two data-generating schemes: a first-order direct-mapping scheme and a geometry-to-first-order scheme incorporating either linear or nonlinear (sigmoid) transformations. p. 4 To address this gap, the current study systematically evaluated and compared the
model-selection accuracy of regression and RSA through computational simulations
and empirical analyses. To formalize the theoretical possibilities outlined above, we
established two data-generating schemes: a first-order direct-mapping scheme and a
geometry-to-first-order scheme incorporating either linear or nonlinear (sigmoid)
transformations.Where the paper says it
Performance Under Linear Mappings
Under linear generative assumptions, regression and PCM showed higher model selection accuracy than RSA. In the empirical analysis, regression showed clearer model discrimination than RSA when under approximately linear conditions. p. 1 We also found that
feature multicollinearity affected these methods differently across generative schemes,
and that orthogonalizing the predictor space via principal component analysis (PCA)
reduced several collinearity-related differences. Finally, analyses of empirical datasets
were consistent with the simulation results under approximately linear conditions, with
regression showing clearer model discrimination than RSA. Data analysis
For each simulated dataset, we evaluated model-response correspondence using both
regression and representational similarity analysis (RSA). Importantly, the analysis
pipeline was identical across the two data-generating schemes (first-order and
geometry-to-first-order); only the generation of Y differed.Where the paper says it
Where the paper says it
Performance Under Nonlinear Mappings
Under nonlinear but order-preserving transformations, rank-based RSA showed an advantage over regression and PCM. When Y was generated from the geometry-to-first-order scheme with a sigmoid readout, RSA could outperform regression in model selection and Spearman’s ρ outperformed Pearson’s r. p. 4 In a firstorder scheme, responses were generated directly from a linear regression model,
defining a known stimulus-response relationship (Figure 1). In a second scheme
(geometry-to-first-order), responses were generated from a one-dimensional coordinate
capturing the representational geometry of X, using either a linear or a sigmoid readout
(Figure 2). Under linear generative assumptions, regression and PCM showed higher modelselection accuracy than RSA. Under nonlinear but order-preserving transformations,
rank-based RSA showed an advantage over regression and PCM.Where the paper says it
Where the paper says it
Influence of Feature Space Structure and Noise
The relative performance of methods depends on the properties of the feature space and noise levels. For instance, increasing feature dimensionality under a linear mapping generally strengthened the advantage of regression over RSA. p. 3 If responses arise from a direct linear mapping from features, first-order
regression may retain an advantage. Conversely, if neural or behavioral responses
reflect a transformed readout of an underlying representational geometry, RSA may in Under nonlinear but order-preserving transformations,
rank-based RSA showed an advantage over regression and PCM. We also found that
feature multicollinearity affected these methods differently across generative schemes,
and that orthogonalizing the predictor space via principal component analysis (PCA)
reduced several collinearity-related differences.Where the paper says it
Where the paper says it
Empirical Validation and PCA Sensitivity
The empirical analysis showed that RSA yielded lower model-selection accuracy and produced more incorrect conclusions than linear regression when applied to the same datasets. Applying Principal Component Analysis (PCA) to the predictor matrices improved RSA’s ability to differentiate the two models, particularly at smaller sample sizes. p. 1 Under nonlinear but order-preserving transformations,
rank-based RSA showed an advantage over regression and PCM. We also found that
feature multicollinearity affected these methods differently across generative schemes,
and that orthogonalizing the predictor space via principal component analysis (PCA)
reduced several collinearity-related differences.Where the paper says it
Conclusion and Practical Implications
The results suggest that regression, RSA, and PCM should not be viewed as mutually exclusive competitors in all settings but rather as complementary tools whose usefulness depends on the structure of the data. When direct feature-to-response mappings are theoretically meaningful and empirically feasible, regression remains highly informative because it provides sensitive first-order tests of competing models under linear conditions. p. 2 One common approach for evaluating the effects of competing models on a response
variable is linear regression. As a first-order approach, regression assumes a direct
mapping between stimulus features and observed responses. These conceptual differences raise a straightforward methodological question: when
both multiple regression and RSA are applicable to the same dataset, which approach
provides more accurate and reliable model selection? We suggest that the answer is
unlikely to be universal and may instead depend on the nature of the stimulus-response
mappings. Under linear generative assumptions, regression and PCM showed higher modelselection accuracy than RSA. Under nonlinear but order-preserving transformations,
rank-based RSA showed an advantage over regression and PCM.Where the paper says it
Where the paper says it
Where the paper says it
Follow-up fMRI Extension Findings
In the fMRI extension of the first-order setting, PCM and regression showed the strongest performance when the mapping was approximately linear. Under the sigmoid-readout condition in multivoxel data, Spearman-based RSA often became the strongest RSA variant. p. 3 However, this flexibility may come at a
cost: transforming first-order data into distance or dissimilarity matrices may discard
information that is relevant for model selection. Indeed, empirical studies often report
relatively modest RSA correlation magnitudes (e. Under linear generative assumptions, regression and PCM showed higher modelselection accuracy than RSA. Under nonlinear but order-preserving transformations,
rank-based RSA showed an advantage over regression and PCM. For example, given a series of word stimuli, researchers can pass them
through a word frequency model to obtain a high-dimensional feature matrix of
frequency values, or through an emotional representation model to extract
psychological dimensional features such as valence, arousal, and dominance. Upon
obtaining model-derived features and empirical observations, a common objective is to
quantify the extent to which a specific model explains the variance in behavioral or
neural responses, or to locate brain regions that best fit the model.Where the paper says it
Where the paper says it
Where the paper says it
Limitations and Future Directions
Several limitations include the nonlinear simulations being based on a single sigmoid transformation, which suggests the observed change in relative ordering should be interpreted as evidence for a specific boundary condition rather than as a general characterization of nonlinear data. Future work should examine whether the same pattern generalizes to other monotonic and non-monotonic transformations. p. 1 However, it remains unclear
whether or how the nature of the underlying data and feature characteristics affect the
performance of these methods. Here, we systematically evaluated regression, RSA,
and Pattern Component Modeling (PCM) across distinct data-generating schemes,
including first-order linear mappings and geometry-to-first-order transformations with
either linear or nonlinear sigmoid readouts, using both univariate behavioral and
multivariate fMRI spatial-pattern simulations. By characterizing when regression, RSA, and PCM perform
similarly or differently, this study aims to provide more specific guidance for method
choice under different assumptions about how data are generated. Simulation Study
Method
To evaluate how transforming first-order data into similarity space affects modelselection accuracy under different ground-truth stimulus-response mappings, we
generated simulated datasets under two controlled data-generating schemes. Finally, analyses of empirical datasets
were consistent with the simulation results under approximately linear conditions, with
regression showing clearer model discrimination than RSA. Overall, these findings
suggest that the relative performance of regression, RSA, and PCM depends on the
form of the mapping between features and responses, as well as on the structure of the
feature space.Where the paper says it
Where the paper says it
Where the paper says it
References
Abdel-Ghaffar, S. A., Huth, A. G., Lescroart, M. D., Stansbury, D., Gallant, J. L., & Bishop, S. J. (2024). Occipital-temporal cortical tuning to semantic and affective features of natural images predicts associated behavioral responses. p. 4 , 2011;
Diedrichsen et al. Finally, we cross-validated the simulation findings using a
large-scale lexical/affective empirical behavioral dataset and examined whether
orthogonalizing the predictor space with Principal Component Analysis (PCA) altered
the relative behavior of these methods under multicollinearity (Nunez-Elizalde, 2018;
Oswal et al. This metric has
been widely adopted for comparing statistical models (Cortese & Khanna, 2022;
Snefjella & Kuperman, 2016). For example, a seminal psycholinguistics study used R²
to compare frequency norms predicting lexical decision times, revealing that
frequencies derived from television/film subtitles outperformed those from written
sources (Brysbaert & New, 2009). This metric has
been widely adopted for comparing statistical models (Cortese & Khanna, 2022;
Snefjella & Kuperman, 2016). For example, a seminal psycholinguistics study used R²
to compare frequency norms predicting lexical decision times, revealing that
frequencies derived from television/film subtitles outperformed those from written
sources (Brysbaert & New, 2009). This metric has
been widely adopted for comparing statistical models (Cortese & Khanna, 2022;
Snefjella & Kuperman, 2016). For example, a seminal psycholinguistics study used R²
to compare frequency norms predicting lexical decision times, revealing that
frequencies derived from television/film subtitles outperformed those from written
sources (Brysbaert & New, 2009). These stimuli can subsequently be
processed by quantitative computational models to derive a specific feature vector for
each stimulus. For example, given a series of word stimuli, researchers can pass them
through a word frequency model to obtain a high-dimensional feature matrix of
frequency values, or through an emotional representation model to extract
psychological dimensional features such as valence, arousal, and dominance. was supported by the NIMH
Intramural Research Program (ZICMH002888) of NIH/HHS, USA. Ethics: This research did not require ethical approval because no new human or animal
data were collected. was supported by the NIMH
Intramural Research Program (ZICMH002888) of NIH/HHS, USA. Ethics: This research did not require ethical approval because no new human or animal
data were collected. was supported by the NIMH
Intramural Research Program (ZICMH002888) of NIH/HHS, USA. Ethics: This research did not require ethical approval because no new human or animal
data were collected. These stimuli can subsequently be
processed by quantitative computational models to derive a specific feature vector for
each stimulus. For example, given a series of word stimuli, researchers can pass them
through a word frequency model to obtain a high-dimensional feature matrix of
frequency values, or through an emotional representation model to extract
psychological dimensional features such as valence, arousal, and dominance. , 2019), cross-modal data comparisons (e. ,
M/EEG versus fMRI; Cichy & Oliva, 2020), or cross-species investigations (e. , accuracy, reaction time, or ratings) or multivariate regional brain activity
(e. , multi-voxel fMRI activation patterns). Since adding
parameters to a model inherently inflates R², models with more parameters are typically
penalized for overcomplexity using the adjusted R² index (Fox, 2015). This metric has
been widely adopted for comparing statistical models (Cortese & Khanna, 2022;
Snefjella & Kuperman, 2016). Since adding
parameters to a model inherently inflates R², models with more parameters are typically
penalized for overcomplexity using the adjusted R² index (Fox, 2015). This metric has
been widely adopted for comparing statistical models (Cortese & Khanna, 2022;
Snefjella & Kuperman, 2016). Finally, we cross-validated the simulation findings using a
large-scale lexical/affective empirical behavioral dataset and examined whether
orthogonalizing the predictor space with Principal Component Analysis (PCA) altered
the relative behavior of these methods under multicollinearity (Nunez-Elizalde, 2018;
Oswal et al. By characterizing when regression, RSA, and PCM perform
similarly or differently, this study aims to provide more specific guidance for method
choice under different assumptions about how data are generated. Across these generative schemes, we manipulated key boundary
conditions, including sample size, noise level, feature dimensionality, and
multicollinearity. We further extended the simulations to multivariate fMRI spatial
patterns by incorporating Pattern Component Modeling (PCM) as an additional
approach for evaluating model performance under spatially structured neural
representations (Diedrichsen & Kriegeskorte, 2017; Diedrichsen et al. Across these generative schemes, we manipulated key boundary
conditions, including sample size, noise level, feature dimensionality, and
multicollinearity. We further extended the simulations to multivariate fMRI spatial
patterns by incorporating Pattern Component Modeling (PCM) as an additional
approach for evaluating model performance under spatially structured neural
representations (Diedrichsen & Kriegeskorte, 2017; Diedrichsen et al. Across these generative schemes, we manipulated key boundary
conditions, including sample size, noise level, feature dimensionality, and
multicollinearity. We further extended the simulations to multivariate fMRI spatial
patterns by incorporating Pattern Component Modeling (PCM) as an additional
approach for evaluating model performance under spatially structured neural
representations (Diedrichsen & Kriegeskorte, 2017; Diedrichsen et al. , 2021; Haxby et al. , 2014; Kriegeskorte, Mur, &
Bandettini, 2008; Nili et al. , 2021; Haxby et al. , 2014; Kriegeskorte, Mur, &
Bandettini, 2008; Nili et al. , 2021; Haxby et al. , 2014; Kriegeskorte, Mur, &
Bandettini, 2008; Nili et al. , 2021; Haxby et al. , 2014; Kriegeskorte, Mur, &
Bandettini, 2008; Nili et al. , 2021; Haxby et al. , 2014; Kriegeskorte, Mur, &
Bandettini, 2008; Nili et al. , 2021; Haxby et al. , 2014; Kriegeskorte, Mur, &
Bandettini, 2008; Nili et al. Overall, these findings
suggest that the relative performance of regression, RSA, and PCM depends on the
form of the mapping between features and responses, as well as on the structure of the
feature space. Keywords: Representational similarity analysis, RSA, regression, Pattern
component modeling, model selection Across these generative schemes, we manipulated key boundary
conditions, including sample size, noise level, feature dimensionality, and
multicollinearity. We further extended the simulations to multivariate fMRI spatial
patterns by incorporating Pattern Component Modeling (PCM) as an additional
approach for evaluating model performance under spatially structured neural
representations (Diedrichsen & Kriegeskorte, 2017; Diedrichsen et al. Across these generative schemes, we manipulated key boundary
conditions, including sample size, noise level, feature dimensionality, and
multicollinearity. We further extended the simulations to multivariate fMRI spatial
patterns by incorporating Pattern Component Modeling (PCM) as an additional
approach for evaluating model performance under spatially structured neural
representations (Diedrichsen & Kriegeskorte, 2017; Diedrichsen et al. , fMRI versus EEG), or conduct cross-species investigations. Representational similarity analysis (RSA) is another widely used approach for
comparing computational models in both behavioral and neuroimaging studies
(Edelman, 1998; Freund et al. These stimuli can subsequently be
processed by quantitative computational models to derive a specific feature vector for
each stimulus. For example, given a series of word stimuli, researchers can pass them
through a word frequency model to obtain a high-dimensional feature matrix of
frequency values, or through an emotional representation model to extract
psychological dimensional features such as valence, arousal, and dominance. 5, with irrelevant-feature coefficients fixed at 0. In the geometry-to-first-order
scheme, the same weaker-versus-stronger manipulation was applied to the latent
geometric signal prior to either a linear or a sigmoid readout. , fMRI versus EEG), or conduct cross-species investigations. Representational similarity analysis (RSA) is another widely used approach for
comparing computational models in both behavioral and neuroimaging studies
(Edelman, 1998; Freund et al. cn;
Svetlana V. Shinkareva, shinkareva@sc. edu and Rutvik H. Desai, rutvik@sc. Shinkarevae, f* and Rutvik H. Desaie, f*
a School of Psychology, Nanjing Normal University, Nanjing, China
b Adolescent Education and Intelligence Support Lab of Nanjing Normal University,
Laboratory of Philosophy and Social Sciences at Universities in Jiangsu Province,
Nanjing, China
c The Joint Research Institute for Brain Science and Mental Health, Nanjing Normal
University and Taizhou Fifth People's Hospital, China
d Scientific and Statistical Computing Core, National Institute of Mental Health,
Bethesda, MD, United States
e Department of Psychology, University of South Carolina, Columbia 29208, USA
f Institute for Mind and Brain, University of South Carolina, Columbia 29208, USA
*Correspondence should be addressed to Chuanji Gao, chuanji. These stimuli can subsequently be
processed by quantitative computational models to derive a specific feature vector for
each stimulus. For example, given a series of word stimuli, researchers can pass them
through a word frequency model to obtain a high-dimensional feature matrix of
frequency values, or through an emotional representation model to extract
psychological dimensional features such as valence, arousal, and dominance. These stimuli can subsequently be
processed by quantitative computational models to derive a specific feature vector for
each stimulus. For example, given a series of word stimuli, researchers can pass them
through a word frequency model to obtain a high-dimensional feature matrix of
frequency values, or through an emotional representation model to extract
psychological dimensional features such as valence, arousal, and dominance. Grounded in this
second-order framework, RSA operates under the theoretical premise that meaningful
information in brain or behavioral responses can be effectively characterized by the
latent geometric structures or similarity relationships of stimuli within a feature space,
rather than strictly by a direct linear superposition of features. This abstraction to
second-order similarity structures constitutes RSA’s core strength: it evaluates how well
a model’s predicted representational geometry (based on stimulus features) aligns with
the geometry of empirically observed behavioral or neural responses. Representational similarity analysis (RSA) is another widely used approach for
comparing computational models in both behavioral and neuroimaging studies
(Edelman, 1998; Freund et al. , 2021; Haxby et al. Across these generative schemes, we manipulated key boundary
conditions, including sample size, noise level, feature dimensionality, and
multicollinearity. We further extended the simulations to multivariate fMRI spatial
patterns by incorporating Pattern Component Modeling (PCM) as an additional
approach for evaluating model performance under spatially structured neural
representations (Diedrichsen & Kriegeskorte, 2017; Diedrichsen et al. Across these generative schemes, we manipulated key boundary
conditions, including sample size, noise level, feature dimensionality, and
multicollinearity. We further extended the simulations to multivariate fMRI spatial
patterns by incorporating Pattern Component Modeling (PCM) as an additional
approach for evaluating model performance under spatially structured neural
representations (Diedrichsen & Kriegeskorte, 2017; Diedrichsen et al. However, it remains unclear
whether or how the nature of the underlying data and feature characteristics affect the
performance of these methods. Here, we systematically evaluated regression, RSA,
and Pattern Component Modeling (PCM) across distinct data-generating schemes,
including first-order linear mappings and geometry-to-first-order transformations with
either linear or nonlinear sigmoid readouts, using both univariate behavioral and
multivariate fMRI spatial-pattern simulations. In a typical experimental paradigm, researchers
present participants with a series of stimuli while recording univariate behavioral
indexes (e. , accuracy, reaction time, or ratings) or multivariate regional brain activity
(e. Declaration of Conflicting Interests: None. Funding: The work was supported by National Natural Science Foundation of China
(32300863) and NIH/NIDCD R01DC017162. Abstract
In cognitive psychology and neuroscience, adjudicating between competing theoretical
models is a common methodological challenge. Researchers often rely on either firstorder direct mapping approaches (e. , first-order
isomorphism), RSA quantifies the relationships among response patterns and compares
these to the relationships among stimulus properties (i. , second-order isomorphism)
(Kriegeskorte & Kievit, 2013; Kriegeskorte, Mur, & Bandettini, 2008). Grounded in this
second-order framework, RSA operates under the theoretical premise that meaningful
information in brain or behavioral responses can be effectively characterized by the
latent geometric structures or similarity relationships of stimuli within a feature space,
rather than strictly by a direct linear superposition of features. This abstraction to
second-order similarity structures constitutes RSA’s core strength: it evaluates how well
a model’s predicted representational geometry (based on stimulus features) aligns with
the geometry of empirically observed behavioral or neural responses. , 2021; Haxby et al. , 2014; Kriegeskorte, Mur, &
Bandettini, 2008; Nili et al. , 2021; Haxby et al. , 2014; Kriegeskorte, Mur, &
Bandettini, 2008; Nili et al. Running Head: REGRESSION AND RSA IN MODEL SELECTION
Model Selection with Regression and Representational
Similarity Analysis for Linear and Nonlinear Data
Chuanji Gaoa,b,c*
, Gang Chend, Svetlana V. Shinkarevae, f* and Rutvik H. ,
M/EEG versus fMRI; Cichy & Oliva, 2020), or cross-species investigations (e. ,
monkey versus human; Kriegeskorte, Mur, Ruff, et al. These stimuli can subsequently be
processed by quantitative computational models to derive a specific feature vector for
each stimulus. For example, given a series of word stimuli, researchers can pass them
through a word frequency model to obtain a high-dimensional feature matrix of
frequency values, or through an emotional representation model to extract
psychological dimensional features such as valence, arousal, and dominance. By directly optimizing
these parameters from the raw data, regression can make efficient use of first-order
variance and can provide high statistical sensitivity when its assumptions are met. The
resulting errors or goodness-of-fit indices from different models can then be compared
(Maxwell et al. Running Head: REGRESSION AND RSA IN MODEL SELECTION
Model Selection with Regression and Representational
Similarity Analysis for Linear and Nonlinear Data
Chuanji Gaoa,b,c*
, Gang Chend, Svetlana V. Shinkarevae, f* and Rutvik H. However, it remains unclear
whether or how the nature of the underlying data and feature characteristics affect the
performance of these methods. Here, we systematically evaluated regression, RSA,
and Pattern Component Modeling (PCM) across distinct data-generating schemes,
including first-order linear mappings and geometry-to-first-order transformations with
either linear or nonlinear sigmoid readouts, using both univariate behavioral and
multivariate fMRI spatial-pattern simulations. Running Head: REGRESSION AND RSA IN MODEL SELECTION
Model Selection with Regression and Representational
Similarity Analysis for Linear and Nonlinear Data
Chuanji Gaoa,b,c*
, Gang Chend, Svetlana V. Shinkarevae, f* and Rutvik H. , 2011;
Diedrichsen et al. Finally, we cross-validated the simulation findings using a
large-scale lexical/affective empirical behavioral dataset and examined whether
orthogonalizing the predictor space with Principal Component Analysis (PCA) altered
the relative behavior of these methods under multicollinearity (Nunez-Elizalde, 2018;
Oswal et al. For example, given a series of word stimuli, researchers can pass them
through a word frequency model to obtain a high-dimensional feature matrix of
frequency values, or through an emotional representation model to extract
psychological dimensional features such as valence, arousal, and dominance. Upon
obtaining model-derived features and empirical observations, a common objective is to
quantify the extent to which a specific model explains the variance in behavioral or
neural responses, or to locate brain regions that best fit the model. , accuracy, reaction time, or ratings) or multivariate regional brain activity
(e. , multi-voxel fMRI activation patterns). Grounded in this
second-order framework, RSA operates under the theoretical premise that meaningful
information in brain or behavioral responses can be effectively characterized by the
latent geometric structures or similarity relationships of stimuli within a feature space,
rather than strictly by a direct linear superposition of features. This abstraction to
second-order similarity structures constitutes RSA’s core strength: it evaluates how well
a model’s predicted representational geometry (based on stimulus features) aligns with
the geometry of empirically observed behavioral or neural responses. However, it remains unclear
whether or how the nature of the underlying data and feature characteristics affect the
performance of these methods. Here, we systematically evaluated regression, RSA,
and Pattern Component Modeling (PCM) across distinct data-generating schemes,
including first-order linear mappings and geometry-to-first-order transformations with
either linear or nonlinear sigmoid readouts, using both univariate behavioral and
multivariate fMRI spatial-pattern simulations. Running Head: REGRESSION AND RSA IN MODEL SELECTION
Model Selection with Regression and Representational
Similarity Analysis for Linear and Nonlinear Data
Chuanji Gaoa,b,c*
, Gang Chend, Svetlana V. Shinkarevae, f* and Rutvik H. Shinkarevae, f* and Rutvik H. Desaie, f*
a School of Psychology, Nanjing Normal University, Nanjing, China
b Adolescent Education and Intelligence Support Lab of Nanjing Normal University,
Laboratory of Philosophy and Social Sciences at Universities in Jiangsu Province,
Nanjing, China
c The Joint Research Institute for Brain Science and Mental Health, Nanjing Normal
University and Taizhou Fifth People's Hospital, China
d Scientific and Statistical Computing Core, National Institute of Mental Health,
Bethesda, MD, United States
e Department of Psychology, University of South Carolina, Columbia 29208, USA
f Institute for Mind and Brain, University of South Carolina, Columbia 29208, USA
*Correspondence should be addressed to Chuanji Gao, chuanji. The positive square root of R² yields the multiple correlation
coefficient (R), equivalent to the simple correlation between observed values (y) and
predicted values (ŷ). Geometrically, R represents the cosine of the angle θ between the
mean-centered vectors of y and ŷ (Rencher & Schaalje, 2008). The positive square root of R² yields the multiple correlation
coefficient (R), equivalent to the simple correlation between observed values (y) and
predicted values (ŷ). Geometrically, R represents the cosine of the angle θ between the
mean-centered vectors of y and ŷ (Rencher & Schaalje, 2008). Accurate model selection
among these alternatives is therefore important for quantitative research in cognitive
psychology and neuroscience. One common approach for evaluating the effects of competing models on a response
variable is linear regression. , Groen et al. , 2018; Riberto et al. For example, given a series of word stimuli, researchers can pass them
through a word frequency model to obtain a high-dimensional feature matrix of
frequency values, or through an emotional representation model to extract
psychological dimensional features such as valence, arousal, and dominance. Upon
obtaining model-derived features and empirical observations, a common objective is to
quantify the extent to which a specific model explains the variance in behavioral or
neural responses, or to locate brain regions that best fit the model. In a typical experimental paradigm, researchers
present participants with a series of stimuli while recording univariate behavioral
indexes (e. , accuracy, reaction time, or ratings) or multivariate regional brain activity
(e. Since adding
parameters to a model inherently inflates R², models with more parameters are typically
penalized for overcomplexity using the adjusted R² index (Fox, 2015). This metric has
been widely adopted for comparing statistical models (Cortese & Khanna, 2022;
Snefjella & Kuperman, 2016). Since adding
parameters to a model inherently inflates R², models with more parameters are typically
penalized for overcomplexity using the adjusted R² index (Fox, 2015). This metric has
been widely adopted for comparing statistical models (Cortese & Khanna, 2022;
Snefjella & Kuperman, 2016). These stimuli can subsequently be
processed by quantitative computational models to derive a specific feature vector for
each stimulus. For example, given a series of word stimuli, researchers can pass them
through a word frequency model to obtain a high-dimensional feature matrix of
frequency values, or through an emotional representation model to extract
psychological dimensional features such as valence, arousal, and dominance. However, it remains unclear
whether or how the nature of the underlying data and feature characteristics affect the
performance of these methods. Here, we systematically evaluated regression, RSA,
and Pattern Component Modeling (PCM) across distinct data-generating schemes,
including first-order linear mappings and geometry-to-first-order transformations with
either linear or nonlinear sigmoid readouts, using both univariate behavioral and
multivariate fMRI spatial-pattern simulations. This makes RSA
particularly useful when the data lack a consistent one-to-one correspondence or share
primarily relational structure, such as across subject-specific regions of interest with
varying voxel counts (e. , Tsantani et al. , 2021; Haxby et al. , 2014; Kriegeskorte, Mur, &
Bandettini, 2008; Nili et al. , 2021; Haxby et al. , 2014; Kriegeskorte, Mur, &
Bandettini, 2008; Nili et al. For example, given a series of word stimuli, researchers can pass them
through a word frequency model to obtain a high-dimensional feature matrix of
frequency values, or through an emotional representation model to extract
psychological dimensional features such as valence, arousal, and dominance. Upon
obtaining model-derived features and empirical observations, a common objective is to
quantify the extent to which a specific model explains the variance in behavioral or
neural responses, or to locate brain regions that best fit the model. Running Head: REGRESSION AND RSA IN MODEL SELECTION
Model Selection with Regression and Representational
Similarity Analysis for Linear and Nonlinear Data
Chuanji Gaoa,b,c*
, Gang Chend, Svetlana V. Shinkarevae, f* and Rutvik H. This metric has
been widely adopted for comparing statistical models (Cortese & Khanna, 2022;
Snefjella & Kuperman, 2016). For example, a seminal psycholinguistics study used R²
to compare frequency norms predicting lexical decision times, revealing that
frequencies derived from television/film subtitles outperformed those from written
sources (Brysbaert & New, 2009). This metric has
been widely adopted for comparing statistical models (Cortese & Khanna, 2022;
Snefjella & Kuperman, 2016). For example, a seminal psycholinguistics study used R²
to compare frequency norms predicting lexical decision times, revealing that
frequencies derived from television/film subtitles outperformed those from written
sources (Brysbaert & New, 2009). These stimuli can subsequently be
processed by quantitative computational models to derive a specific feature vector for
each stimulus. For example, given a series of word stimuli, researchers can pass them
through a word frequency model to obtain a high-dimensional feature matrix of
frequency values, or through an emotional representation model to extract
psychological dimensional features such as valence, arousal, and dominance. Across these generative schemes, we manipulated key boundary
conditions, including sample size, noise level, feature dimensionality, and
multicollinearity. We further extended the simulations to multivariate fMRI spatial
patterns by incorporating Pattern Component Modeling (PCM) as an additional
approach for evaluating model performance under spatially structured neural
representations (Diedrichsen & Kriegeskorte, 2017; Diedrichsen et al. , accuracy, reaction time, or ratings) or multivariate regional brain activity
(e. , multi-voxel fMRI activation patterns). Since adding
parameters to a model inherently inflates R², models with more parameters are typically
penalized for overcomplexity using the adjusted R² index (Fox, 2015). This metric has
been widely adopted for comparing statistical models (Cortese & Khanna, 2022;
Snefjella & Kuperman, 2016). These stimuli can subsequently be
processed by quantitative computational models to derive a specific feature vector for
each stimulus. For example, given a series of word stimuli, researchers can pass them
through a word frequency model to obtain a high-dimensional feature matrix of
frequency values, or through an emotional representation model to extract
psychological dimensional features such as valence, arousal, and dominance. , 2014; Popal et al. , 2019; Weaverdyck et al. , 2011;
Diedrichsen et al. Finally, we cross-validated the simulation findings using a
large-scale lexical/affective empirical behavioral dataset and examined whether
orthogonalizing the predictor space with Principal Component Analysis (PCA) altered
the relative behavior of these methods under multicollinearity (Nunez-Elizalde, 2018;
Oswal et al. Shinkarevae, f* and Rutvik H. Desaie, f*
a School of Psychology, Nanjing Normal University, Nanjing, China
b Adolescent Education and Intelligence Support Lab of Nanjing Normal University,
Laboratory of Philosophy and Social Sciences at Universities in Jiangsu Province,
Nanjing, China
c The Joint Research Institute for Brain Science and Mental Health, Nanjing Normal
University and Taizhou Fifth People's Hospital, China
d Scientific and Statistical Computing Core, National Institute of Mental Health,
Bethesda, MD, United States
e Department of Psychology, University of South Carolina, Columbia 29208, USA
f Institute for Mind and Brain, University of South Carolina, Columbia 29208, USA
*Correspondence should be addressed to Chuanji Gao, chuanji.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
--- Page 34 ---
Gimenes, M., & New, B. (2016). Worldlex: Twitter and blog word frequencies for 66 languages. Groen, I. I., Greene, M. R., Baldassano, C., Fei-Fei, L., Beck, D. M., & Baker, C. I. (2511.00395>.
--- Page 28 ---
Figure 11 shows the empirical behavioral data: regression shows greater model separability than RSA. Applying PCA to the predictor matrices improved RSA’s ability to differentiate the two models, particularly at smaller sample sizes. p. 9 These intervals represent the range of values within one
standard deviation 𝑆𝑆𝑆𝑆 of the mean 𝑀𝑀, offering an intuitive measure to assess the
overlap and separability of the distributions for the two models. Second, we computed
an effect size measure, Cohen’s d, to quantify the standardized difference between the
sampling distributions of the larger-effect and smaller-effect models for each method
(RSA or regression). These intervals represent the range of values within one
standard deviation 𝑆𝑆𝑆𝑆 of the mean 𝑀𝑀, offering an intuitive measure to assess the
overlap and separability of the distributions for the two models. Second, we computed
an effect size measure, Cohen’s d, to quantify the standardized difference between the
sampling distributions of the larger-effect and smaller-effect models for each method
(RSA or regression).Where the paper says it
Where the paper says it
--- Page 34 ---
Figure 11 shows the empirical behavioral data: regression shows greater model separability than RSA whereas PCA improves RSA discriminability.
--- Page 28 ---
Based on previous literature, the frequency-composite model was expected to show a stronger association with visual lexical decision time than the affect-composite model. Across repeated random subsets, this difference was evident under both regression and RSA: the frequency-composite model yielded higher adjusted R2 in regression and stronger DSM correspondence in RSA. p. 9 To obtain RSA
scores, we vectorized the lower triangle of each DSM and computed the association
between X and Y DSM vectors using Spearman’s rank correlation and Pearson
correlation. For regression, we fitted an ordinary least squares model predicting Y from X and
summarized model fit using adjusted R² that is given by:
𝑅𝑅𝑎𝑎𝑎𝑎
2 = 1 − (1 − 𝑅𝑅2)(𝑛𝑛 − 1)
𝑛𝑛 − 𝑝𝑝 − 1 ,
where n is the number of observations, p is the number of predictors, 𝑅𝑅2 is the
coefficient of determination. We also found that
feature multicollinearity affected these methods differently across generative schemes,
and that orthogonalizing the predictor space via principal component analysis (PCA)
reduced several collinearity-related differences. Finally, analyses of empirical datasets
were consistent with the simulation results under approximately linear conditions, with
regression showing clearer model discrimination than RSA.Where the paper says it
Where the paper says it
--- Page 34 ---
Despite this broad agreement in model ranking, RSA DSM correspondence values remained comparatively small even in settings where regression achieved moderate-to-high adjusted R2. This pattern is consistent with the simulation results and suggests that second-order DSM alignment captures only part of the information available in direct first-order feature-to-response mappings. p. 1 Here, we systematically evaluated regression, RSA,
and Pattern Component Modeling (PCM) across distinct data-generating schemes,
including first-order linear mappings and geometry-to-first-order transformations with
either linear or nonlinear sigmoid readouts, using both univariate behavioral and
multivariate fMRI spatial-pattern simulations. Our results suggest that the relative
performance of these methods depends on the underlying generative mechanism. These intervals represent the range of values within one
standard deviation 𝑆𝑆𝑆𝑆 of the mean 𝑀𝑀, offering an intuitive measure to assess the
overlap and separability of the distributions for the two models. Second, we computed
an effect size measure, Cohen’s d, to quantify the standardized difference between the
sampling distributions of the larger-effect and smaller-effect models for each method
(RSA or regression).Where the paper says it
Where the paper says it
--- Page 34 ---
When n was low to moderate (approximately 50–200 items), regression more consistently distinguished the frequency-composite model from the affect-composite model, whereas RSA showed weaker and more variable separation. Applying PCA to the predictor matrices improved RSA’s ability to differentiate the two models, particularly at smaller sample sizes. Overall, these empirical results are consistent with the simulation findings: regression retained an advantage for model selection in first-order prediction, whereas PCA altered the predictor representation in a way that improved RSA discriminability when predictors were highly correlated. p. 1 Finally, analyses of empirical datasets
were consistent with the simulation results under approximately linear conditions, with
regression showing clearer model discrimination than RSA. Overall, these findings
suggest that the relative performance of regression, RSA, and PCM depends on the
form of the mapping between features and responses, as well as on the structure of the
feature space.Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Regression-based model selection can be prioritized when responses arise from a direct linear mapping from features, as
regression consistently showed stronger discrimination than RSA
under first-order linear schemes. -
Rank-based Representational Similarity Analysis (RSA) should be favored when the underlying mapping is nonlinear but preserves ordinal structure, as this condition leads to
rank-based RSA showing an advantage over regression and PCM.
-
When evaluating multivoxel fMRI data generated under a linear condition, Pattern Component Modeling (PCM) and regression are shown to show the
strongest performance,
whereas RSA performs less well. -
When the sigmoid readout is used for geometry-to-first-order transformations,
rank-based RSA often performed better than regression and Pearson based RSA,
suggesting that rank-based second order comparison can be more informative than linear regression or Pearson based RSA in such cases. -
To improve model selection under feature space distortions, applying Principal Component Analysis (PCA) to the predictor space can
attenuate the collinearity driven decline in RSA
in first-order schemes andremove the apparent RSA benefit associated with higher collinearity
in geometry-to-first-order linear readout schemes. -
In empirical applications involving highly correlated predictors, PCA can be used as a sensitivity analysis to assess whether observed results
depend strongly on the original feature representation,
by improving RSA's ability to differentiate models. -
For multivoxel data under sigmoid readout conditions, the relative ordering of methods can shift, with
Spearman's rho often performing better than regression and Pearson based RSA
at larger sample sizes compared to linear mapping conditions. -
The system can be configured to use PCA as a sensitivity analysis tool to reveal whether apparent method differences
depend strongly on the original feature representation rather than uniformly improving RSA performance across conditions.
Abstract
Selecting among competing computational models is central to cognitive psychology and neuroscience, yet the relative model-selection performance of regression and representational similarity analysis (RSA) remains unclear. We compared these approaches using Monte Carlo simulations and empirical data. Univariate simulations compared generating and rival feature models against the same response under direct-linear and geometry-based mappings. Under direct-linear generation, regression outperformed correlation-distance RSA/Spearman but not RSA/Pearson. Under geometry-linear generation, neither approach showed a significant advantage, whereas correlation-distance RSA outperformed regression under a sigmoid readout, although this advantage did not extend to Euclidean-distance RSA. Feature dimensionality, covariance, and PCA preprocessing further influenced model-selection accuracy. In multivoxel simulations, pattern component modeling (PCM) showed superior performance under spatially correlated noise, while crossnobis improved selection relative to squared Euclidean RSA under the same condition. Feature reweighting improved held-out representational correspondence for both generating and rival models without consistently improving model selection. Separate signal-strength comparisons yielded different method rankings. In empirical data from 4,026 words, all methods preferred frequency over affect features, but small-sample preference stability depended on distance measures and preprocessing. Together, these findings demonstrate that model-selection performance depends on stimulus-response mappings, representational geometry, measurement noise, and inferential objectives.
Related papers
- Learning Robust Treatment Rules for Censored Data
- Neural Estimation of Summary Statistics for Simulation-Based Inference with Non-Asymptotic Guarantees
- Spectral divide-and-conquer MCMC for long stationary time series
- Improving the Efficiency of Subgroup Analysis in Randomized Controlled Trials with TMLE
- Personalized Federated Vector Autoregression with Personalization Diversity
- Doubly Robust and Efficient Calibration of Prediction Sets for Right-Censored Time-to-Event Outcomes