CrossMean: Model Merging by Pairing Merged and Expert Activations
The authors introduce CrossMean, a model merging method that improves upon existing closed-form techniques by pairing activations from the merged network and individual experts using a new bilinear statistic. This approach outperforms previous methods empirically across various vision backbones and BERT encoders, suggesting that combining merged and expert data yields better results.