Back into Plato's Cave: Examining Cross-modal Representational Convergence at Scale
The study tested if neural networks trained on different modalities, like images and text, converge to a single representation of reality as they scale. Findings show that cross-modal alignment is fragile and depends heavily on the evaluation method. Models may learn rich, but distinct, representations rather than identical ones.