Word-Class and Construction-Like Structure Emerges in Neural Successor Representations Trained on Natural Language
Researchers trained Successor Representations (SRs) to predict future word distributions across multiple time steps instead of just the next token. This unsupervised training caused the SR embeddings to spontaneously organize into a geometry mirroring language's hierarchical structure, revealing categories like nouns, verbs, and adjectives without any explicit linguistic supervision.