World Properties without World Models: Distributional Associations and the Interpretation of Decoding Results from Language Models
Researchers tested if static word embeddings (like GloVe) preserve hidden spatial and temporal structures from text. They found that geographic data like latitude and temperature are linearly predictable from these vectors, proving language already encodes a compressed imprint of geography and history. This suggests simple word statistics retain richer structure than previously thought.