Functional Subspace, where language models can use vector algebra to solve problems
The study hypothesizes that large language models (LLMs) use subspaces and vector algebra within them to solve complex tasks like in-context learning (ICL). By analyzing functional modules, researchers found that LLMs create subspaces where evidence accumulates. ICL tasks can then be solved by performing simple algebraic operations within these identified low-dimensional spaces.