Ordinary Least Squares as an Attention Mechanism
Ordinary Least Squares (OLS) predictions are reformulated as a similarity-based method within an orthonormal space, mirroring attention mechanisms in large language models. OLS is viewed as optimizing input encoding and decoding rather than estimating coefficients. The prediction becomes a weighted average based on inner products between factors, demonstrating that OLS is fundamentally a form of scaled attention.