How Optimality Structures Sparse Dictionaries: Theory for Interpreting SAE Representations
This research analyzes sparse autoencoders by studying dictionary learning objectives. It derives necessary geometric constraints that explain how features relate to one another, leading to insights into hierarchical splitting and feature absorption observed in neural representations. The study also constructs a novel convex problem revealing limits on the number of atoms per datapoint.