The Ball and the Box: Two Geometries of Computation in Superposition
The paper investigates how much dimension is needed for neural representations to compute Boolean gates using superposition. It distinguishes between two error criteria: expected counts (ball geometry) and joint reliability (box geometry). By analyzing these geometries, the study provides precise mathematical bounds on the required layer dimension for reliable computation.