Mechanistic Evidence for Spectral Structures in Prior-Data Fitted Networks
The research investigates if PFN latents encode spectral structure from context and if this structure can be extracted as an explicit kernel. It found that spectral information is linearly decodable from residual streams, and a Filter Bank Decoder converts these representations into explicit stationary kernels usable for Gaussian process regression, offering competitive performance at a fraction of the cost.