Compute Allocation in Bilinear Contrastive Representation Learning via Scaling Laws
The study develops a theoretical scaling law for bilinear contrastive learning under sketched linear conditions. It decomposes model risk into irreducible risk, approximation error, and noise terms related to gradient descent bias and variance. This provides a framework for understanding how model size, data, and training time interact to optimize resource allocation.