Perfect Reconstruction Cannot Show Whether Two Codebooks on a VAE's Latent Space Agree
The paper investigates whether standard training reports reveal if two codebooks used for encoding and decoding data in a Variational Autoencoder (VAE) actually agree on the latent space representations. It constructs VAEs where agreement can differ based on inference versus generation, proving that agreement is not always visible in training metrics. A simple 'Certify-or-Measure' protocol is proposed to detect broken codebooks.