Perfect Reconstruction Cannot Show Whether Two Codebooks on a VAE's Latent Space Agree

The gist: No item of a standard training report can reveal whether two codebooks on a VAE’s latent space agree on encoded data, and agreement can drift when the model generates from its prior

Invisibility of Agreement

The paper proves that no item of a standard training report can reveal whether the written and read codes agree on encoded data A variational autoencoder (VAE) is the natural place to ask The pipeline holds two codebooks on one space, and their agreement can change when the model generates Every item of the marginal package matches — agreement is 1 versus 0

The Role of Budgets in Certification

The variational gap and the rate bound how far a measured agreement can move under exact inference and under generation The gap bounds the distance between Aq and its value under exact inference, and the rate bounds the distance between Aq and Aprior A budget B allows no interval narrower than 1 − e − B For Gaussian posteriors, a mean-field encoder pays an exact price in the gap that depends on its choice of axes On probabilistic PCA, the gap certifies a trained encoder in all 75 settings we test, at agreement between 0.76 and 0.97

The Certify-or-Measure Protocol

The paper introduces a simple protocol, Certify-or-Measure, for any pipeline that reads a VAE as codes The protocol is sound in both branches by Theorem 2(b) and a union bound over the two Monte Carlo failure events The certify branch by Theorem 2(b) and part (b) is data processing plus the rate decomposition Its abstention region is a theorem, not an inconvenience

Bounds on Certification

Theorem 3 gives the exact price that a mean-field encoder, which works coordinate by coordinate, pays in the gap for Gaussian posteriors The floor dictates that a latent cannot carry more than log N nats about the N examples over which the rate is averaged On probabilistic PCA, where every quantity is exact, the gap certifies a trained mean-field encoder in all 75 cells of a sweep over latent dimension at agreement between 0.76 and 0.97

Experimental Findings

The audit detects four failure modes in the models below Generative drift shows Aq = 0.770 → Aprior = 0.525 on MNIST with a task probe and a label-free prototype control falling in step (both scored by the same external classifier, so the second is a sensitivity analysis rather than independent validation) The dataset floor predicts which models cannot be certified On six FashionMNIST models, the measured ˆI sits within 0.01 nats of log N, so DKL(¯q∥p) = R¯ − log N to that accuracy

Conclusion

A clean training report does not show that the code a pipeline writes is the code it reads Log the joint code table, and certify from the ELBO’s own terms when their budgets are small enough The practical rule is short: log the joint code table in the pipeline’s own indices, report agreement with its code map and a mean-field budget with its axes, and use Certify-or-Measure to decide whether to certify or to measure

How it works

The paper constructs two VAEs that reconstruct every input perfectly, infer exactly, and match on every per-example ELBO and every code histogram, yet agreement is one in the first and zero in the second The joint table of written against read codes shows the difference at once

The Price of Certification

Theorem 3(e) forbids certification in advance for 66 models, 0 certify The optimal mean-field Gaussian fit in (a) is classical (Bishop, 2006); what is new is reading its value as a price paid in Q and pairing it with the invariance statement (b)

The Protocol

The protocol fires where the theory says it can, and none fires where the dataset floor rules it out The protocol certifies broken codebooks, not working ones

Experimental Details and Disclosures

The audit detects four failure modes in the models below What the audit detects. Four failure modes occur in the models below, each named by the reading that found it The protocol is run on the rate side for a licensing reason The frontier, at scale, at d ≥ 5 and R >¯ log N + log 2 forbids certification

The Final Verdict

The protocol certifies broken codebooks, not working ones Overall 131 of the 240 configurations certify, against 1 of the 6 SVHN models the licensed budget was re-audited on above The frontier is at scale and w∗ was not tuned under any outcome

How it works

The protocol fires where the channel is most degenerate—two-percent agreement, where B∗ is loosest The two routes differ; the destination does not The certificate fires 75/75 against 0/75 for a trained linear encoder, with widths growing 0.106 → 0.487 and crossing w∗ just above d = 32 The protocol certifies broken codebooks, not working ones

The Final Verdict

The protocol certifies broken codebooks, not working ones Overall 131 of the 240 configurations certify, against 1 of the 6 SVHN models the licensed budget was re-audited on above The frontier is at scale and w∗ was not tuned under any outcome The protocol certifies broken codebooks, not working ones

How it works

The protocol fires where the channel is most degenerate—two-percent agreement, where B∗ is loosest The two routes differ; the destination does not The certificate fires 75/75 against 0/75 for a trained linear encoder, with widths growing 0.106 → 0.

1. Confidence in latent space interpretation: Implement a protocol where systems log the joint code table in the pipeline’s own indices, and certify from the ELBO’s own terms when their budgets are small enough, as this is shown to be a robust method to detect if the written and read codes agree on encoded data even when standard reports fail.

2. Budget-based certification: Develop a Certify-or-Measure protocol where systems check if the budget B is small enough, stating that a budget B allows no band narrower than 1 − e − B, to decide whether to certify: report U, which contains Aprior with probability ≥ 1 − δB − δq or abstain and measure.

3. Rate-side certification licensing: Establish a rule where systems check the rate budget against the bound, noting that the dataset floor rules out rate-side certification in advance for 66 models, to prevent attempting to certify when "R >¯ log N + log 2."

4. Map-sensitivity awareness: Require systems to report agreement with its code map because a change of code map moves the quantity and not the budget, specifically noting that on a given checkpoint, the agreement is reported with its code map because the declared map’s majority labels cover a strict subset.

5. Model-specific certificate pricing: For Gaussian posteriors, implement pricing based on the mean-field computation where the variational gap bounds the distance between Aq and its value under exact inference, using the coordinate price that depends on its choice of axes to determine if certification is feasible.

Many systems read the latent space of a variational autoencoder (VAE) as discrete codes, and rely on each latent being read back as the code it was written as. Shannon (1948) took such a shared codebook as given; inside a VAE nothing enforces it. We prove that no item of a standard training report can reveal whether the written and read codes agree on encoded data. We construct two VAEs that reconstruct every input perfectly, infer exactly, and match on every per-example ELBO and every code histogram, yet agreement is one in the first and zero in the second. We then show that the ELBO's own terms bound how far a measured agreement can move: the variational gap bounds the shift to exact inference, and the rate bounds the shift to generation from the prior. For Gaussian posteriors, a mean-field encoder pays an exact price in the gap that depends on its choice of axes. On probabilistic PCA the gap certifies a trained encoder in all 75 settings we test, at agreement between 0.76 and 0.97. On 240 trained VAEs, a floor set by the rate and the dataset size rules out rate-side certification in advance for 66 models, and none of them certifies. The results yield a simple protocol, Certify-or-Measure, for any pipeline that reads a VAE as codes.