Language Models Without a Trainable Input Embedding Table: Learning from Fixed Minimal Binary Token Codes
Researchers tested if trainable input embedding tables are necessary for language models by replacing them with fixed minimal binary codes derived from token IDs. They found that using these deterministic codes, which preserve exact token identity, yielded performance competitive with the standard setup, eliminating millions of trainable parameters. This suggests that a stable symbolic interface is sufficient for learning effective internal representations.