TernaryLLM: Ternarized Large Language Model
TernaryLLM addresses high costs of Large Language Models by using Ternarization, an extreme quantization method that replaces multiplications with additions. It introduces Dual Learnable Ternarization to handle weight outliers and Outlier-Friendly Feature Knowledge Distillation to recover lost semantic information. This results in a model that surpasses previous low-bit methods on benchmarks like LLaMA-3.