TensorDex: A Compact, Tensor-Centric Storage System for Modern AI Models
TensorDex is a new storage system that compresses large AI models by focusing on individual tensors instead of entire models. It uses a fingerprint to predict how compressible pairs of tensors are and clusters them into groups. This approach exploits hidden redundancy between different parts of different models, resulting in up to 70.5% storage reduction compared to current methods.