SAE++: Cascaded Sparse Autoencoders Learn Multi-Level Visual Concepts in Multimodal LLMs
Cascaded Sparse Autoencoders (CSAEs) learn hierarchical visual concepts in Multimodal Large Language Models by training a second-level SAE directly on the decoder weights of the first level. This approach creates 'concepts of concepts,' improving coherence and enabling effective group-level interventions in MLLM outputs compared to existing methods.