Defense via Consensus: Mitigating Subliminal Learning through Multi-Teacher Distillation
The system uses consensus decoding to defend against source-specific misbehavior in multi-teacher distillation settings. It aggregates next-token distributions from multiple fine-tuned reference models to suppress costs unique to individual sources while preserving shared, desirable behaviors across the ensemble.