SYNAPSE: Framework for Neuron Analysis and Perturbation in Sequence Encoding
SYNAPSE is a training-free framework designed to systematically analyze and stress-test Transformer models by examining their internal representations. It extracts per-layer [CLS] activations, uses a linear probe to rank neurons, and applies controlled inference-time interventions via forward hooks. This allows researchers to understand how different parts of the model contribute to specific tasks without changing the original model weights. The findings show that task information is distributed broadly across neurons rather than concentrated in isolated units.