Learning Cross-Model Activation Alignments with Explicit Many-to-Many Layer Maps

As a fastidious researcher, I have meticulously reviewed both provided texts concerning the paper Learning Cross-Model Activation Alignments with Explicit Many-to-Many Layer Maps. The two summaries offer complementary perspectives: one focusing on the practical application (MATCHA) and its empirical findings, and the other delving into the underlying theoretical guarantees regarding layer mapping.

My synthesized summary will integrate these aspects to provide a comprehensive, high-fidelity understanding of the research.

Comprehensive Research Synthesis: Learning Cross-Model Activation Alignments with Explicit Many-to-Many Layer Maps (MATCHA & Theoretical Foundations)

This research addresses the fundamental question of how activations from two independently trained Large Language Models (LLMs) relate across different layers and architectures, moving beyond simplistic, fixed one-to-one correspondences. The study introduces MATCHA, a novel method designed to learn an explicit, input-dependent mapping between the layer activations of a source model and a target model. This learning framework is built upon rigorous theoretical propositions that characterize the nature of these learned alignments—whether they are restricted to simple one-to-one mappings or encompass complex many-to-many relationships.

I. The Practical Method: MATCHA for Activation Alignment

The core contribution in practice is the MATCHA method, which recasts activation alignment as a learning problem using a lightweight network to predict the mapping directly from input data. MATCHA factorizes this alignment into two distinct components:

1. Layer Map Function (alpha): This component learns the explicit correspondence between source and target layers. It is constructed via cross-attention, where learned, input-independent queries (associated with target layers) interact with input-dependent keys derived from the source activations. This results in a target-layer times source-layer (Lt times Ls) correspondence matrix that dictates how source representations are combined to reconstruct each target layer on a per-input basis.

2. Feature Map: This component handles the transformation of the mixed representations from the source model's hidden space into the target model's space. It employs a shared projection matrix (W) followed by a residual feed-forward network (FF3).

Key Empirical Findings and Applications:

• Superior Reconstruction and Retrieval: Across 42 diverse pairs of models spanning three families (1B to 8B parameters) and various scales, MATCHA consistently reconstructs target activations more faithfully than all baselines. It achieves stronger performance across multiple retrieval metrics (Recall@1, Precision@1, Jaccard@5). The largest gains are observed when transferring between models from different families or with different depths.

• Layer Correspondence Nature: Crucially, MATCHA reveals that cross-model layer correspondences are not one-to-one. Instead, they follow an approximately monotone correspondence, meaning each target layer draws upon a band of neighboring source layers. The study demonstrates that restricting the map to a single source layer per target (one-to-one alignment) significantly degrades performance, whereas the full multi-layer map outperforms all such restricted variants.

• Behavioral Teleportation: Beyond reconstruction, MATCHA enables behavioral teleportation. A probe (for hallucination detection) and a steering intervention developed for a target model can be transferred to a source model with better transfer performance than any baseline across four different datasets.

• Efficiency: The latency associated with transferring between activation spaces is extremely low, approximately 0.06 milliseconds, making the method practically viable for real-time use.

• Transferability of Interventions: MATCHA provides a practical framework for reusing activation-based tools (steering vectors and probes) across models without retraining them for each new target model.

Limitations: The primary limitations involve the necessity of accessing internal activations/hidden states and the requirement to train a separate activation map for every model pair due to differing activation spaces between models.

II. Theoretical Guarantees on Layer Mapping

The theoretical analysis provides the mathematical underpinning for why MATCHA's learned maps are effective and what their structural properties imply. The research establishes formal propositions regarding the realizability of different types of layer correspondences:

• One-to-One Maps: The study proves that every one-to-one map is approximable to arbitrary accuracy at a specific attention width (d=2), provided a certain assumption holds (Assumption 1: Linearly decodable layer identity). This means simple, fixed correspondences (like relative depth or terminal) are contained within the closure of learnable maps.

• Soft Many-to-Many Maps: The paper investigates the interior of the set of row-stochastic matrices—the many-to-many correspondences where a target layer draws on several source layers simultaneously. It proves that these soft correspondences can be realized exactly and input-independently (Proposition 4) if a stronger condition (Assumption 2: Exactly decodable layer identity) is met, provided the attention width d is sufficiently large (d (Lt, Ls)).

• Logit Margin Bound: Lemma 1 quantifies the error incurred when approximating one-to-one correspondences based on the logit margin—the difference in scores between the designated source layer and its nearest neighbor.

III. Synthesis and Conclusion

The research presents a powerful synergy between empirical discovery and theoretical rigor. MATCHA successfully learns input-dependent, many-to-many layer maps that are inherently monotone (target layers draw from neighboring source layers), which is superior to the prior assumption of one-to-one mapping. The theoretical work confirms that while simple one-to-one mappings are approximable, the method's success lies in its ability to capture the richer, multi-source dependencies inherent in cross-model relationships. Ultimately, MATCHA provides a practical mechanism for transferring activation tools—reconstruction knowledge and behavioral interventions—across disparate LLM architectures with high fidelity and minimal latency.

1. Bold layer-wise activation reconstruction for cross-model transfer: MATCHA reconstructs target’s activations more faithfully and can be used for transfer of activation-space interventions, allowing applications developed for one model to be reused with another without retraining.

2. Explicit, input-dependent layer mapping: The method learns an interpretable, input-dependent mapping between layers where each target layer is associated with a learned query and source layers act as keys to determine how representations are combined on a per-input basis.

3. Transfer of control mechanisms: Steering vectors and probes developed for one model can be transferred to another, enabling the transfer of activation-space interventions, allowing developers to use tools like steering vectors and probes developed for one model in a new context without retraining.

4. Cross-model structural insight: The learned layer maps reveal structural relationships, specifically showing that the resulting maps are not one-to-one: they follow an approximately monotone correspondence, while each target layer draws on a band of neighboring source layers.

5. Performance gains in model transfer: MATCHA achieves substantial improvements in retrieval-based metrics across 42 pairs of seven models, particularly effective when transferring between models from different families or with different depths.