One Adapter Pair per Model: A Universal Activation Interface for Language Models

The paper introduces the Universal Activation Bus, a framework that provides a common activation interface across compatible language models. The core idea is to learn a shared dense space Z using a small set of source models, with each model having one lightweight linear encoder–decoder adapter pair. After source training, the interface is frozen, and a new model joins by fitting only its adapter pair on unlabeled matched text.

The framework is designed to satisfy two properties: reconstruction (Dm(Em(hm(x))) ≈ hm(x)) and alignment (Em(hm(x)) ≈ En(hn(x))). This enables a state from model m to be translated into model n through Dn(Em(hm(x))), without learning a connector specific to the pair (m, n). The architecture includes model-specific normalization (z-score and ZCA whitening), linear encoder/decoder adapters, and two small shared residual MLPs (Φin and Φout). Training uses self-reconstruction, cross-reconstruction, and contrastive alignment losses. Onboarding a new model freezes the shared modules and existing adapters, optimizing only the new model's adapter pair using a source consensus representation.

The paper evaluates the interface using Llama-3.2-3B, Qwen3-4B, SmolLM2-1.7B, Falcon3-3B, and OLMo-2-7B. The first four are source models, and OLMo is onboarded post-hoc. Results show near-perfect in-distribution retrieval (R@1 of 99.6–100%) under both joint training and freeze-then-onboard. Out-of-distribution R@1 is lower (60.8–78.7%) because each axis contains many semantically similar texts, but R@10 of 79.6–90.8% shows the exact match generally remains in the same local neighborhood. Self-FVE (round-trip reconstruction fidelity) is 0.989–0.993 for source models and 0.734 for OLMo, which has a wider native space (dOLMo = 4096 > D = 3072).

The paper demonstrates three forms of tool reuse. First, shared linear probes trained on pooled source states achieve macro AUROC of 0.966 on source models and 0.959 on onboarded OLMo, matching separately fitted native probes (0.963). Second, a single shared TopK SAE (32,768 features, k=32) trained on source models achieves Top-32 active-set Jaccard overlaps of 0.591–0.614 across source pairs and 0.602–0.613 for pairs involving OLMo, with no onboarding gap. The SAE retains 93.8–98.8% of native SAE utility relative to a zero-state baseline. Third, a carrier-specific NLA trained for Qwen2.5-7B can be reused by translating states from other models into the carrier's native space. A text classifier trained only on native-carrier verbalizations achieves 88.2–94.4% accuracy on routed source models and 84.9% on OLMo, compared with 95.3% on native-carrier states.

The paper also tests cross-model state execution by translating a front model's cut-layer activation and inserting it into a back model's frozen upper layers. Across cross-model paths, the composed model agrees with native back model on 72–88% of next tokens, exceeding native A–B agreement by 7.5–27.2 percentage points in every direction. Identity examples show the back model controls output behavior (e.g., Llama→Qwen identifies as Qwen). Paths into Llama, Qwen3, SmolLM2, and Falcon3 incur only −0.023 to +0.071 nats of additional cross-entropy, while OLMo is a more difficult receiver (+0.273 to +0.381 nats).

The frozen interface extends to ten additional checkpoints spanning 1.24B–14.8B parameters, all achieving minimum ID R@1 of 99.9–100.0% and correct behavioral-axis assignment (9/9 axes). Calibration efficiency shows OLMo R@1 saturates at 10K matched positions (99.6%), with no material improvement at 30K or 100K. Sensitivity checks show self-FVE improves modestly with more source models (0.712 to 0.734 for OLMo), varies smoothly with cut depth (0.771 to 0.762 across 0.25–0.75), and is stable across random seeds (ID R@1 of 99.949 ± 0.008%).

The paper concludes that the Universal Activation Bus establishes a stable, model-wise activation contract for reusable tools across compatible language models, replacing pair-specific connectors with an extensible activation ecosystem that remains stable across compatible model families, scales, and calibration budgets.

Improvements to AI Systems:

1. Modular Cross-Model Tool Reuse

• Train interpretability tools (linear probes, sparse autoencoders) once on a shared activation space, then deploy them across any compatible model without retraining.

• This eliminates the need for per-model tool adaptation, reducing engineering overhead and enabling instant deployment of safety/alignment probes on new models.

2. Zero-Shot Model Interchangeability

• Translate activations from one model into another’s native space, allowing a front-end model’s internal states to be executed by a back-end model’s frozen layers.

• This enables swapping model components (e.g., using a smaller, faster model for early layers and a larger, more capable model for later layers) without fine-tuning, improving inference efficiency and flexibility.

3. Unified Behavioral Control via Shared Probes

• Use a single shared linear probe to detect or steer specific behaviors (e.g., toxicity, sentiment, factual consistency) across all compatible models.

• This allows consistent, model-agnostic safety filtering and behavior monitoring, reducing the risk of inconsistent oversight when models are updated or replaced.

4. Cross-Model Knowledge Distillation and Ensemble

• Leverage the shared activation space to transfer states between models, enabling ensemble predictions where multiple models’ intermediate representations are combined in a common space.

• This can improve robustness and accuracy by aggregating diverse model perspectives without needing to align their output distributions directly.

5. On-the-Fly Model Onboarding for New Architectures

• When a new model is introduced, only its lightweight adapter pair needs calibration (using 10K matched text positions), not a full retraining of tools or connectors.

• This accelerates deployment of new models into existing AI pipelines, supporting rapid iteration and experimentation with minimal downtime.

6. Cross-Model State Execution for Controlled Generation

• Translate a source model’s hidden states into a target model’s space to influence generation style, persona, or factual grounding.

• For example, a model with better reasoning can drive a more fluent but weaker model’s output, or a model with a desired persona can control another model’s identity, enabling flexible behavior shaping without retraining.

7. Improved Out-of-Distribution Retrieval and Semantic Search

• Use the shared activation space to perform cross-model semantic retrieval, where queries from one model can retrieve relevant texts from another model’s representations.

• This enables cross-lingual or cross-domain search and recommendation systems that are model-agnostic, improving scalability and interoperability.

8. Efficient Calibration and Resource Allocation

• Since calibration saturates at 10K matched positions, systems can allocate minimal data and compute for onboarding, freeing resources for other tasks.

• This is particularly useful in resource-constrained environments or when continuously updating models with new data.

What the Improved AI System Can Do:

• Deploy a single suite of interpretability and safety tools across an entire fleet of language models, ensuring uniform oversight and reducing maintenance costs.

• Dynamically combine models at inference time—e.g., using a lightweight model for early layers and a heavyweight model for later layers—to balance speed and accuracy without retraining.

• Perform zero-shot cross-model state transfer for style, persona, or task control, enabling rapid customization of AI assistants for different user groups or applications.

• Onboard new models into an existing AI ecosystem within minutes, using only a small calibration set, and immediately benefit from all pre-trained tools and connectors.

• Build ensemble systems that fuse intermediate representations from multiple models in a shared space, improving robustness to distribution shift and adversarial inputs.

• Provide consistent, model-agnostic semantic search and retrieval across different model families, facilitating interoperability in multi-model platforms (e.g., AI agent swarms, federated learning).