MaxModShift: Model Privacy via Designed Shifts

MaxModShift: Model Privacy via Designed Shifts

Nomaan A. Kherani, Urbashi Mitra

University of Southern California

Summary

This paper addresses the problem of model privacy in federated learning, where an eavesdropper (Eve) attempts to learn the global model by intercepting all uplink communications between agents and the central server (Bob). The authors treat Eve's model learning as an estimation problem and propose a signaling design that drives the Fisher Information Matrix (FIM) of Eve's estimation problem to singularity, thereby preventing her from learning the model.

The system model consists of K agents communicating with a central server to collectively learn a global model w* ∈ R d. Each agent has a local dataset and computes gradients locally. The paper uses FedAvg with a modification where devices share the difference δ k(n) = w k,R(n) − w(n) between their local model after R local epochs and the global model, rather than sharing the full local model. Both Bob and Eve receive signals over orthogonal channels with white Gaussian noise and known channel state information.

The proposed scheme, MaxModShift, introduces intentional model shifts to the transmitted signals. Eve's received signal is modeled as:

y E k(n) = S k(n)δ k(n) + z E k(n), where S k(n) = I + uγ k(n) T, γ k(n) ∈ R d is a designed vector, and u is a shift direction vector agreed upon prior to learning.

The paper first establishes conditions for FIM singularity. Proposition 1 shows that the FIM J(δ k(n)) = (2h E k(n)2/σ2 E) S k(n) T S k(n) is singular when γ k(n) T u = −1. Proposition 2 generalizes this by allowing γ k(n) = h(δ k(n)) to be a function of δ k(n), with the FIM conditions remaining unchanged if δ k(n) T ∇h k,n(δ k(n)) = 0.

The main contribution is the design of shifts that maximize the model shift under a power constraint. The authors formulate an optimization problem P1 that maximizes the squared norm of the model shift experienced by Eve, subject to constraints including: h k,n(δ k(n)) T u = −1, δ k(n) T ∇h k,n(δ k(n)) = 0, a power constraint S k(n)δ k(n)2 + (h k,n(δ k(n)) T δ k(n))2 ≤ αδ k(n)2, and h k,n(δ k(n))2 = α/(1 + u2).

Since the optimization problem is separable across agents and rounds, the authors propose two sub-optimal closed-form solutions. Proposition 3 provides the solutions:

• γ max k(n) = χ k(n)δ k(n) − u(1 + (δ k(n) T u)χ k(n))/u2

• γ min k(n) = −χ k(n)δ k(n) − u(1 − (δ k(n) T u)χ k(n))/u2

where χ k(n) = sqrt((α′ − 1)/(u2δ k(n)2 − (δ k(n) T u)2)) and α′ = αu2/(1 + u2). The condition α′ ≥ 1 is required for χ k(n) to be real. Both solutions satisfy the condition in Equation 9. The proof uses the method of Lagrange multipliers.

The simulation results use a linear regression problem on a synthetic dataset with 100 agents, d = 60 dimensions, Gaussian data with identity covariance, and mean square error loss. The key findings are:

1. Comparison with ModShift: MaxModShift (Min) leads to a higher final loss for Eve compared to the prior ModShift design for all α values. MaxModShift (Max) dominates ModShift for α ≥ 1.2. Both MaxModShift schemes achieve better performance with only 24% of the transmission power required by ModShift.

2. Comparison with noise injection: Both MaxModShift schemes consistently lead Eve to a higher final loss than noise injection schemes. The noise injection scheme requires much larger average transmission power (at least 4 times the highest Pavg of MaxModShift schemes) and requires a greater bandwidth secret channel. The final loss under noise injection is roughly 4 orders of magnitude lower than MaxModShift.

3. Power analysis: The average transmission power Pavg is defined as the ratio of the sum of squared shifted differences plus the scalar shared through the secret channel to the sum of squared original differences. MaxModShift ensures the power constraint scales down with the transmitted signal, while noise injection does not scale down as the updates get smaller.

The paper also notes that the new schemes pass a model tamper test that the eavesdropper may employ, making the use of model shifts more secure. The algorithm for implementing the model shifts is proven to converge, following approaches from prior work.

The conclusions state that MaxModShift offers superior performance in hiding the model from an eavesdropper while requiring significantly lower transmission power compared to both ModShift and noise injection methods. Future work includes developing a greedy approach where agents make decisions in each round based on feedback from Bob.

Improvements to AI Systems:

1. Privacy-Preserving Federated Learning with Adaptive Signal Shaping: Implement MaxModShift’s FIM-singularity-driven shift design directly into federated learning frameworks. The improved AI system can actively manipulate the geometry of the eavesdropper’s estimation problem (by driving Fisher Information to zero) rather than relying on passive noise addition. This allows secure model aggregation over public channels with provable privacy guarantees, even when the adversary has full channel knowledge and intercepts all uplink transmissions.

2. Power-Efficient Secure Aggregation: Replace noise-injection-based privacy mechanisms with the closed-form shift solutions (γ max and γ min) from Proposition 3. The improved system achieves the same or better privacy (higher Eve’s loss) while using only 24% of the transmission power of prior ModShift and at least 4x less power than noise injection. This enables deployment on energy-constrained edge devices (e.g., IoT sensors, mobile phones) without sacrificing model accuracy or convergence speed.

3. Dynamic Privacy-Utility Trade-off Control: Use the power constraint parameter α (from the optimization problem P1) as a tunable knob. The improved AI system can dynamically adjust α per round or per agent to balance between model utility (for Bob) and privacy (against Eve). For example, when the global model is near convergence (small δ k(n)), the system can automatically reduce α to maintain privacy without wasting power, as MaxModShift’s power scales down with the signal magnitude—unlike noise injection.

4. Tamper-Resistant Model Updates: Leverage the model tamper test pass property of MaxModShift. The improved system can detect and reject adversarial manipulations by Eve (e.g., injecting fake gradients or altering updates) because the designed shifts are mathematically tied to the true δ k(n) and the secret direction u. This adds a layer of integrity verification to federated learning, preventing model poisoning attacks while preserving privacy.

5. Scalable Multi-Agent Privacy with Orthogonal Channels: Apply the separable optimization structure (across agents and rounds) to design a decentralized privacy mechanism. The improved system can handle hundreds of agents (as simulated with 100 agents, d=60) with minimal coordination overhead—only a shared secret vector u is needed. This makes it suitable for large-scale cross-device federated learning (e.g., Google’s Gboard, healthcare networks) where per-agent encryption or secure multiparty computation is too costly.

6. Convergence-Guaranteed Private Training: Integrate the proven convergence of MaxModShift (following prior FedAvg analysis) into the training loop. The improved AI system can guarantee that Bob’s global model still converges to w* while Eve’s estimate diverges (final loss orders of magnitude higher). This provides a formal correctness-privacy trade-off, unlike heuristic defenses (e.g., gradient clipping or differential privacy) that often degrade model accuracy.

7. Real-Time Eavesdropper Defense with Minimal Secret Channel: The system only requires a single scalar (h k,n(δ k(n)) T δ k(n)) to be shared over a secret channel per round, not the full shift vector. The improved AI system can operate with a narrowband secret channel (e.g., a low-rate side channel) while the main uplink remains public, reducing infrastructure requirements for secure communication.

8. Adaptive Shift Selection via Feedback: The paper’s future work suggests a greedy approach with Bob’s feedback. An improved system can implement this by having Bob broadcast a small signal indicating the current privacy margin (e.g., FIM determinant). Agents then choose between γ max or γ min (or adjust α) in real time to maximize Eve’s loss while ensuring Bob’s model update quality, enabling autonomous privacy adaptation in non-stationary environments.

Model learning by an eavesdropper is treated as an estimation problem in a federated environment. The Fisher Information Matrix for the eavesdropper's estimation problem is driven to singularity through a signaling design; this ensures that the eavesdropper cannot learn the model. Herein, the innovation of prior designs is that model shifts are designed to maximize the difference in the model learned by Eve and the central server while satisfying a transmission power constraint for the agents. Two shift schemes are provided. MaxModShift outperforms a prior ModShift design while requiring lesser transmission power. Compared to a noise injection scheme, MaxModShift performs better while requiring a lower bandwidth secret channel and a reduced average power consumption.