Breaking via Adaptive Neuron Clustering in Cryptanalytic Model Extraction
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.
Nadia: I'm Nadia.
Elias: And I'm Elias, here with Priya.
Nadia: We're looking at the paper: "Breaking via Adaptive Neuron Clustering in Cryptanalytic Model Extraction".
Elias: Here's the gist of the paper. Cryptanalytic model extraction poses a fundamental threat to deep learning model confidentiality by enabling precise parameter recovery with theoretical guarantees of functional equivalence The single most informative sentence about the paper's subject and finding is SimBreak, an adaptive clustering rule that breaks the weight-similarity defense of TraintoDefend.
Nadia: The Threat Landscape Cryptanalytic model extraction represents a severe threat to deep learning models by drawing on cryptanalysis principles to design informative queries, thereby recovering hidden parameters with high precision and reconstructing functionally equivalent models Advances in this field have broadened the range of models and interfaces exposed to cryptanalytic extraction, motivating defenses that account for increasingly capable adversaries.
Elias: TraintoDefend was introduced as the first dedicated defense against cryptanalytic extraction by modifying the training objective to encourage different neurons to learn similar weights so that they switch activation states together.
Nadia: The Failure Mechanism of TraintoDefend The defense in TraintoDefend rests on the premise that similar weights cause neurons to switch activation states together, mixing their contributions to local derivative measurements and hindering signature recovery However, the original extraction attack can merge signatures from distinct neurons when sufficiently many normalized weight coordinates agree, even if the remaining coordinates differ substantially.
Paper summary: Nadia: We talked about the core idea of this paper, "Breaking via Adaptive Neuron Clustering in Cryptanalytic Model Extraction." The main point is that cryptanalytic model extraction is a serious threat because it allows people to recover hidden model parameters with high precision and guarantees that the recovered model acts functionally like the original one.
Elias: And to stop this, they looked at TraintoDefend, which aimed to make neurons learn similar weights so they switch their activation states together, making them harder to distinguish individually. But the paper finds a flaw in that defense structure.
Priya: What is that flaw specifically? Why can't neurons with similar weights always be distinguished by switching their activation states?
Nadia: The researchers noticed that neurons with similar weights do not necessarily switch activation states together when you make small changes to the model, which means the original extraction attack can still merge signatures from distinct neurons.
Elias: This merging of signatures is what they call signature clustering, and it’s a concrete obstacle that TraintoDefend introduces into the existing extraction attack. The original rule can combine signatures from different neurons even if some of their weight coordinates are very different.
Priya: So, this clustering is a technical problem that the paper identifies as a weakness in how TraintoDefend handles neuron similarity.
Nadia: Precisely, and they then introduce SimBreak to solve this clustering obstacle. SimBreak works by formulating pairwise signature matching as a test of whether two local observations agree under a common relative scale.
Elias: The key part of that is how it estimates the relative scale, which they figure out by least squares over all jointly observed weight coordinates, which is described in Section four point one <ref:2611.07370#pg1>.
Priya: So the method involves estimating this scale using a least squares approach across all the weights to check for agreement between local measurements. That’s a specific mathematical mechanism they are using to overcome the issue of clustering signatures.
Nadia: And they show that this results in a normalized residual that incorporates discrepancies across every single jointly observed coordinate, which prevents agreement on just a small subset of coordinates from being enough to pass the test.
Elias: It really forces the two local observations to agree based on a much broader set of data than what was previously used, which is exactly what breaks the clustering mechanism.
Priya: So they are essentially making the required agreement condition much stricter by demanding consistency across all observed weight coordinates simultaneously. That sounds like a solid way to handle noisy data in this context.
Conclusion: Nadia: Wrapping up, the paper "Breaking via Adaptive Neuron Clustering in Cryptanalytic Model Extraction" shows that TraintoDefend’s defense is not as strong as it appeared against adaptive attacks. SimBreak successfully breaks the layer signature recovery barrier by raising recovery from zero percent to one hundred percent across all six tested models.
Elias: The authors conclude that this finding challenges the premise of TraintoDefend, which was that weight similarity automatically prevents cryptanalytic extraction by forcing neurons to switch states together.
Priya: So what does this mean for the practical security landscape right now? Does it suggest we need to rethink how we design defenses against these kinds of adaptive attacks?
Nadia: It suggests that defenses based on neuron similarity must be more sophisticated; they have to account for attacks that exploit both weight directions and affine offsets, not just the simple coordinate matching assumptions used in existing implementations.
Elias: That’s the big picture—defenses need to adapt to how adversaries design their queries rather than just relying on a single defensive mechanism that seems robust.
Priya: So, for someone listening who isn't deep into the math, this paper tells us that you can't rely on simple similarity checks alone when trying to secure deep learning models from sophisticated extraction attacks.
Nadia: That’s right. The work demonstrates that adaptive attacks can exploit subtle discrepancies in how neurons operate even when they look similar on the surface.
Haojie Wu, Haolin Liu, Tianshuo Cong, Guoyan Zhang
School of Cryptologic Science and Engineering, Shandong University · School of Computer Science, Shanghai Jiao Tong University
cs.CR
Submitted: 2026-11-05
Updated: 2026-11-05
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 87/100
The gist: Cryptanalytic model extraction poses a fundamental threat to deep learning model confidentiality by enabling precise parameter recovery with theoretical guarantees of functional equivalence The…
Key concepts
- Cryptanalytic Model Extraction
- This is a method where an attacker uses mathematical principles, similar to cryptography, to query a neural network. The goal is to recover the hidden internal parameters (weights) of the model with high accuracy, even if those parameters are secret.
- TraintoDefend
- This was an initial defense mechanism designed against extraction attacks. It worked by training the model so that neurons learning similar weights would also tend to switch their activation states together. The idea was that this similarity would confuse the signature recovery process.
- SimBreak
- This is a new adaptive clustering rule used to counter TraintoDefend. Instead of relying on simple weight similarity, SimBreak estimates a consistent relative scale across all observed weight coordinates and uses that scale to test for both weight direction and bias consistency. This breaks the defense's assumption.
- Signature Clustering
- This is the phenomenon where an extraction attack merges the signatures (information about weights) from different neurons because their weight coordinates agree under a certain condition. TraintoDefend tried to prevent this, but SimBreak exploits this clustering by finding a common scale that allows these distinct signatures to be matched.
Terminology
Summary
Cryptanalytic model extraction poses a fundamental threat to deep learning model confidentiality by enabling precise parameter recovery with theoretical guarantees of functional equivalence The single most informative sentence about the paper's subject and finding is SimBreak, an adaptive clustering rule that breaks the weight-similarity defense of TraintoDefend.
The Threat Landscape
Cryptanalytic model extraction represents a severe threat to deep learning models by drawing on cryptanalysis principles to design informative queries, thereby recovering hidden parameters with high precision and reconstructing functionally equivalent models Advances in this field have broadened the range of models and interfaces exposed to cryptanalytic extraction, motivating defenses that account for increasingly capable adversaries. TraintoDefend was introduced as the first dedicated defense against cryptanalytic extraction by modifying the training objective to encourage different neurons to learn similar weights so that they switch activation states together. p. 1 To counter this threat, TraintoDefend [12] was introduced as the first dedicated
defense against cryptanalytic extraction, making its robustness a central question
for the protection of model parameters. TraintoDefend modifies the training
objective to encourage different neurons to learn similar weights.Where the paper says it
The Failure Mechanism of TraintoDefend
The defense in TraintoDefend rests on the premise that similar weights cause neurons to switch activation states together, mixing their contributions to local derivative measurements and hindering signature recovery However, the original extraction attack can merge signatures from distinct neurons when sufficiently many normalized weight coordinates agree, even if the remaining coordinates differ substantially. This signature clustering is a concrete obstacle introduced by TraintoDefend in the existing extraction attack. p. 2 Joint switching depends on the probe center,
direction, and step size, so weight similarity alone does not ensure that a probe
crosses multiple activation boundaries. We identify signature clustering as a
concrete obstacle introduced by TraintoDefend in the existing extraction attack.Where the paper says it
The SimBreak Solution
SimBreak is an adaptive clustering rule that breaks the weight-similarity defense of TraintoDefend It estimates a relative scale from all jointly observed weight coordinates and uses the same scale to test both weight direction and bias consistency. The rule formulates pairwise signature matching as a test of whether two local observations agree under a common relative scale It estimates this scale by least squares over all jointly observed weight coordinates, as described in Section 4. p. 2 To overcome
the clustering obstacle introduced by TraintoDefend, SimBreak formulates pairwise
signature matching as a test of whether two local observations agree under a
common relative scale. It estimates this scale by least squares over all jointly
observed weight coordinates, as described in Section 4.Where the paper says it
Experimental Validation
The experimental evaluation demonstrates that replacing the original rule with SimBreak raises first-layer signature recovery from 0% to 100% across all six models. This complete recovery is achieved across widths from 8 to 64 in two-hidden-layer networks and also in the deeper models, as shown in Section 5. p. 2 Replacing the original rule with
SimBreak raises first-layer signature recovery from 0% to 100% across all six
models, with a maximum reported relative recovery error of 2
−22. Complete
recovery is achieved across widths from 8 to 64 in two-hidden-layer networks and
also in the deeper models, as shown in Section 5. We
evaluate SimBreak on six models with varying widths and depths and compare
it with the original coordinate agreement rule. Replacing the original rule with
SimBreak raises first-layer signature recovery from 0% to 100% across all six
models, with a maximum reported relative recovery error of 2
−22.Where the paper says it
Where the paper says it
Conclusion and Implications
The work challenges TraintoDefend’s premise that weight similarity prevents cryptanalytic extraction by causing neurons to switch activation states together SimBreak bypasses the protected layer signature recovery barrier of TraintoDefend, raising recovery from 0% to 100% across all six evaluated models with a maximum reported relative error of 2−22.27. This underscores the need to evaluate defenses against adaptive attacks. p. 1 These advances
broaden the range of models and interfaces exposed to cryptanalytic extraction,
motivating defenses that account for increasingly capable adversaries. To counter this threat, TraintoDefend [12] was introduced as the first dedicated
defense against cryptanalytic extraction, making its robustness a central question
for the protection of model parameters.Where the paper says it
References
Asselineau, R., Derbez, P., Fouque, P.A., Minaud, B.: Cryptanalytic extraction of deep neural networks with non-linear activations Canales-Martínez, I. p. 0 Our work challenges the use of neuron weight
similarity as a sufficient basis for protection and establishes adaptive
evaluation as essential to assessing cryptanalytic extraction defenses. Keywords: Cryptanalytic Model Extraction · TraintoDefend · Adaptive
Signature Clustering
1 Introduction
Deep Neural Networks (DNNs) constitute valuable digital assets, embodying
considerable investment in proprietary data, computational resources, and engineering know-how. Cryptanalytic model
extraction, by contrast, draws on cryptanalysis principles to design informative
queries, thereby recovering hidden parameters with high precision and reconstructing functionally equivalent models, turning black-box access into model
possession in effect. Since Carlini et al. Our work challenges the use of neuron weight
similarity as a sufficient basis for protection and establishes adaptive
evaluation as essential to assessing cryptanalytic extraction defenses. Keywords: Cryptanalytic Model Extraction · TraintoDefend · Adaptive
Signature Clustering
1 Introduction
Deep Neural Networks (DNNs) constitute valuable digital assets, embodying
considerable investment in proprietary data, computational resources, and engineering know-how. However, their adaptive attack recovers models by estimating directional derivatives over larger probing intervals. Moreover, cryptanalytic extraction has also
been demonstrated in the hard-label setting, where the attacker observes only
predicted class labels [3, 5]. However, their adaptive attack recovers models by estimating directional derivatives over larger probing intervals. Moreover, cryptanalytic extraction has also
been demonstrated in the hard-label setting, where the attacker observes only
predicted class labels [3, 5]. However, their adaptive attack recovers models by estimating directional derivatives over larger probing intervals. Moreover, cryptanalytic extraction has also
been demonstrated in the hard-label setting, where the attacker observes only
predicted class labels [3, 5]. These activations serve as known inputs to the next layer,
allowing the attacker to repeat the above operations for the next layer [4, 16]. The process continues through the hidden layers, after which the final affine layer
is recovered by solving a linear system using the recovered hidden activations
and oracle outputs. 2 Cryptanalytic Extraction Attacks
Cryptanalytic extraction reconstructs fully connected ReLU networks by exploiting their piecewise affine structure through queries that return raw numerical
outputs or hard labels [3–5]. The attack proceeds layer by layer, recovering neuron
signatures and then determining their signs before moving to the next layer. Keywords: Cryptanalytic Model Extraction · TraintoDefend · Adaptive
Signature Clustering
1 Introduction
Deep Neural Networks (DNNs) constitute valuable digital assets, embodying
considerable investment in proprietary data, computational resources, and engineering know-how. To commercialize DNNs while preserving control over their
parameters, providers typically offer them under the Model-as-a-Service (MaaS)
arXiv:2611. [4] demonstrated how the piecewise affine structure of
ReLU networks can be exploited to recover model parameters through output
queries, cryptanalytic model extraction has advanced along several directions. Improvements in signature estimation and sign recovery have made extraction
more efficient and enabled attacks on deeper networks [16]. Cryptanalytic model
extraction, by contrast, draws on cryptanalysis principles to design informative
queries, thereby recovering hidden parameters with high precision and reconstructing functionally equivalent models, turning black-box access into model
possession in effect. Since Carlini et al. Conventional learning-based extraction
mainly approximates task behavior from sampled queries. Cryptanalytic model
extraction, by contrast, draws on cryptanalysis principles to design informative
queries, thereby recovering hidden parameters with high precision and reconstructing functionally equivalent models, turning black-box access into model
possession in effect. Nevertheless, recent progress in cryptanalytic model extraction [2, 4, 5] represents a severe threat. Conventional learning-based extraction
mainly approximates task behavior from sampled queries. However, their adaptive attack recovers models by estimating directional derivatives over larger probing intervals. Moreover, cryptanalytic extraction has also
been demonstrated in the hard-label setting, where the attacker observes only
predicted class labels [3, 5]. Our work challenges the use of neuron weight
similarity as a sufficient basis for protection and establishes adaptive
evaluation as essential to assessing cryptanalytic extraction defenses. Keywords: Cryptanalytic Model Extraction · TraintoDefend · Adaptive
Signature Clustering
1 Introduction
Deep Neural Networks (DNNs) constitute valuable digital assets, embodying
considerable investment in proprietary data, computational resources, and engineering know-how. Our work challenges the use of neuron weight
similarity as a sufficient basis for protection and establishes adaptive
evaluation as essential to assessing cryptanalytic extraction defenses. Keywords: Cryptanalytic Model Extraction · TraintoDefend · Adaptive
Signature Clustering
1 Introduction
Deep Neural Networks (DNNs) constitute valuable digital assets, embodying
considerable investment in proprietary data, computational resources, and engineering know-how. 4 Other Defenses
Noman et al [17]. investigate output rounding as a defense against cryptanalytic
extraction, reducing the precision of returned outputs while leaving the model
parameters unchanged. Our work challenges the use of neuron weight
similarity as a sufficient basis for protection and establishes adaptive
evaluation as essential to assessing cryptanalytic extraction defenses. Keywords: Cryptanalytic Model Extraction · TraintoDefend · Adaptive
Signature Clustering
1 Introduction
Deep Neural Networks (DNNs) constitute valuable digital assets, embodying
considerable investment in proprietary data, computational resources, and engineering know-how. Our work challenges the use of neuron weight
similarity as a sufficient basis for protection and establishes adaptive
evaluation as essential to assessing cryptanalytic extraction defenses. Keywords: Cryptanalytic Model Extraction · TraintoDefend · Adaptive
Signature Clustering
1 Introduction
Deep Neural Networks (DNNs) constitute valuable digital assets, embodying
considerable investment in proprietary data, computational resources, and engineering know-how. However, their adaptive attack recovers models by estimating directional derivatives over larger probing intervals. Moreover, cryptanalytic extraction has also
been demonstrated in the hard-label setting, where the attacker observes only
predicted class labels [3, 5].Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Adaptive defense mechanism for cryptanalytic model extraction: The improved system can employ SimBreak to distinguish neurons with similar weights by jointly evaluating weight direction and bias consistency under a relative scale fitted over all observed weight coordinates.
-
Enhanced signature recovery capability: The improved system can achieve complete first layer signature recovery in models protected by TraintoDefend, raising recovery from 0% to 100% across all evaluated configurations with a maximum reported relative error of 2-22.27.
-
Robust defense against adaptive attacks: The improved system resists adaptive extraction by challenging the use of neuron weight similarity as a sufficient basis for protection, demonstrating that
adaptive evaluation is essential to assessing cryptanalytic extraction defenses.
Abstract
Cryptanalytic model extraction poses a fundamental threat to deep learning model confidentiality by enabling precise parameter recovery with theoretical guarantees of functional equivalence. To counter this growing threat, (NeurIPS 2025) was proposed as the first dedicated defense against cryptanalytic extraction and remains the only defense believed to be effective to date. It encourages neurons to learn similar weights so that they switch activation states together, making their individual contributions harder to distinguish. However, we notice that neurons with similar weights need not switch activation states together under small perturbations. Instead, weight similarity can cause the original clustering rule to merge signatures from distinct neurons when only a subset of their normalized weight coordinates agree. Motivated by this observation, we introduce, an adaptive clustering rule that breaks the weight-similarity defense of. estimates a relative scale from all jointly observed weight coordinates and uses the same scale to test both weight direction and bias consistency. Across six protected fully connected networks trained on MNIST, raises target layer signature recovery from 0% to 100%, with a maximum relative recovery error of 2-22.27. Our work challenges the use of neuron weight similarity as a sufficient basis for protection and establishes adaptive evaluation as essential to assessing cryptanalytic extraction defenses.
Related papers
- From Strategy Search to Policy Learning: MCTS-Guided Adversarial Co-Evolution for LLM Safety Alignment
- Privacy Leakage from Recurrent State in LLM Inference
- Incrementally Verifiable Oracle Aided Computation
- Privilege and confidentiality in generative AI workflows
- BadMoE: Backdooring Mixture-of-Experts LLMs via Routing-Aware Triggers and Task-Underutilized Experts
- CoVault: Secure, Scalable Analytics of Personal Data