GraphIP-Bench: How Hard Is It to Steal a Graph Neural Network, and Can We Stop It?

arXiv:2605.12827 · cs.CR, cs.AI, cs.LG · Submitted 2026-05-12 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.

Nadia: I'm Nadia.

Elias: And I'm Elias, here with Priya.

Nadia: We're looking at the paper: "GraphIP-Bench: How Hard Is It to Steal a Graph Neural Network, and Can We Stop It?".

Elias: Here's the gist of the paper. Detailed Research Summary of GraphIP-Bench The research paper introduces GraphIP-Bench, a unified benchmark designed to comprehensively evaluate both model stealing (extraction) and stopping mechanisms against Graph Neural Networks (GNNs) deployed as cloud services.

Nadia: The core innovation of this benchmark is its unified black-box protocol, which allows for the simultaneous testing of twelve distinct extraction attacks against twelve different defense families under standardized conditions.

Elias: Benchmark Scope and Components: GraphIP-Bench integrates a rigorous suite of components to ensure a fair and thorough evaluation: Attacks: Twelve extraction attacks are included, categorized into nine data-driven methods and three data-free methods.

Nadia: Defenses: Twelve defenses span four distinct families: watermarking, output-perturbation, prediction-rounding, and query-pattern detection.

Title and authors: Nadia: So we're looking at this paper called GraphIP-Bench: How Hard Is It to Steal a Graph Neural Network, and Can We Stop It? It seems like the whole point is to give us one standard way to test both the stealing part and the stopping part of these models.

Elias: Exactly. The authors are asking two main questions: how difficult is it for someone to steal a GNN, and what defenses can actually stop that theft. They set out to fix the problems people have had before because previous experiments used different datasets, different threat models, and different ways of measuring success.

Priya: From what I see in the introduction, the authors are introducing this unified benchmark which evaluates both sides under one black-box protocol. That sounds like a big step because it brings consistency to testing these kinds of security mechanisms.

Nadia: Right, so they've built this unified protocol to test twelve different extraction attacks against twelve different defense families all at the same time, which is pretty comprehensive. They use ten public graphs covering different types—homophilic, heterophilic—and three GNN backbones like GCN and GraphSAGE.

Elias: And they aren't just looking at stealing; they are also testing twelve defenses, including watermarking and output perturbation methods, to see how well they actually work when you try to steal the model. They even report on things like fidelity for attacks, task utility for defenses, and the computational cost of everything involved.

Priya: It’s interesting that they are including different types of graphs because the structure seems important in this whole problem. I wonder if testing on just one type of graph would give us a misleading picture about how hard it actually is to steal a GNN.

Nadia: Right, so looking at the results, the paper finds that stealing a GNN is often relatively easy when you look at medium query budgets. The strongest attacks frequently hit fidelities over ninety percent on most datasets under those conditions.

Elias: But they also found some really important things about the defenses themselves, especially when you try to check for ownership. They noticed that defenses like parameter-side watermarks can verify nearly perfectly on the protected model, but they often lose a lot of their verification signal when tested against an extracted surrogate model.

Priya: That means the authors are pointing out that you need to re-evaluate those watermark designs specifically on the extracted surrogate, not just on the original deployed model. That’s a crucial detail for anyone trying to build a real defense system.

Nadia: Right, and they also found that the difficulty of stealing is tied directly to how the graph is structured. Heterophilic graphs are systematically harder to steal than homophilic ones, and certain attacks, like CEGA, are sensitive to the graph's degree distribution.

Title and authors: Elias: And they gave some practical advice about query budgets; they suggest operators shouldn't query beyond about zero point five times or zero point seven five times the training set size because the cost for getting better fidelity just levels off after that.

Priya: So, to sum up what we know so far from GraphIP-Bench, it shows that while stealing can be relatively simple under certain conditions, defenses aren't always as robust as they seem when you factor in the surrogate model.

Nadia: Right, and they also found that no single defense is better everywhere; for instance, backdoor-style triggers might give the best balance between protection and utility across different graph structures.

Elias: And for future work, they suggest that watermarks should be evaluated on the extracted surrogate as the primary metric and should rely more on query-time mechanisms instead of just checking model parameters.

Priya: That makes sense because if we're building these defenses, we need to know what actually survives the extraction process in a real scenario.

Nadia: So, as a wrap up for this paper, GraphIP-Bench gives us a very rigorous and standardized way to see how vulnerable GNNs are to stealing and what our current defenses can handle.

Elias: The main contribution is setting up that unified protocol with public splits and shared queries so that the comparison between the twelve attacks and twelve defenses is fair across all scenarios.

Priya: And another big thing they added was this joint attack-and-defense track which measures watermark survival on the extracted surrogate, which shows us how much protection a defense actually keeps when it's been attacked.

Nadia: So, to wrap up the implications of GraphIP-Bench, it really means that we can't just test defenses on the model itself; we have to test them against models that have already been stolen into a surrogate form.

Elias: That leads us into some future directions where designers need to focus on making sure those surrogate-surviving watermarks are truly robust, maybe by anchoring the verification in how the model responds during a query.

Priya: And we should also keep an eye on those query-detection defenses, because they show some partial survival against extraction even if their verification signal is still tricky.

Nadia: That’s what this paper tells us about where we are right now with GNN security testing before we move on to the next set of research questions.

Elias: It sets a clear standard for how we should measure the difficulty of stealing and how effective our stopping mechanisms really are under realistic, standardized conditions.

Priya: It gives us a much clearer picture of what those defenses actually achieve when you put them through the full extraction process.

The paper's summary: Nadia: So, this paper is basically introducing GraphIP-Bench, which is this new standard testing environment for GNN security.

Elias: Right, it’s a unified black-box protocol that lets you test twelve different stealing attacks against twelve different defenses all at once under the same conditions.

Nadia: It’s a lot of components they put in there, like ten public graphs—some regular ones, some complex ones—and three GNN backbones like GCN and GraphSAGE.

Elias: And they aren't just measuring how hard it is to steal; they are tracking the utility of the defenses too, which is pretty important.

Nadia: The results show that stealing a GNN can actually be pretty easy if you look at medium query budgets. Most strong attacks hit ninety percent fidelity on most datasets then.

Elias: That’s where things get interesting for the defense side, because they found that many defenses, like parameter-side watermarks, are great on the protected model but lose their signal when you test them against a stolen surrogate model.

Nadia: So the authors are saying you can't just trust those defenses by looking at how well they work on the original GNN; you have to re-test them specifically on that extracted surrogate.

Elias: Exactly, and they also found that the graph structure matters a lot, with heterophilic graphs being systematically harder to steal than homophilic ones.

Nadia: And there’s this budget thing they recommend—don't query beyond about zero point five times or zero point seven five times the training set size because the cost doesn't drop much after that.

Elias: They also pointed out that no single defense wins every time; for instance, certain trigger-based graph watermarks seem to offer the best balance between protection and utility across different graph structures.

Nadia: So what this means practically is that we need a more rigorous way to test defenses because the surrogate model is a real threat in the wild.

Elias: It sets up this new standard for how we measure both how easy it is to steal and how effective our stopping mechanisms really are under realistic, standardized conditions.

Nadia: And they're suggesting that future watermarking designs should focus on surviving extraction by using query-time mechanisms more than just checking the model parameters.

Elias: That leads us into the next part of the paper where they discuss how these structural dependencies influence attack performance and what kind of defenses are actually worth pursuing.

The paper's improvements: Tom: So, this paper isn't just about testing; they're suggesting how we should actually design better defenses moving forward.

Nadia: They are proposing a few big tweaks to the benchmark itself to make it more useful for real security work.

Elias: Right, they’re focusing on making sure the measurement—that is, what we count as success—is really capturing what happens when a model is actually stolen.

Nadia: One major improvement they suggest is that instead of just checking the model parameters for a watermark, you should look at how that watermark verification works on the extracted surrogate model.

Elias: That’s the core idea there; they think that’s the most important metric because it shows us what protection actually survives after an attack.

Nadia: So, they are saying we need to re-evaluate those watermarks specifically against that extracted version, not just the original deployed model.

Elias: And in terms of strategy, they link defense performance directly to the graph structure itself—things like edge homophily and degree distribution matter a lot.

Nadia: They suggest this helps us figure out which type of defense configuration is actually best depending on the specific kind of data graph we are dealing with.

Elias: They even point towards backdoor-style triggers as potentially offering the most reliable protection utility balance in their setting, which is something to keep in mind for defense architecture.

Nadia: They also suggest that query-detection defenses that have a bounded clean accuracy cost are worth pursuing because they can offer protection at a fraction of the utility cost compared to other methods.

Elias: That makes sense, it means we might not need the most complex or resource-heavy defenses if we can find these more efficient query-based ones.

Nadia: The big picture here is that this benchmark isn't just a test; it’s guiding us toward designing watermarks and detection methods that are actually robust against the extraction process.

Elias: So, the next step for researchers is to focus on building these surrogate-surviving watermarks and anchoring verification in those query-time mechanisms we talked about.

Conclusion: Tom: So we’ve covered how GraphIP-Bench sets a new standard for testing GNN security by unifying twelve attacks and twelve defenses under one black-box protocol.

Nadia: Exactly, and the results really show that stealing a GNN can be relatively easy if you look at medium query budgets, but it also highlights how defenses perform when tested on those extracted models.

Elias: It’s about moving past just looking at the model itself to testing defenses against what an actual stolen surrogate model looks like in practice.

Priya: So, the main point is that we can't trust a defense by just checking its performance on the original system because it might lose its effectiveness once a model has been successfully extracted.

Nadia: Right, and they’ve given us some concrete advice on budget limits, suggesting operators shouldn't query beyond about zero point five times or zero point seven five times the training set size for better fidelity.

Elias: And they’ve also shown that the structure of the graph—like whether it’s homophilic or heterophilic—directly impacts how hard it is to steal a model.

Priya: That structural dependency is important because it tells us which types of models are inherently more vulnerable based on their design, not just random luck.

Nadia: It really changes how we think about defending these systems; it means we have to tailor our defenses based on the specific graph structure and the query budget we expect.

Elias: And they’ve laid out some clear directions for future work, pushing researchers to focus on making those surrogate-surviving watermarks more robust by anchoring them in query-time mechanisms.

Priya: I think focusing on the query-time verification is a smart move because it grounds the defense in something that happens during actual usage, not just at rest.

Nadia: So, to wrap up this whole discussion about GraphIP-Bench, it’s established as a necessary and rigorous testing ground for evaluating GNN security against extraction threats.

Elias: The unified protocol is the real contribution here; it gives us a consistent way to compare different attacks and defenses across all those different scenarios.

Priya: It gives us a much clearer picture of what those defenses actually achieve when you put them through the full extraction process, which is crucial for building trust in these systems.

Nadia: So, we’ve seen how this benchmark forces us to look beyond simple model evaluations and into the real threat of model stealing.

Elias: And it sets a clear standard for how we should be measuring both the difficulty of stealing and the actual stopping power of our countermeasures.

University of Notre Dame · Florida State University · University of California, Berkeley

cs.CR, cs.AI, cs.LG

Submitted: 2026-05-12

Updated: 2026-11-04

Comments: Accepted at NeurIPS 2026. Code: https://github.com/LabRAI/GraphIP-Bench

Code: https://github.com/LabRAI/GraphIP-Bench

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 97/100

The gist: Detailed Research Summary of GraphIP-Bench The research paper introduces GraphIP-Bench, a unified benchmark designed to comprehensively evaluate both model stealing (extraction) and stopping…

Key concepts

Unified Black-Box Protocol
This standardized testing framework ensures a fair comparison by fixing public splits, shared queries, and budget assumptions across all twelve attacks and twelve defenses. It allows researchers to perform consistent hyperparameter searches and metric evaluations simultaneously.
Extracted Surrogate Model
This is the GNN model after it has been successfully stolen or extracted from its original deployment. Testing defenses on this surrogate model is crucial because it reveals how much protection a defense actually retains after the model has been copied.
Watermark Survival Fidelity
This metric measures how accurately a watermark can be verified when tested against the extracted surrogate. The paper finds that many parameter-based watermarks lose significant verification signal on surrogates, highlighting the need to design watermarks that survive this extraction process.

Terminology

Summary

Detailed Research Summary of GraphIP-Bench

The research paper introduces GraphIP-Bench, a unified benchmark designed to comprehensively evaluate both model stealing (extraction) and stopping mechanisms against Graph Neural Networks (GNNs) deployed as cloud services. The core innovation of this benchmark is its unified black-box protocol, which allows for the simultaneous testing of twelve distinct extraction attacks against twelve different defense families under standardized conditions.

Benchmark Scope and Components:

GraphIP-Bench integrates a rigorous suite of components to ensure a fair and thorough evaluation:

  • Attacks: Twelve extraction attacks are included, categorized into nine data-driven methods and three data-free methods.

  • Defenses: Twelve defenses span four distinct families: watermarking, output-perturbation, prediction-rounding, and query-pattern detection.

  • Datasets and Models: The benchmark utilizes ten public graphs representing various regimes—homophilic, heterophilic, and large-scale structures—and tests three different GNN backbones: Graph Convolutional Networks (GCN), Graph Attention Networks (GAT), and GraphSAGE. It also incorporates three distinct graph-learning tasks for evaluation. p. 0

    Where the paper says it

    We introduce GraphIP-Bench, a unified benchmark which evaluates both sides under a single black-box protocol. It integrates twelve extraction attacks, twelve defenses spanning watermarking, output-perturbation, and query-pattern-detection families, ten public graphs covering homophilic, heterophilic, and large-scale regimes, three GNN backbones, and three graph-learning tasks, and it reports fidelity, task utility, ownership verification, and computational cost on shared splits, queries, and budgets.

  • Metrics: The benchmark reports several critical metrics: fidelity for attacks, task utility for defenses, ownership verification success for watermarking/fingerprinting defenses, and computational cost.

Key Empirical Findings:

The empirical results reveal significant insights into the vulnerability of GNNs to extraction:

  1. Vulnerability to Stealing: The paper concludes that stealing a GNN is often relatively easy, particularly at medium query budgets, as the strongest attacks frequently achieve fidelities exceeding 90% on most datasets under these conditions.

  2. Defense Efficacy and Limitations: Most defenses do not substantially change the ease of extraction at medium budgets. A critical finding from the joint attack-and-defense track is that while parameter-side or trigger-based graph watermarks verify near-perfectly on the protected model, they often lose much of their verification signal when tested against an extracted surrogate model. p. 1

    Where the paper says it

    Going beyond prior single-track evaluations, we add a joint attack-and-defense track which runs every attack on every defended target and a watermark-survival metric which measures verification on the surrogate produced by each attack, together exposing how much protection a defense retains after the model is actually stolen. The headline finding of that joint track is that most parameter-side or trigger-based graph watermarks verify near-perfectly on the protected model but lose much of their verification signal on the extracted surrogate, while query-time mechanisms partially survive — watermark designs must therefore be re-evaluated on the surrogate, not only on the deployed model.

    This suggests that watermark designs must be re-evaluated specifically on the extracted surrogate, not just the deployed target.

  3. Structural Dependencies: The difficulty of stealing is influenced by graph structure:

  • Heterophilic graphs are systematically harder to steal than homophilic ones.

  • The performance of certain attacks, such as CEGA, is sensitive to the graph's degree distribution (e.g., destabilization on high-average-degree graphs like 'Computers').

  1. Budget Sensitivity: The budget-grid recommendation suggests that operators should not query beyond approximately 0.5 times to 0. p. 2

    Where the paper says it

    GraphIP-Bench defines a single black-box protocol that fixes four disjoint splits for each dataset (train, validation, test, query), shares the same query sets across methods, and uses standardized query budgets at 0. 00 times the test size.

    75 times of the training set, as the cost-per-fidelity point becomes essentially flat beyond this range. p. 6
    Where the paper says it

    Utility drop is defended accuracy minus undefended accuracy on the same dataset, and ownership verification is accuracy on a fixed verification set; we summarize across datasets with the median and report variability with the inter-quartile range. Table 2 reports the median values together with time and memory; the corresponding boxplot view and per-dataset numbers are reported in Appendix F.

  2. Defense Performance Nuances: While no single defense dominates across all scenarios (e.g., BackdoorWM is the largest envelope), query-time mechanisms show partial survival against extraction, though their verification signal still requires re-evaluation on the surrogate. p. 8

    Where the paper says it

    5 Joint Attack-and-Defense Evaluation and Watermark Survival (RQ5) To answer RQ5, we run every attack on every defended target and report two metrics: surrogate fidelity to the defended model (whether extraction is still effective) and the watermark verification rate measured on the surrogate, which we call watermark survival (whether a defense still yields a verifiable signal after extraction). We use the same ten datasets and shared query sets as RQ1 with budget fixed at 0.

Main Contributions:

The paper's primary contributions center on establishing a rigorous, standardized framework:

  • Unified Protocol: They establish a unified protocol featuring public splits, shared queries, standardized budgets, and explicit endpoint assumptions to ensure fair comparison across all twelve attacks and twelve defenses.

  • Joint Attack-and-Defense Track: This track measures watermark survival, which is the fidelity of watermark verification when measured on the extracted surrogate. This metric is deemed more crucial for ownership tracing than verification solely on the protected model. p. 8

    Where the paper says it

    First, designs that rely on training-time signal modification cannot be defended against an extracted surrogate that need not preserve the original parameters: verification on the protected model is necessary but insufficient for ownership tracing. Second, future graph watermarks should be evaluated on the extracted surrogate as the primary metric, and at minimum match the survival of Integrity, which in our protocol means anchoring verification in a querytime mechanism rather than in the model parameters.

In essence, GraphIP-Bench provides a necessary, comprehensive testing ground that moves beyond single-model evaluations by explicitly measuring how robust defenses are against models that have been successfully extracted into surrogate forms. p. 0

Where the paper says it

The empirical picture is short: stealing a GNN is easy at medium query budgets and most defenses do not change this; several watermarks verify reliably on the protected model but lose most of their verification signal on the extracted surrogate, which exposes a gap that singlemodel evaluations miss; and heterophilic graphs are systematically harder to steal, while a cross-architecture mismatch between target and surrogate reduces but does not prevent extraction. We release GraphIP-Bench with reproducible scripts and configurations, and integrate the attacks and defenses into the PyGIP library.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Unified Benchmark for Model Extraction and Defense Evaluation

The GraphIP-Bench provides a single black-box protocol that fixes public splits, shared queries, standardized budgets, and explicit endpoint assumptions. This allows researchers to perform a unified hyperparameter search and a consistent metric suite across twelve extraction attacks and twelve defenses simultaneously.

  1. Detection of Ineffective Ownership Tracing Defenses

The joint attack-and-defense track measures watermark verification on the extracted surrogate, which exposes the protection that a defense retains after extraction. This reveals that most parameter-side or trigger-based graph watermarks verify near-perfectly on the protected model but lose much of their verification signal on the extracted surrogate.

  1. Identification of Optimal Defense Strategies Based on Graph Structure

Protection-utility analysis links defense configurations to graph structural properties (edge homophily, degree, density) and to backbone choice. This allows for the determination of which defenses are best under different regimes, such as identifying that backdoor-style triggers provide the most reliable protection-utility balance in our setting.

  1. Development of Robust Graph Watermarking Schemes

Future work suggests that surrogate-surviving watermarks should be evaluated on the extracted surrogate as the primary metric and should anchor verification in a query-time mechanism rather than in the model parameters. This guides the design of new watermarks to ensure they are re-evaluated on the extracted surrogate, not only on the deployed model.

  1. Targeted Query-Detection Defenses with Reduced Utility Loss

The results indicate that query-detection defenses with bounded clean-accuracy cost should be pursued, suggesting that these methods can achieve protection at a fraction of the utility cost compared to other information-limiting defenses.

Abstract

Graph neural networks (GNNs) deployed as cloud services can be *stolen* through *model-extraction attacks*, which train a surrogate from query responses to reproduce the target's behaviour, and a growing line of ownership defenses tries to prevent or trace such theft. The title of this paper asks two questions: *how hard is it to steal a GNN?*, and *can we stop it?* Prior work cannot answer either, because experiments use inconsistent datasets, threat models, and metrics. We introduce *GraphIP-Bench*, a unified benchmark which evaluates both sides under a single black-box protocol. It integrates twelve extraction attacks, twelve defenses spanning watermarking, output-perturbation, and query-pattern-detection families, ten public graphs covering homophilic, heterophilic, and large-scale regimes, three GNN backbones, and three graph-learning tasks, and it reports fidelity, task utility, ownership verification, and computational cost on shared splits, queries, and budgets. We further add a joint attack-and-defense track which runs every attack on every defended target and measures watermark verification on the resulting surrogate, which exposes the protection that a defense retains after extraction. The empirical picture is short: stealing a GNN is easy at medium query budgets and most defenses do not change this; several watermarks verify reliably on the protected model but lose most of their verification signal on the extracted surrogate, which exposes a gap that single-model evaluations miss; and heterophilic graphs are systematically harder to steal, while a cross-architecture mismatch between target and surrogate reduces but does not prevent extraction. We release *GraphIP-Bench* with reproducible scripts and configurations, and integrate the attacks and defenses into the PyGIP library.

Sources

Related papers