CellularS2-Bench: A Staged, Evidence-Grounded Benchmark for Cellular Network Security Reasoning

arXiv:2601.12716 · cs.CR · Submitted 2026-01-19 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.

Nadia: I'm Nadia.

Elias: And I'm Elias, here with Priya.

Nadia: We're looking at the paper: "CellularS2-Bench: A Staged, Evidence-Grounded Benchmark for Cellular Network Security Reasoning".

Elias: Here's the gist of the paper. Cellular networks are critical infrastructure supporting billions of worldwide users and safety- and mission-critical services, motivating growing efforts to analyze 3GPP specifications that define required device and operator behavior.

Nadia: To address the challenges of faithful interpretation of normative language, reasoning across cross-referenced clauses, and verifiable conclusions grounded in multimodal evidence such as tables and figures in cellular specifications, the authors propose CellSpecSec-ARI, a unified Adapt–Retrieve–Integrate framework for systematic understanding and standarddriven security analysis of 3GPP specifications.

Elias: They establish CellularSpecSec-Bench, a staged benchmark containing newly constructed highquality datasets with expert-verified and corrected subsets from prior open-source resources to quantify progress in specification understanding and security reasoning in the cellular network security domain.

Nadia: The CellSpecSec-ARI Framework CellSpecSec-ARI is a unified framework designed to solve standardized tasks in (1) interpreting complex 3GPP specifications written in domain-specific language with multimodal context, and (2) performing the challenging security analysis over cellular network standards.

Title and authors: Nadia: So, we're looking at the paper called CellularS2-Bench: A Staged, Evidence-Grounded Benchmark for Cellular Network Security Reasoning. This whole thing is about building a way to test how well models actually understand and can reason about those really complex 3GPP specifications <ref:2601.12716#pg1>.

Elias: Right. It’s tackling the difficulty of interpreting that dense, normative language and making sure the answers are actually grounded in the evidence, not just guessing based on what sounds right.

Nadia: Exactly. They introduce this whole framework called CellSpecSec-ARI, which has three stages: Adapt, Retrieve, and Integrate. It’s like they're trying to teach a model everything from reading a simple rule to synthesizing information across all the different parts of the specification at an expert level.

Priya: From my side, I'm wondering if this structure actually captures how security reasoning works in practice. Are we just looking at surface-level comprehension, or is this framework actually testing if the model can follow the chain of dependencies?

Elias: That’s a fair point, Priya. The Adapt part, SpecFusion, seems designed to get the model familiar with the domain first by turning those tables and figures into text descriptions. It helps it learn how rules are written in those non-textual formats.

Nadia: Right, so SpecFusion is about adapting to the multimodal nature of the specs. Then you have SpecRAG, which does the actual evidence retrieval, using a hybrid search method that mixes keyword matching with dense vector embeddings for semantic similarity.

Priya: I'm thinking about that retrieval part—how they’re trying to get the right chunks when you have thousands of pages and figures. Does this hybrid approach actually help filter out the noise effectively?

Elias: The hybrid search uses sparse component ssp(q, c) with a BM25 ranker for lexical precision, combined with dense component sde(q, c) from TF-IDF vectors projected into a lower-dimensional latent space for semantic relevance. That way, you get both precise terms and the broader meaning of what you’re asking.

Nadia: And then SpecReasoning takes all that retrieved evidence and turns it into a knowledge graph. This graph maps out all the connections between entities like procedures, messages, and security properties, defining core relations and specific security relations.

Title and authors: Priya: That sounds like a big step toward modeling the structure of the standard itself. So instead of just getting an answer, we’re trying to see if it can map out the procedural logic of the cellular network for us.

Elias: Precisely. They construct this taxonomy by first analyzing every clause in a specification to figure out how things are organized hierarchically before they define those relations for the reasoning module.

Nadia: And then they use that graph to perform security analysis over cellular network standards, which is the ultimate goal here. It moves beyond just understanding a single sentence or procedure.

Priya: But what about the benchmark itself? The paper mentions CellularSpecSec-Bench covers core specifications from Release seventeen processing over one thousand seven hundred sixty paragraphs and three hundred forty tables <ref:2601.12716#pg2,processing over 1,760 paragraphs>. That’s a substantial amount of material for one test set.

Elias: Yeah, that scale is important because it shows they’re not just testing on tiny snippets. They are testing against a real volume of specification content to see how the model handles cross-clause reasoning over that entire context.

Nadia: It also includes forty-three real-world vulnerabilities reported in prior work, which is what gives the security reasoning tasks their grounding and relevance in the actual cellular network world.

Priya: So, while they’ve built this large dataset, I need to ask about its limitations. The paper does mention that prior datasets focused more on sentence or paragraph level understanding rather than standards-driven procedural reasoning that needs evidence across multiple clauses and figures. Does CellularS2-Bench fully address that gap?

Elias: They explicitly state that the prior work had limitations, so this benchmark is designed to move past those. It aims to evaluate evidence-grounded performance, which they call critical for complex reasoning over procedures and security properties.

Nadia: The authors also flag that their security focus isn't just on text classification; it’s about reasoning about end-to-end protocol behavior under adversarial exploitation conditions, which is a tougher test.

Priya: That’s interesting because the evaluation shows that incorporating the CellSpecSec-ARI modules substantially improves performance across all tasks compared to the base model. That tells us the structure of their framework actually helps overcome those initial struggles with normative style and logic.

Title and authors: Elias: It suggests that breaking down the task into adaptation, retrieval, and integration stages provides a clear path for improving standards comprehension systematically. The error analysis also pointed out that failures often come from incomplete evidence rather than totally wrong reasoning.

Nadia: So the biggest lesson there is that if you want reliable security analysis on these specs, you need to ensure the model has all the necessary pieces of evidence assembled before it makes a claim.

Priya: That means we need to look closely at how much context is being retrieved and integrated during those retrieval and reasoning steps. The completeness verifier they suggest seems like a good idea for enforcing that evidence requirement.

Elias: It definitely emphasizes that security reasoning in standards is compositional, which means you can’t just look at one clause in isolation; you have to see how it connects structurally to everything else.

Nadia: So, CellularS2-Bench is presented as a way to quantify this progress, giving the community a reproducible standard for evaluating these systems. It moves the conversation from "can the LLM read this?" to "can the LLM reason securely about this?"

Priya: And that’s what matters for real-world deployment. A benchmark that tests against real vulnerabilities helps us actually build better tools and defenses, instead of just guessing at their capabilities.

Elias: So, wrapping up on CellularS2-Bench, it sets a new standard for how we can systematically evaluate models against the rigorous demands of 3GPP specifications through this Adapt-Retrieve-Integrate approach <ref:2601.12716#pg1>.

Nadia: It’s a solid piece of work because it gives us a unified methodology for tackling those multimodal, complex tasks in cellular security analysis.

Priya: I just hope they keep expanding that coverage beyond Release seventeen to really reflect the evolving standards across different releases <ref:2601.12716#pg2>.

Elias: That’s the direction they suggest, and building upon this foundation means we can start comparing models on more challenging reasoning scenarios in the future.

Nadia: We'll stick around to see how these benchmarks evolve and what kind of security insights they unlock for us down the line.

The paper's summary: Nadia: So, we just looked at how they built this benchmark called CellularS2-Bench, and what it’s really trying to do for testing these language models against cell network security.

Elias: Right. It’s basically a structured way to see if a model can actually reason about those dense 3GPP standards when there are pictures and tables involved, not just reading sentences <ref:2601.12716#pg1>.

Nadia: Exactly. The core idea is this three-stage framework, Adapt, Retrieve, Integrate. They’re trying to mimic how an expert learns something—from basic understanding to deep synthesis across all the pieces of a standard.

Elias: That adaptation part they call SpecFusion is where they teach the model the domain knowledge by turning those non-textual bits like tables and figures into structured text descriptions. It shows it how those rules are actually written in that format.

Nadia: Then you have retrieval, SpecRAG, which uses a hybrid search—keyword matching mixed with dense vector embeddings—to pull out the exact evidence chunks needed for the question at hand.

Elias: And then SpecReasoning takes all that retrieved evidence and turns it into a knowledge graph. This graph maps out all the structural connections between things like procedures and security properties, which lets it do that cross-clause integration you mentioned earlier.

Nadia: It’s really about moving past simple comprehension to complex reasoning about protocol behavior under attack conditions, which is what they’re focusing on for the security analysis tasks.

Elias: And the benchmark itself is built on a huge chunk of Release seventeen specs, with over seventeen hundred paragraphs and four hundred twenty-four figures to test against. That scale is what makes it feel more like real-world testing than just looking at a few examples.

Nadia: But Priya, you were concerned about the data quality. What does this benchmark actually show us about the model’s reasoning compared to just having a big test set?

Priya: Well, they included forty-three real vulnerabilities from prior work. That gives the security reasoning tasks something concrete to ground themselves in, making it more relevant than just general knowledge tests.

Elias: It grounds the task in real-world issues reported by others, which is important for testing how a model handles actual protocol flaws and requirements.

Nadia: The results show that without this full framework—the adaptation and the grounding—the base models struggle a lot with the normative style and logic of those specifications. But when you add those modules in, performance improves substantially across all the stages.

Elias: It proves that structuring the learning process into these specific steps actually helps improve standards comprehension, which is what we need for reliable security analysis.

Nadia: The paper also points out that their main finding in error analysis is that failures often come from missing evidence rather than just making a wrong logical jump. So they suggest adding a completeness verifier to make sure the model has all the pieces before it claims something.

Priya: That’s what I was thinking, because it makes sense. We need to enforce that evidence requirement so we don't trust conclusions that aren't fully supported by the retrieved data.

Elias: It highlights how compositional security reasoning is; you have to see all those dependencies together, not just isolate one clause or one figure for a quick answer.

Nadia: So, CellularS2-Bench isn’t just another test set; it’s proposing a way to systematically evaluate models against the rigorous demands of 3GPP specifications through this Adapt-Retrieve-Integrate approach <ref:2601.12716#pg1>.

Priya: It moves the goal from just testing if a model can read the text to seeing if it can reason securely about the protocol structure itself.

Elias: And that sets a new direction for how we need to build evaluation methods for these kinds of complex, multimodal engineering documents.

The paper's improvements: Nadia: So, we're talking about how they plan to fix the limitations of CellularS2-Bench right now, focusing on what comes next for this security testing setup.

Elias: Right. The paper talks about a few key ways they want to push this benchmark further out because the current scope is pretty limited to Release seventeen specs.

Nadia: They suggest expanding the coverage way beyond just one release, aiming to capture how specifications evolve across different versions over time and different standards bodies.

Elias: That makes sense. If you only test against one version, you don't really see if a model can generalize its understanding when the underlying rules change in a new spec.

Nadia: They also mentioned selectively integrating other state-of-the-art benchmarks from the wider cellular and telecom NLP literature into this system.

Elias: So they're not just relying on their own data, but they’re looking at things like TeleQnA or TSpec LLM to see how models perform on broader, existing benchmarks.

Priya: I wonder what that integration actually means for the data we trust. If you pull in stuff from different sources, how do you make sure everything stays consistent and actually reflects the core security requirements?

Elias: That’s a big caveat they have to deal with. They stress that any external datasets have to go through rigorous verification and correction to ensure they match the specification evidence perfectly.

Nadia: It means that while they are broadening the coverage, their focus remains on making sure whatever data goes into the system is high quality and directly tied back to specification evidence.

Priya: So it’s less about just adding more documents, and more about building a smarter filtering mechanism for external knowledge.

Elias: Exactly. It’s about combining the deep standards-ground reasoning with broader telecom knowledge in a way that stays reliable, which is the whole point of this selective integration strategy.

Nadia: Overall, it sounds like they're trying to build something that isn't just a snapshot of one moment in time, but a more dynamic tool for assessing model capabilities across the entire lifecycle of cellular standards.

Elias: And that brings us to the bigger picture: if we get this kind of structured benchmark working well, it changes how we even think about what’s possible to test against these complex specifications.

Conclusion: Nadia: So we’re wrapping up on CellularS2-Bench, which is this staged benchmark for cellular network security reasoning. It shows how to systematically test if language models can actually handle those complex 3GPP specs when they involve pictures and tables <ref:2601.12716#pg1>.

Elias: Right. The authors are showing that by using that Adapt, Retrieve, Integrate framework, you get a much better picture of a model's actual comprehension and security reasoning ability.

Priya: I just want to reiterate what we talked about regarding the data quality—that the strength of this benchmark really rests on how well they’ve curated those real-world vulnerability datasets.

Nadia: That’s true, Priya, because when you look at it, the biggest improvement is moving past just checking if a model can read a single paragraph to testing if it can actually reason across different clauses and figures.

Elias: It really does make you realize that for security analysis in standards, you need that whole chain of evidence assembled before the model makes any claim.

Priya: And that leads me to thinking about how we use this—it’s a tool for seeing where the models are still failing, like when the evidence set is incomplete.

Nadia: Exactly. The paper flags that dominant failures often come from missing evidence rather than just making a mistake in logic, so they suggest adding a completeness verifier to block conclusions unless all supporting evidence is present.

Elias: It emphasizes that security reasoning in these standards is compositional; you can't isolate one piece of the puzzle without seeing how it connects to everything else structurally.

Priya: So for someone listening who isn't deep in the technical details, this means we need to start looking at these kinds of benchmarks as a way to measure if an AI can actually follow protocol logic under stress.

Nadia: It’s a step toward making sure that when we deploy these models in safety-critical areas, we have a reproducible way to verify their understanding of the underlying rules.

Elias: And this work on CellularS2-Bench sets a new standard for how we can structure evaluation for these kinds of multimodal engineering documents.

Priya: It’s definitely a solid piece of work because it gives us that unified methodology we were missing to truly test standards-driven security reasoning.

Nadia: That’s the summary on CellularS2-Bench. We’ll be looking for more papers on how this framework helps tackle those complex, multimodal tasks next week.

Utah State University · University of Maryland, Baltimore County

cs.CR

Submitted: 2026-01-19

Updated: 2026-11-10

Comments: Accepted to the 40th Conference on Neural Information Processing Systems (NeurIPS 2026), Evaluations and Datasets Track

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 90/100

The gist: Cellular networks are critical infrastructure supporting billions of worldwide users and safety- and mission-critical services…

Key concepts

CellSpecSec-ARI Framework
A unified framework designed for systematic understanding of 3GPP specifications. It follows a three-stage approach: SpecFusion (adaptation), SpecRAG (retrieval), and SpecReasoning (integration). This structure mimics skill acquisition, moving from basic comprehension to expert synthesis across complex technical documents.
SpecFusion
The first stage of the framework that adapts Large Language Models to 3GPP domain knowledge. It decomposes specifications into text, table, and figure components. Tables and figures are converted into structured JSON descriptions, allowing the model to learn how rules are expressed in these non-textual formats.
SpecRAG
The retrieval module that provides evidence-grounded reasoning. It uses a hybrid search strategy combining sparse (BM25) and dense (TF-IDF vectors) retrieval to find relevant chunks from a database for any given query, ensuring the model has access to the most pertinent information.
SpecReasoning
The final stage that transforms retrieved evidence into a taxonomy-based knowledge graph. This graph explicitly models structural and procedural dependencies between specification entities (like procedures and messages), allowing for complex, cross-clause security analysis.

Terminology

Summary

Cellular networks are critical infrastructure supporting billions of worldwide users and safety- and mission-critical services, motivating growing efforts to analyze 3GPP specifications that define required device and operator behavior. p. 0

Where the paper says it

edu Abstract Cellular networks are critical infrastructure supporting billions of worldwide users and safetyand mission-critical services. Vulnerabilities in cellular networks can therefore cause service disruption, privacy breaches, and broad societal harm, motivating growing efforts to analyze 3GPP specifications that define required device and operator behavior.

To address the challenges of faithful interpretation of normative language, reasoning across cross-referenced clauses, and verifiable conclusions grounded in multimodal evidence such as tables and figures in cellular specifications, the authors propose CellSpecSec-ARI, a unified Adapt–Retrieve–Integrate framework for systematic understanding and standarddriven security analysis of 3GPP specifications. p. 0
Where the paper says it

While large language models (LLMs) have demonstrated the capability for reading technical documents, cellular specifications impose unique challenges: faithful interpretation of normative language, reasoning across cross-referenced clauses, and verifiable conclusions grounded in multimodal evidence such as tables and figures. To address these challenges, we propose CellSpecSec-ARI, a unified Adapt–Retrieve–Integrate framework for systematic understanding and standarddriven security analysis of 3GPP specifications; CellularSpecSec-Bench, a staged benchmark, containing newly constructed highquality datasets with expert-verified and corrected subsets from prior open-source resources.

They establish CellularSpecSec-Bench, a staged benchmark containing newly constructed highquality datasets with expert-verified and corrected subsets from prior open-source resources to quantify progress in specification understanding and security reasoning in the cellular network security domain. p. 0
Where the paper says it

To address these challenges, we propose CellSpecSec-ARI, a unified Adapt–Retrieve–Integrate framework for systematic understanding and standarddriven security analysis of 3GPP specifications; CellularSpecSec-Bench, a staged benchmark, containing newly constructed highquality datasets with expert-verified and corrected subsets from prior open-source resources. Together, they establish an accessible and reproducible foundation for quantifying progress in specification understanding and security reasoning in the cellular network security domain.

The CellSpecSec-ARI Framework

CellSpecSec-ARI is a unified framework designed to solve standardized tasks in (1) interpreting complex 3GPP specifications written in domain-specific language with multimodal context, and (2) performing the challenging security analysis over cellular network standards. This framework follows a three-stage architecture inspired by the Dreyfus model of skill acquisition: Adapt corresponds to novice-level rule-based comprehension, Retrieve aligns with competent, evidence-guided decision making, and Integrate reflects expert-level synthesis across multiple sources. p. 2

Where the paper says it

This three-stage framework is inspired by the Dreyfus model of skill acquisition (Dreyfus et al. , 1986): the Adapt stage corresponds to novice-level rule-based comprehension, Retrieve aligns with competent, evidence-guided decision making, and Integrate reflects expert-level synthesis across multiple sources.

The three modules are SpecFusion (Adaption), SpecRAG (Retrieval), and SpecReasoning (Integration)

Multimodal Specification Adaptation

The first module, SpecFusion, enables LLMs to adapt to the domain knowledge in 3GPP specifications by decomposing them into textual components (Ctext), table components (Ctable), and figure components (Cfigure). SpecFusion transforms non-textual components into a unified textual representation by converting tables and figures into structured JSON descriptions. p. 2

Where the paper says it

Instead of applying a specific encoder for each modality, SpecFusion transforms non-textual components into a unified textual representation. Specifically, tables and figures in 3GPP specifications are extracted and converted into structured JSON descriptions as shown in Appendix B.

This allows the framework to learn how rules and mappings are expressed in the tables of 3GPP specifications. p. 2
Where the paper says it

Table components (Ctable) represent normative tables that encode structured relationships among parameters, states, timers, configuration options, and actions. Including tabular chunks in the adaptation process allows CellSpecSec-ARI to learn how rules and mappings are expressed in the tables of 3GPP specifications.

Evidence-Grounding via SpecRAG

The SpecRAG module performs retrieval-augmented reasoning by researching and returning a set of relevant evidence chunks from its RAG database for each query. It uses a hybrid retrieval strategy combining sparse component ssp(q, c) using BM25 ranker and dense component sde(q, c) obtained via TF–IDF vectors projected into a lower-dimensional latent space. p. 3

Where the paper says it

The sparse component ssp(q, c) is computed using a BM25 ranker that captures exact keyword matches between the query q and candidate chunk c. The dense component sde(q, c) is obtained by encoding the query and each chunk into TF–IDF vectors, projecting them into a lower-dimensional latent space via truncated SVD, and computing cosine similarity between the resulting vectors.

The top-k highest-scoring chunks form the Retrieved Evidence context supplied to the model. p. 3
Where the paper says it

The final relevance score is a weighted combination: s(q, c) = α ssp(q, c)+(1−α) sde(q, c), α ∈ [0, 1], where α is tuned on a small validation set to balance lexical precision and semantic recall. The top-k highest-scoring chunks form the Retrieved Evidence context supplied to the model.

Taxonomy-Based Knowledge Integration

The SpecReasoning module transforms multimodal outputs from SpecRAG into a taxonomy-based knowledge graph (KG) that explicitly models structural and procedural dependencies among 3GPP specification entities. This KG uses entity types like Procedure, Message, Information Element, Identifier, Timer, State, Condition, and Property Entity types for taxonomy-based KG. p. 4

Where the paper says it

By systematically analyzing each clause, including all multimodal components, we identify the recurring entities from the majority of clauses and summarize a set of entity types as shown in Table 2: signaling procedures, messages, information elements, identifiers, timers, states, conditions, and system-level properties. They have consistent meanings across 3GPP specifications and form the basic entity vocabulary used in the taxonomy-based KGs.

We first construct an entity taxonomy that reflects the hierarchical organization of 3GPP specifications by systematically analyzing each clause and summarizing recurring entities. p. 4
Where the paper says it

We first construct an entity taxonomy that reflects the hierarchical organization of 3GPP specifications. By systematically analyzing each clause, including all multimodal components, we identify the recurring entities from the majority of clauses and summarize a set of entity types as shown in Table 2: signaling procedures, messages, information elements, identifiers, timers, states, conditions, and system-level properties.

Relations are defined into Core relations Rcore representing structural dependencies and Security relations Rsec capturing dependencies related to protection requirements. p. 4
Where the paper says it

Two complementary families of relations are defined. Core relations Rcore represent the structural dependencies that govern protocol execution logic.

Benchmark Construction and Evaluation

CellularSpecSec-Bench covers core specifications in Release-17, processing over 1,760 paragraphs, 13,800 sentences, 424 figures, and 340 tables CellularSpecSec-Bench covers the core specifications in Release-17 (3GP, 2022a), (3GP, 2022b), (3GP, 2020), processing over 1,760 paragraphs, 13,800 sentences, 424 figures, and 340 tables The security reasoning dataset incorporates 43 real-world vulnerabilities reported in prior works (Shaik et al., 2015; Bassil et al. p. 1

Where the paper says it

The security reasoning dataset incorporates 43 real-world vulnerabilities reported in prior works (Shaik et al. , 2015; Bassil et al.

p. 1
Where the paper says it

The security reasoning dataset incorporates 43 real-world vulnerabilities reported in prior works (Shaik et al. , 2015; Bassil et al.

, 2013; Kambourakis et al. p. 1
Where the paper says it

, 2015; Bassil et al. , 2013; Kambourakis et al.

p. 1
Where the paper says it

, 2015; Bassil et al. , 2013; Kambourakis et al.

, 2011; Lee et al. p. 1
Where the paper says it

, 2013; Kambourakis et al. , 2011; Lee et al.

p. 1
Where the paper says it

, 2013; Kambourakis et al. , 2011; Lee et al.

, 2009; Leong et al. p. 1
Where the paper says it

, 2011; Lee et al. , 2009; Leong et al.

p. 1
Where the paper says it

, 2011; Lee et al. , 2009; Leong et al.

, 2014; Kim et al. p. 1
Where the paper says it

, 2009; Leong et al. , 2014; Kim et al.

p. 1
Where the paper says it

, 2009; Leong et al. , 2014; Kim et al.

, 2019; Van Den Broek et al. p. 1
Where the paper says it

, 2014; Kim et al. , 2019; Van Den Broek et al.

p. 1
Where the paper says it

, 2014; Kim et al. , 2019; Van Den Broek et al.

, 2015; Park et al. p. 1
Where the paper says it

, 2019; Van Den Broek et al. , 2015; Park et al.

p. 1
Where the paper says it

, 2019; Van Den Broek et al. , 2015; Park et al.

, 2016; Chlosta et al. p. 1
Where the paper says it

, 2015; Park et al. , 2016; Chlosta et al.

p. 1
Where the paper says it

, 2015; Park et al. , 2016; Chlosta et al.

, 2019; Yu and Chen, 2019; Michau and Devine, 2016; Hussain et al. p. 1
Where the paper says it

, 2016; Chlosta et al. , 2019; Yu and Chen, 2019; Michau and Devine, 2016; Hussain et al.

p. 1
Where the paper says it

, 2016; Chlosta et al. , 2019; Yu and Chen, 2019; Michau and Devine, 2016; Hussain et al.

, 2019b; Borgaonkar et al. p. 1
Where the paper says it

, 2019; Yu and Chen, 2019; Michau and Devine, 2016; Hussain et al. , 2019b; Borgaonkar et al.

p. 1
Where the paper says it

, 2019; Yu and Chen, 2019; Michau and Devine, 2016; Hussain et al. , 2019b; Borgaonkar et al.

, 2018; Cao et al. p. 1
Where the paper says it

, 2019b; Borgaonkar et al. , 2018; Cao et al.

p. 1
Where the paper says it

, 2019b; Borgaonkar et al. , 2018; Cao et al.

, 2020; Chlosta et al. p. 1
Where the paper says it

, 2018; Cao et al. , 2020; Chlosta et al.

p. 1
Where the paper says it

, 2018; Cao et al. , 2020; Chlosta et al.

, 2021), capturing both textual and multimodal specification semantics The security reasoning dataset incorporates 43 real-world vulnerabilities reported in prior works (Shaik et al. p. 1
Where the paper says it

CellularSpecSec-Bench covers the core specifications in Release-17 (3GP, 2022a), (3GP, 2022b), (3GP, 2020), processing over 1,760 paragraphs, 13,800 sentences, 424 figures, and 340 tables, capturing both textual and multimodal specification semantics. The security reasoning dataset incorporates 43 real-world vulnerabilities reported in prior works (Shaik et al.

, 2015; Bassil et al., 2013; Kambourakis et al., 2011; Lee et al., 2009; Leong et al., 2014; Kim et al., 2019; Van Den Broek et al., 2015; Park et al., 2016; Chlosta et al., 2019; Yu and Chen, 2019; Michau and Devine, 2016; Hussain et al., 2019b; Borgaonkar et al., 2018; Cao et al., 2020; Chlosta et al., 2021), capturing both textual and multimodal specification semantics>. p. 1
Where the paper says it

CellSpecSec-ARI is a unified framework to solve standardized tasks in (1) interpreting complex 3GPP specifications written in domain-specific language with multimodal context, and (2) performing the challenging security analysis over cellular network standards. CellularSpecSec-Bench covers the core specifications in Release-17 (3GP, 2022a), (3GP, 2022b), (3GP, 2020), processing over 1,760 paragraphs, 13,800 sentences, 424 figures, and 340 tables, capturing both textual and multimodal specification semantics.

The benchmark includes expert-constructed security reasoning tasks grounded in real-world vulnerabilities, alongside selectively integrated compatible tasks from prior datasets after careful correctness validation It not only includes expert-constructed security reasoning tasks grounded in real-world vulnerabilities, but also selectively integrates compatible tasks from prior datasets after careful correctness validation

Performance Gains and Challenges

The evaluation shows that the base model performs poorly on most tasks, indicating struggles with the normative style, precision, and logic in 3GPP specifications The base model performs poorly on most tasks, indicating that the base model struggles with the normative style, precision, and logic in 3GPP specifications>. Incorporating CellSpecSec-ARI modules substantially improves performance across all tasks Incorporating CellSpecSec-ARI modules substantially improves performance, demonstrating that the domain adaptation and structured reasoning improve standards comprehension and reasoning capability>. p. 6

Where the paper says it

The base model performs poorly on most tasks, indicating that the base model struggles with the normative style, precision, and logic in 3GPP specifications. Incorporating CellSpecSec-ARI modules substantially improves performance, demonstrating that the domain adaptation and structured reasoning improve standards comprehension and reasoning capability.

Effective evidence grounding via retrieval augmentation significantly improves answer quality when tasks require evidence Enabling SpecRAG improves both answer quality>

Error Analysis Insights

Error analysis reveals that the dominant failures in security-focused tasks stem from incomplete evidence rather than purely incorrect security reasoning For security-focused tasks, vulnerability/inconsistency labeling, the dominant failures stem from incomplete evidence, rather than purely incorrect security reasoning>. Explanations are often directionally correct but incomplete because the cited evidence set is missing critical dependencies Explanations are directionally correct but incomplete, and the cited evidence set is missing one or more critical dependencies>. p. 7

Where the paper says it

For security-focused tasks, vulnerability/inconsistency labeling, the dominant failures stem from incomplete evidence, rather than purely incorrect security reasoning. Explanations are directionally correct but incomplete, and the cited evidence set is missing one or more critical dependencies.

This emphasizes that security reasoning in standards is compositional This patterns emphasize that security reasoning in standards is compositional>. p. 7
Where the paper says it

Explanations are directionally correct but incomplete, and the cited evidence set is missing one or more critical dependencies. These patterns emphasize that security reasoning in standards is compositional.

The authors suggest mitigating this by adding a completeness verifier to reject conclusions unless the assembled evidence jointly supports the claim To mitigate this, we can add a completeness verifier that rejects conclusions or causal explanations unless the assembled evidence jointly supports the claim>. p. 7
Where the paper says it

These patterns emphasize that security reasoning in standards is compositional. To mitigate this, we can add a completeness verifier that rejects conclusions or causal explanations unless the assembled evidence jointly supports the claim.

Benchmark Scope and Limitations

CellularSpecSec-Bench focuses on a limited set of 3GPP specifications in Release 17, which enables controlled evaluation but does not cover the full breadth of the corpus First, CellularSpecSec-Bench focuses on a limited set of 3GPP specifications in Release 17. While this design scope enables controlled and reproducible evaluation, it does not cover the full breadth of the 3GPP corpus or reflect the ongoing evolution of specifications across releases>. p. 8

Where the paper says it

First, CellularSpecSec-Bench focuses on a limited set of 3GPP specifications in Release 17. While this design scope enables controlled and reproducible evaluation, it does not cover the full breadth of the 3GPP corpus or reflect the ongoing evolution of specifications across releases.

Furthermore, external datasets are curated through verification and correction to ensure consistency with specification evidence To broaden coverage and support additional task formats, CellularSpecSec-Bench selectively integrates stateof-the-art benchmarks from the telecom and cellular NLP literature. p. 4
Where the paper says it

Verified and corrected datasets. To broaden coverage and support additional task formats, CellularSpecSec-Bench selectively integrates stateof-the-art benchmarks from the telecom and cellular NLP literature.

It includes TeleQnA (Maa-touk et al. p. 4
Where the paper says it

To broaden coverage and support additional task formats, CellularSpecSec-Bench selectively integrates stateof-the-art benchmarks from the telecom and cellular NLP literature. It includes TeleQnA (Maa-

, 2023), Telco-DPR (Saraiva et al. p. 5
Where the paper says it

touk et al. , 2023), Telco-DPR (Saraiva et al.

, 2024), TSpec-LLM (Nikbakht et al. p. 1
Where the paper says it

TSpec-LLM (Nikbakht et al. , 2024) releases a large-scale corpus of 3GPP specifications (Release 8–19).

, 2024), and TeleLLMs (Maatouk et al. p. 5
Where the paper says it

, 2024), TSpec-LLM (Nikbakht et al. , 2024), and TeleLLMs (Maatouk et al.

, 2025) for assessing how models understand 3GPP specifications Through this selective integration and expert verification, CellularSpecSec-Bench combines standards-grounded control-plane reasoning with broader telecom knowledge, resulting in a benchmark that is more comprehensive and reliable than prior datasets in this domain>. p. 5
Where the paper says it

Through this selective integration and expert verification, CellularSpecSec-Bench combines standards-grounded control-plane reasoning with broader telecom knowledge, resulting in a benchmark that is more comprehensive and reliable than prior datasets in this domain. 1 Task Types We formalize the tasks involved in comprehending 3GPP specifications and identifying security design vulnerabilities into three types.

The work also notes that the experiments apply DeepSeek-V3. p. 8
Where the paper says it

While this strict evaluation aligns with the goal of trustworthy specification reasoning, it may under-credit partially correct evidence or alternative but valid supporting sets. Last, while this work focuses on establishing the first unified framework CellSpecSec-ARI and developing the reproducible CellularSpecSec-Bench, our experiments apply DeepSeek-V3.

2-Exp as the base model While this work focuses on establishing the first unified framework CellSpecSec-ARI and developing the reproducible CellularSpecSec-Bench, our experiments apply DeepSeek-V3. p. 8
Where the paper says it

Last, while this work focuses on establishing the first unified framework CellSpecSec-ARI and developing the reproducible CellularSpecSec-Bench, our experiments apply DeepSeek-V3. 2-Exp as the base model.

2-Exp as the base model>. p. 6
Where the paper says it

We evaluate two configurations: (1) Base, which uses the raw DeepSeek-V3. 2-Exp model without any CellSpecSec-ARI modules, and (2) CellSpecSec-ARI, which augments the same underlying model with the pipeline enabled module by module.

Conclusion

This work introduces CellSpecSec-ARI, a unified framework supporting domain adaptation to 3GPP specifications, retrieval of multimodal evidence with verifiable citations, and integration reasoning for security analysis We introduce CellSpecSec-ARI, a unified framework that supports domain adaptation to 3GPP specifications, retrieval of multimodal evidence with verifiable citations, and integration reasoning for security analysis>. The authors establish CellularSpecSec-Bench as a staged benchmark spanning clause-local comprehension, evidencegrounded answering, and cross-clause, securityrelevant reasoning We establish CellularSpecSec-Bench a staged benchmark spanning clause-local comprehension, evidencegrounded answering, and cross-clause, securityrelevant reasoning>. p. 7

Where the paper says it

We introduce CellSpecSec-ARI, a unified framework that supports domain adaptation to 3GPP specifications, retrieval of multimodal evidence with verifiable citations, and integration reasoning for security analysis. We establish CellularSpecSec-Bench a staged benchmark spanning clause-local comprehension, evidencegrounded answering, and cross-clause, securityrelevant reasoning, built from core specifications and complemented with carefully verified and corrected subsets from prior datasets.

This work addresses a key gap in cellular security research: the lack of a unified methodology and reproducible benchmark for evaluating 3GPP specification understanding and standards-driven security reasoning This work addresses a key gap in cellular security research: the lack of a unified methodology and reproducible benchmark for evaluating 3GPP specification understanding and standards-driven security reasoning>. p. 7
Where the paper says it

To mitigate this, we can add a completeness verifier that rejects conclusions or causal explanations unless the assembled evidence jointly supports the claim. 7 Conclusion This work addresses a key gap in cellular security research: the lack of a unified methodology and reproducible benchmark for evaluating 3GPP specification understanding and standards-driven security reasoning.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Adaption - SpecFusion improves LLMs by enabling them to adapt to domain knowledge in 3GPP specifications by transforming non-textual components like tables and figures into a unified textual representation as structured JSON descriptions. This allows the model to learn how rules and mappings are expressed in the tables of 3GPP specifications.

  2. Retrieval - SpecRAG enhances retrieval-augmented reasoning by using a hybrid retrieval strategy that combines sparse BM25 ranking with dense TF-IDF embeddings projected into a lower-dimensional latent space via truncated SVD for semantic similarity search. This ensures that the retrieved evidence is both lexically precise and semantically relevant to the query.

  3. Integration - SpecReasoning enables complex reasoning by transforming multimodal outputs from SpecRAG into a taxonomy-based knowledge graph (KG) that explicitly models structural and procedural dependencies among 3GPP specification entities, including distinct core and security relation types. This allows the system to perform cross-clause integration necessary for security analysis, as demonstrated by its ability to construct a SpecGraph instance.

  4. System capabilities: The CellSpecSec-ARI framework enables systematic analysis for 3GPP specifications by supporting (1) interpreting complex 3GPP specifications written in domain-specific language with multimodal context, and (2) performing the challenging security analysis over cellular network standards.

Abstract

Cellular networks are critical infrastructure supporting billions of worldwide users and safety- and mission-critical services. Vulnerabilities in cellular networks can therefore cause service disruption, privacy breaches, and broad societal harm, motivating growing efforts to analyze 3GPP specifications that define required device and operator behavior. However, existing benchmarks for this domain share three systematic deficiencies: they rely on LLM-generated entries without expert verification, they do not require faithful interpretation of normative language, reasoning across cross-referenced clauses, and verifiable conclusions grounded in multimodal evidence such as tables and figures, and they do not evaluate security reasoning over end-to-end protocol behavior. To address these gaps, we present CellularS2-Bench, a staged benchmark. CellularS2-Bench covers core control-plane specifications, processing over 1,866 paragraphs, 22,033 sentences, 424 figures, and 248 tables. It comprises 13,851 questions and instances across three progressive stages, of which 10,269 (74.14%) are newly constructed from 3GPP specifications and 43 real-world vulnerabilities. Beyond newly constructed tasks, CellularS2-Bench integrates and corrects 3,582 (25.86%) instances from 7 prior datasets to broaden task coverage. To establish a reproducible baseline on CellularS2-Bench, we introduce CellularS2-ARI, a unified Adapt-Retrieve-Integrate framework for systematic understanding and specification-driven security analysis of 3GPP specifications. Together, they establish an accessible and reproducible foundation for quantifying progress in specification understanding and security reasoning in cellular network security.

Sources

Related papers