CellularS2-Bench: A Staged, Evidence-Grounded Benchmark for Cellular Network Security Reasoning
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.
Nadia: I'm Nadia.
Elias: And I'm Elias, here with Priya.
Nadia: We're looking at the paper: "CellularS2-Bench: A Staged, Evidence-Grounded Benchmark for Cellular Network Security Reasoning".
Elias: Here's the gist of the paper. Cellular networks are critical infrastructure supporting billions of worldwide users and safety- and mission-critical services, motivating growing efforts to analyze 3GPP specifications that define required device and operator behavior.
Nadia: To address the challenges of faithful interpretation of normative language, reasoning across cross-referenced clauses, and verifiable conclusions grounded in multimodal evidence such as tables and figures in cellular specifications, the authors propose CellSpecSec-ARI, a unified Adapt–Retrieve–Integrate framework for systematic understanding and standarddriven security analysis of 3GPP specifications.
Elias: They establish CellularSpecSec-Bench, a staged benchmark containing newly constructed highquality datasets with expert-verified and corrected subsets from prior open-source resources to quantify progress in specification understanding and security reasoning in the cellular network security domain.
Nadia: The CellSpecSec-ARI Framework CellSpecSec-ARI is a unified framework designed to solve standardized tasks in (1) interpreting complex 3GPP specifications written in domain-specific language with multimodal context, and (2) performing the challenging security analysis over cellular network standards.
Title and authors: Nadia: So, we're looking at the paper called CellularS2-Bench: A Staged, Evidence-Grounded Benchmark for Cellular Network Security Reasoning. This whole thing is about building a way to test how well models actually understand and can reason about those really complex 3GPP specifications <ref:2601.12716#pg1>.
Elias: Right. It’s tackling the difficulty of interpreting that dense, normative language and making sure the answers are actually grounded in the evidence, not just guessing based on what sounds right.
Nadia: Exactly. They introduce this whole framework called CellSpecSec-ARI, which has three stages: Adapt, Retrieve, and Integrate. It’s like they're trying to teach a model everything from reading a simple rule to synthesizing information across all the different parts of the specification at an expert level.
Priya: From my side, I'm wondering if this structure actually captures how security reasoning works in practice. Are we just looking at surface-level comprehension, or is this framework actually testing if the model can follow the chain of dependencies?
Elias: That’s a fair point, Priya. The Adapt part, SpecFusion, seems designed to get the model familiar with the domain first by turning those tables and figures into text descriptions. It helps it learn how rules are written in those non-textual formats.
Nadia: Right, so SpecFusion is about adapting to the multimodal nature of the specs. Then you have SpecRAG, which does the actual evidence retrieval, using a hybrid search method that mixes keyword matching with dense vector embeddings for semantic similarity.
Priya: I'm thinking about that retrieval part—how they’re trying to get the right chunks when you have thousands of pages and figures. Does this hybrid approach actually help filter out the noise effectively?
Elias: The hybrid search uses sparse component ssp(q, c) with a BM25 ranker for lexical precision, combined with dense component sde(q, c) from TF-IDF vectors projected into a lower-dimensional latent space for semantic relevance. That way, you get both precise terms and the broader meaning of what you’re asking.
Nadia: And then SpecReasoning takes all that retrieved evidence and turns it into a knowledge graph. This graph maps out all the connections between entities like procedures, messages, and security properties, defining core relations and specific security relations.
Title and authors: Priya: That sounds like a big step toward modeling the structure of the standard itself. So instead of just getting an answer, we’re trying to see if it can map out the procedural logic of the cellular network for us.
Elias: Precisely. They construct this taxonomy by first analyzing every clause in a specification to figure out how things are organized hierarchically before they define those relations for the reasoning module.
Nadia: And then they use that graph to perform security analysis over cellular network standards, which is the ultimate goal here. It moves beyond just understanding a single sentence or procedure.
Priya: But what about the benchmark itself? The paper mentions CellularSpecSec-Bench covers core specifications from Release seventeen processing over one thousand seven hundred sixty paragraphs and three hundred forty tables <ref:2601.12716#pg2,processing over 1,760 paragraphs>. That’s a substantial amount of material for one test set.
Elias: Yeah, that scale is important because it shows they’re not just testing on tiny snippets. They are testing against a real volume of specification content to see how the model handles cross-clause reasoning over that entire context.
Nadia: It also includes forty-three real-world vulnerabilities reported in prior work, which is what gives the security reasoning tasks their grounding and relevance in the actual cellular network world.
Priya: So, while they’ve built this large dataset, I need to ask about its limitations. The paper does mention that prior datasets focused more on sentence or paragraph level understanding rather than standards-driven procedural reasoning that needs evidence across multiple clauses and figures. Does CellularS2-Bench fully address that gap?
Elias: They explicitly state that the prior work had limitations, so this benchmark is designed to move past those. It aims to evaluate evidence-grounded performance, which they call critical for complex reasoning over procedures and security properties.
Nadia: The authors also flag that their security focus isn't just on text classification; it’s about reasoning about end-to-end protocol behavior under adversarial exploitation conditions, which is a tougher test.
Priya: That’s interesting because the evaluation shows that incorporating the CellSpecSec-ARI modules substantially improves performance across all tasks compared to the base model. That tells us the structure of their framework actually helps overcome those initial struggles with normative style and logic.
Title and authors: Elias: It suggests that breaking down the task into adaptation, retrieval, and integration stages provides a clear path for improving standards comprehension systematically. The error analysis also pointed out that failures often come from incomplete evidence rather than totally wrong reasoning.
Nadia: So the biggest lesson there is that if you want reliable security analysis on these specs, you need to ensure the model has all the necessary pieces of evidence assembled before it makes a claim.
Priya: That means we need to look closely at how much context is being retrieved and integrated during those retrieval and reasoning steps. The completeness verifier they suggest seems like a good idea for enforcing that evidence requirement.
Elias: It definitely emphasizes that security reasoning in standards is compositional, which means you can’t just look at one clause in isolation; you have to see how it connects structurally to everything else.
Nadia: So, CellularS2-Bench is presented as a way to quantify this progress, giving the community a reproducible standard for evaluating these systems. It moves the conversation from "can the LLM read this?" to "can the LLM reason securely about this?"
Priya: And that’s what matters for real-world deployment. A benchmark that tests against real vulnerabilities helps us actually build better tools and defenses, instead of just guessing at their capabilities.
Elias: So, wrapping up on CellularS2-Bench, it sets a new standard for how we can systematically evaluate models against the rigorous demands of 3GPP specifications through this Adapt-Retrieve-Integrate approach <ref:2601.12716#pg1>.
Nadia: It’s a solid piece of work because it gives us a unified methodology for tackling those multimodal, complex tasks in cellular security analysis.
Priya: I just hope they keep expanding that coverage beyond Release seventeen to really reflect the evolving standards across different releases <ref:2601.12716#pg2>.
Elias: That’s the direction they suggest, and building upon this foundation means we can start comparing models on more challenging reasoning scenarios in the future.
Nadia: We'll stick around to see how these benchmarks evolve and what kind of security insights they unlock for us down the line.
The paper's summary: Nadia: So, we just looked at how they built this benchmark called CellularS2-Bench, and what it’s really trying to do for testing these language models against cell network security.
Elias: Right. It’s basically a structured way to see if a model can actually reason about those dense 3GPP standards when there are pictures and tables involved, not just reading sentences <ref:2601.12716#pg1>.
Nadia: Exactly. The core idea is this three-stage framework, Adapt, Retrieve, Integrate. They’re trying to mimic how an expert learns something—from basic understanding to deep synthesis across all the pieces of a standard.
Elias: That adaptation part they call SpecFusion is where they teach the model the domain knowledge by turning those non-textual bits like tables and figures into structured text descriptions. It shows it how those rules are actually written in that format.
Nadia: Then you have retrieval, SpecRAG, which uses a hybrid search—keyword matching mixed with dense vector embeddings—to pull out the exact evidence chunks needed for the question at hand.
Elias: And then SpecReasoning takes all that retrieved evidence and turns it into a knowledge graph. This graph maps out all the structural connections between things like procedures and security properties, which lets it do that cross-clause integration you mentioned earlier.
Nadia: It’s really about moving past simple comprehension to complex reasoning about protocol behavior under attack conditions, which is what they’re focusing on for the security analysis tasks.
Elias: And the benchmark itself is built on a huge chunk of Release seventeen specs, with over seventeen hundred paragraphs and four hundred twenty-four figures to test against. That scale is what makes it feel more like real-world testing than just looking at a few examples.
Nadia: But Priya, you were concerned about the data quality. What does this benchmark actually show us about the model’s reasoning compared to just having a big test set?
Priya: Well, they included forty-three real vulnerabilities from prior work. That gives the security reasoning tasks something concrete to ground themselves in, making it more relevant than just general knowledge tests.
Elias: It grounds the task in real-world issues reported by others, which is important for testing how a model handles actual protocol flaws and requirements.
Nadia: The results show that without this full framework—the adaptation and the grounding—the base models struggle a lot with the normative style and logic of those specifications. But when you add those modules in, performance improves substantially across all the stages.
Elias: It proves that structuring the learning process into these specific steps actually helps improve standards comprehension, which is what we need for reliable security analysis.
Nadia: The paper also points out that their main finding in error analysis is that failures often come from missing evidence rather than just making a wrong logical jump. So they suggest adding a completeness verifier to make sure the model has all the pieces before it claims something.
Priya: That’s what I was thinking, because it makes sense. We need to enforce that evidence requirement so we don't trust conclusions that aren't fully supported by the retrieved data.
Elias: It highlights how compositional security reasoning is; you have to see all those dependencies together, not just isolate one clause or one figure for a quick answer.
Nadia: So, CellularS2-Bench isn’t just another test set; it’s proposing a way to systematically evaluate models against the rigorous demands of 3GPP specifications through this Adapt-Retrieve-Integrate approach <ref:2601.12716#pg1>.
Priya: It moves the goal from just testing if a model can read the text to seeing if it can reason securely about the protocol structure itself.
Elias: And that sets a new direction for how we need to build evaluation methods for these kinds of complex, multimodal engineering documents.
The paper's improvements: Nadia: So, we're talking about how they plan to fix the limitations of CellularS2-Bench right now, focusing on what comes next for this security testing setup.
Elias: Right. The paper talks about a few key ways they want to push this benchmark further out because the current scope is pretty limited to Release seventeen specs.
Nadia: They suggest expanding the coverage way beyond just one release, aiming to capture how specifications evolve across different versions over time and different standards bodies.
Elias: That makes sense. If you only test against one version, you don't really see if a model can generalize its understanding when the underlying rules change in a new spec.
Nadia: They also mentioned selectively integrating other state-of-the-art benchmarks from the wider cellular and telecom NLP literature into this system.
Elias: So they're not just relying on their own data, but they’re looking at things like TeleQnA or TSpec LLM to see how models perform on broader, existing benchmarks.
Priya: I wonder what that integration actually means for the data we trust. If you pull in stuff from different sources, how do you make sure everything stays consistent and actually reflects the core security requirements?
Elias: That’s a big caveat they have to deal with. They stress that any external datasets have to go through rigorous verification and correction to ensure they match the specification evidence perfectly.
Nadia: It means that while they are broadening the coverage, their focus remains on making sure whatever data goes into the system is high quality and directly tied back to specification evidence.
Priya: So it’s less about just adding more documents, and more about building a smarter filtering mechanism for external knowledge.
Elias: Exactly. It’s about combining the deep standards-ground reasoning with broader telecom knowledge in a way that stays reliable, which is the whole point of this selective integration strategy.
Nadia: Overall, it sounds like they're trying to build something that isn't just a snapshot of one moment in time, but a more dynamic tool for assessing model capabilities across the entire lifecycle of cellular standards.
Elias: And that brings us to the bigger picture: if we get this kind of structured benchmark working well, it changes how we even think about what’s possible to test against these complex specifications.
Conclusion: Nadia: So we’re wrapping up on CellularS2-Bench, which is this staged benchmark for cellular network security reasoning. It shows how to systematically test if language models can actually handle those complex 3GPP specs when they involve pictures and tables <ref:2601.12716#pg1>.
Elias: Right. The authors are showing that by using that Adapt, Retrieve, Integrate framework, you get a much better picture of a model's actual comprehension and security reasoning ability.
Priya: I just want to reiterate what we talked about regarding the data quality—that the strength of this benchmark really rests on how well they’ve curated those real-world vulnerability datasets.
Nadia: That’s true, Priya, because when you look at it, the biggest improvement is moving past just checking if a model can read a single paragraph to testing if it can actually reason across different clauses and figures.
Elias: It really does make you realize that for security analysis in standards, you need that whole chain of evidence assembled before the model makes any claim.
Priya: And that leads me to thinking about how we use this—it’s a tool for seeing where the models are still failing, like when the evidence set is incomplete.
Nadia: Exactly. The paper flags that dominant failures often come from missing evidence rather than just making a mistake in logic, so they suggest adding a completeness verifier to block conclusions unless all supporting evidence is present.
Elias: It emphasizes that security reasoning in these standards is compositional; you can't isolate one piece of the puzzle without seeing how it connects to everything else structurally.
Priya: So for someone listening who isn't deep in the technical details, this means we need to start looking at these kinds of benchmarks as a way to measure if an AI can actually follow protocol logic under stress.
Nadia: It’s a step toward making sure that when we deploy these models in safety-critical areas, we have a reproducible way to verify their understanding of the underlying rules.
Elias: And this work on CellularS2-Bench sets a new standard for how we can structure evaluation for these kinds of multimodal engineering documents.
Priya: It’s definitely a solid piece of work because it gives us that unified methodology we were missing to truly test standards-driven security reasoning.
Nadia: That’s the summary on CellularS2-Bench. We’ll be looking for more papers on how this framework helps tackle those complex, multimodal tasks next week.
Utah State University · University of Maryland, Baltimore County
cs.CR
Submitted: 2026-01-19
Updated: 2026-11-10
Comments: Accepted to the 40th Conference on Neural Information Processing Systems (NeurIPS 2026), Evaluations and Datasets Track
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 90/100
The gist: Cellular networks are critical infrastructure supporting billions of worldwide users and safety- and mission-critical services…
Key concepts
- CellSpecSec-ARI Framework
- A unified framework designed for systematic understanding of 3GPP specifications. It follows a three-stage approach: SpecFusion (adaptation), SpecRAG (retrieval), and SpecReasoning (integration). This structure mimics skill acquisition, moving from basic comprehension to expert synthesis across complex technical documents.
- SpecFusion
- The first stage of the framework that adapts Large Language Models to 3GPP domain knowledge. It decomposes specifications into text, table, and figure components. Tables and figures are converted into structured JSON descriptions, allowing the model to learn how rules are expressed in these non-textual formats.
- SpecRAG
- The retrieval module that provides evidence-grounded reasoning. It uses a hybrid search strategy combining sparse (BM25) and dense (TF-IDF vectors) retrieval to find relevant chunks from a database for any given query, ensuring the model has access to the most pertinent information.
- SpecReasoning
- The final stage that transforms retrieved evidence into a taxonomy-based knowledge graph. This graph explicitly models structural and procedural dependencies between specification entities (like procedures and messages), allowing for complex, cross-clause security analysis.
Terminology
Summary
Cellular networks are critical infrastructure supporting billions of worldwide users and safety- and mission-critical services, motivating growing efforts to analyze 3GPP specifications that define required device and operator behavior. p. 0 edu
Abstract
Cellular networks are critical infrastructure supporting billions of worldwide users and safetyand mission-critical services. Vulnerabilities
in cellular networks can therefore cause service disruption, privacy breaches, and broad
societal harm, motivating growing efforts to analyze 3GPP specifications that define required
device and operator behavior. While large language models (LLMs) have demonstrated the
capability for reading technical documents, cellular specifications impose unique challenges:
faithful interpretation of normative language,
reasoning across cross-referenced clauses, and
verifiable conclusions grounded in multimodal
evidence such as tables and figures. To address
these challenges, we propose CellSpecSec-ARI,
a unified Adapt–Retrieve–Integrate framework
for systematic understanding and standarddriven security analysis of 3GPP specifications; CellularSpecSec-Bench, a staged benchmark, containing newly constructed highquality datasets with expert-verified and corrected subsets from prior open-source resources. To address
these challenges, we propose CellSpecSec-ARI,
a unified Adapt–Retrieve–Integrate framework
for systematic understanding and standarddriven security analysis of 3GPP specifications; CellularSpecSec-Bench, a staged benchmark, containing newly constructed highquality datasets with expert-verified and corrected subsets from prior open-source resources. Together, they establish an accessible
and reproducible foundation for quantifying
progress in specification understanding and security reasoning in the cellular network security
domain.Where the paper says it
Where the paper says it
Where the paper says it
The CellSpecSec-ARI Framework
CellSpecSec-ARI is a unified framework designed to solve standardized tasks in (1) interpreting complex 3GPP specifications written in domain-specific language with multimodal context, and (2) performing the challenging security analysis over cellular network standards. This framework follows a three-stage architecture inspired by the Dreyfus model of skill acquisition: Adapt corresponds to novice-level rule-based comprehension, Retrieve aligns with competent, evidence-guided decision making, and Integrate reflects expert-level synthesis across multiple sources. p. 2 This three-stage framework is inspired by the Dreyfus model of skill acquisition (Dreyfus et al. , 1986): the Adapt stage corresponds to novice-level rule-based comprehension,
Retrieve aligns with competent, evidence-guided
decision making, and Integrate reflects expert-level
synthesis across multiple sources.Where the paper says it
Multimodal Specification Adaptation
The first module, SpecFusion, enables LLMs to adapt to the domain knowledge in 3GPP specifications by decomposing them into textual components (Ctext), table components (Ctable), and figure components (Cfigure). SpecFusion transforms non-textual components into a unified textual representation by converting tables and figures into structured JSON descriptions. p. 2 Instead of applying a specific encoder for each modality, SpecFusion transforms non-textual components into a
unified textual representation. Specifically, tables
and figures in 3GPP specifications are extracted
and converted into structured JSON descriptions as
shown in Appendix B. Table components (Ctable) represent normative tables that encode structured relationships among parameters, states, timers, configuration options, and
actions. Including tabular chunks in the adaptation
process allows CellSpecSec-ARI to learn how rules
and mappings are expressed in the tables of 3GPP
specifications.Where the paper says it
Where the paper says it
Evidence-Grounding via SpecRAG
The SpecRAG module performs retrieval-augmented reasoning by researching and returning a set of relevant evidence chunks from its RAG database for each query. It uses a hybrid retrieval strategy combining sparse component ssp(q, c) using BM25 ranker and dense component sde(q, c) obtained via TF–IDF vectors projected into a lower-dimensional latent space. p. 3 The sparse component ssp(q, c) is computed using a BM25 ranker that captures exact keyword matches between the query q and candidate
chunk c. The dense component sde(q, c) is obtained
by encoding the query and each chunk into TF–IDF
vectors, projecting them into a lower-dimensional
latent space via truncated SVD, and computing cosine similarity between the resulting vectors. The final
relevance score is a weighted combination:
s(q, c) = α ssp(q, c)+(1−α) sde(q, c), α ∈ [0, 1],
where α is tuned on a small validation set to balance lexical precision and semantic recall. The
top-k highest-scoring chunks form the Retrieved
Evidence context supplied to the model.Where the paper says it
Where the paper says it
Taxonomy-Based Knowledge Integration
The SpecReasoning module transforms multimodal outputs from SpecRAG into a taxonomy-based knowledge graph (KG) that explicitly models structural and procedural dependencies among 3GPP specification entities. This KG uses entity types like Procedure, Message, Information Element, Identifier, Timer, State, Condition, and Property Entity types for taxonomy-based KG. p. 4 By
systematically analyzing each clause, including all
multimodal components, we identify the recurring
entities from the majority of clauses and summarize
a set of entity types as shown in Table 2: signaling
procedures, messages, information elements, identifiers, timers, states, conditions, and system-level
properties. They have consistent meanings across
3GPP specifications and form the basic entity vocabulary used in the taxonomy-based KGs. We first
construct an entity taxonomy that reflects the hierarchical organization of 3GPP specifications. By
systematically analyzing each clause, including all
multimodal components, we identify the recurring
entities from the majority of clauses and summarize
a set of entity types as shown in Table 2: signaling
procedures, messages, information elements, identifiers, timers, states, conditions, and system-level
properties. Two complementary families of relations are defined. Core
relations Rcore represent the structural dependencies that govern protocol execution logic.Where the paper says it
Where the paper says it
Where the paper says it
Benchmark Construction and Evaluation
CellularSpecSec-Bench covers core specifications in Release-17, processing over 1,760 paragraphs, 13,800 sentences, 424 figures, and 340 tables CellularSpecSec-Bench covers the core specifications in Release-17 (3GP, 2022a), (3GP, 2022b), (3GP, 2020), processing over 1,760 paragraphs, 13,800 sentences, 424 figures, and 340 tables The security reasoning dataset incorporates 43 real-world vulnerabilities reported in prior works (Shaik et al., 2015; Bassil et al. p. 1 The security reasoning dataset incorporates 43 real-world vulnerabilities reported in
prior works (Shaik et al. , 2015; Bassil et al. The security reasoning dataset incorporates 43 real-world vulnerabilities reported in
prior works (Shaik et al. , 2015; Bassil et al. , 2015; Bassil et al. , 2013;
Kambourakis et al. , 2015; Bassil et al. , 2013;
Kambourakis et al. , 2013;
Kambourakis et al. , 2011; Lee et al. , 2013;
Kambourakis et al. , 2011; Lee et al. , 2011; Lee et al. , 2009; Leong
et al. , 2011; Lee et al. , 2009; Leong
et al. , 2009; Leong
et al. , 2014; Kim et al. , 2009; Leong
et al. , 2014; Kim et al. , 2014; Kim et al. , 2019; Van Den Broek et al. , 2014; Kim et al. , 2019; Van Den Broek et al. , 2019; Van Den Broek et al. ,
2015; Park et al. , 2019; Van Den Broek et al. ,
2015; Park et al. ,
2015; Park et al. , 2016; Chlosta et al. ,
2015; Park et al. , 2016; Chlosta et al. , 2016; Chlosta et al. , 2019; Yu and
Chen, 2019; Michau and Devine, 2016; Hussain
et al. , 2016; Chlosta et al. , 2019; Yu and
Chen, 2019; Michau and Devine, 2016; Hussain
et al. , 2019; Yu and
Chen, 2019; Michau and Devine, 2016; Hussain
et al. , 2019b; Borgaonkar et al. , 2019; Yu and
Chen, 2019; Michau and Devine, 2016; Hussain
et al. , 2019b; Borgaonkar et al. , 2019b; Borgaonkar et al. , 2018; Cao et al. , 2019b; Borgaonkar et al. , 2018; Cao et al. , 2018; Cao et al. ,
2020; Chlosta et al. , 2018; Cao et al. ,
2020; Chlosta et al. CellularSpecSec-Bench covers the core specifications in Release-17 (3GP, 2022a), (3GP, 2022b),
(3GP, 2020), processing over 1,760 paragraphs,
13,800 sentences, 424 figures, and 340 tables, capturing both textual and multimodal specification
semantics. The security reasoning dataset incorporates 43 real-world vulnerabilities reported in
prior works (Shaik et al. CellSpecSec-ARI is a unified framework to solve standardized tasks in (1)
interpreting complex 3GPP specifications written in domain-specific language with multimodal
context, and (2) performing the challenging security analysis over cellular network standards. CellularSpecSec-Bench covers the core specifications in Release-17 (3GP, 2022a), (3GP, 2022b),
(3GP, 2020), processing over 1,760 paragraphs,
13,800 sentences, 424 figures, and 340 tables, capturing both textual and multimodal specification
semantics.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Performance Gains and Challenges
The evaluation shows that the base model performs poorly on most tasks, indicating struggles with the normative style, precision, and logic in 3GPP specifications The base model performs poorly on most tasks, indicating that the base model struggles with the normative style, precision, and logic in 3GPP specifications>. Incorporating CellSpecSec-ARI modules substantially improves performance across all tasks Incorporating CellSpecSec-ARI modules substantially improves performance, demonstrating that the domain adaptation and structured reasoning improve standards comprehension and reasoning capability>. p. 6 The base model performs poorly on most tasks, indicating that the base
model struggles with the normative style, precision,
and logic in 3GPP specifications. Incorporating
CellSpecSec-ARI modules substantially improves
performance, demonstrating that the domain adaptation and structured reasoning improve standards
comprehension and reasoning capability.Where the paper says it
Error Analysis Insights
Error analysis reveals that the dominant failures in security-focused tasks stem from incomplete evidence rather than purely incorrect security reasoning For security-focused tasks, vulnerability/inconsistency labeling, the dominant failures stem from incomplete evidence, rather than purely incorrect security reasoning>. Explanations are often directionally correct but incomplete because the cited evidence set is missing critical dependencies Explanations are directionally correct but incomplete, and the cited evidence set is missing one or more critical dependencies>. p. 7 For security-focused tasks, vulnerability/inconsistency labeling, the dominant failures
stem from incomplete evidence, rather than purely
incorrect security reasoning. Explanations are directionally correct but incomplete, and the cited
evidence set is missing one or more critical dependencies. Explanations are directionally correct but incomplete, and the cited
evidence set is missing one or more critical dependencies. These patterns emphasize that security
reasoning in standards is compositional. These patterns emphasize that security
reasoning in standards is compositional. To mitigate this, we can add a completeness verifier that
rejects conclusions or causal explanations unless
the assembled evidence jointly supports the claim.Where the paper says it
Where the paper says it
Where the paper says it
Benchmark Scope and Limitations
CellularSpecSec-Bench focuses on a limited set of 3GPP specifications in Release 17, which enables controlled evaluation but does not cover the full breadth of the corpus First, CellularSpecSec-Bench focuses on a limited set of 3GPP specifications in Release 17. While this design scope enables controlled and reproducible evaluation, it does not cover the full breadth of the 3GPP corpus or reflect the ongoing evolution of specifications across releases>. p. 8 First,
CellularSpecSec-Bench focuses on a limited set
of 3GPP specifications in Release 17. While this
design scope enables controlled and reproducible
evaluation, it does not cover the full breadth of
the 3GPP corpus or reflect the ongoing evolution of specifications across releases. Verified and corrected datasets. To broaden
coverage and support additional task formats,
CellularSpecSec-Bench selectively integrates stateof-the-art benchmarks from the telecom and cellular NLP literature. To broaden
coverage and support additional task formats,
CellularSpecSec-Bench selectively integrates stateof-the-art benchmarks from the telecom and cellular NLP literature. It includes TeleQnA (Maa- touk et al. , 2023), Telco-DPR (Saraiva et al. TSpec-LLM (Nikbakht
et al. , 2024) releases a large-scale corpus of 3GPP
specifications (Release 8–19). , 2024),
TSpec-LLM (Nikbakht et al. , 2024), and TeleLLMs (Maatouk et al. Through this selective integration and expert verification, CellularSpecSec-Bench combines
standards-grounded control-plane reasoning with
broader telecom knowledge, resulting in a benchmark that is more comprehensive and reliable than
prior datasets in this domain. 1 Task Types
We formalize the tasks involved in comprehending
3GPP specifications and identifying security design
vulnerabilities into three types. While this strict evaluation
aligns with the goal of trustworthy specification
reasoning, it may under-credit partially correct evidence or alternative but valid supporting sets. Last,
while this work focuses on establishing the first
unified framework CellSpecSec-ARI and developing the reproducible CellularSpecSec-Bench, our
experiments apply DeepSeek-V3. Last,
while this work focuses on establishing the first
unified framework CellSpecSec-ARI and developing the reproducible CellularSpecSec-Bench, our
experiments apply DeepSeek-V3. 2-Exp as the base
model. We evaluate two configurations:
(1) Base, which uses the raw DeepSeek-V3. 2-Exp
model without any CellSpecSec-ARI modules, and
(2) CellSpecSec-ARI, which augments the same underlying model with the pipeline enabled module
by module.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Conclusion
This work introduces CellSpecSec-ARI, a unified framework supporting domain adaptation to 3GPP specifications, retrieval of multimodal evidence with verifiable citations, and integration reasoning for security analysis We introduce CellSpecSec-ARI, a unified framework that supports domain adaptation to 3GPP specifications, retrieval of multimodal evidence with verifiable citations, and integration reasoning for security analysis>. The authors establish CellularSpecSec-Bench as a staged benchmark spanning clause-local comprehension, evidencegrounded answering, and cross-clause, securityrelevant reasoning We establish CellularSpecSec-Bench a staged benchmark spanning clause-local comprehension, evidencegrounded answering, and cross-clause, securityrelevant reasoning>. p. 7 We introduce CellSpecSec-ARI,
a unified framework that supports domain adaptation to 3GPP specifications, retrieval of multimodal evidence with verifiable citations, and integration reasoning for security analysis. We establish CellularSpecSec-Bench a staged benchmark
spanning clause-local comprehension, evidencegrounded answering, and cross-clause, securityrelevant reasoning, built from core specifications
and complemented with carefully verified and corrected subsets from prior datasets. To mitigate this, we can add a completeness verifier that
rejects conclusions or causal explanations unless
the assembled evidence jointly supports the claim. 7 Conclusion
This work addresses a key gap in cellular security research: the lack of a unified methodology
and reproducible benchmark for evaluating 3GPP
specification understanding and standards-driven
security reasoning.Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Adaption - SpecFusion improves LLMs by enabling them to adapt to domain knowledge in 3GPP specifications by transforming non-textual components like tables and figures into a unified textual representation as structured JSON descriptions. This allows the model to learn how rules and mappings are expressed in the tables of 3GPP specifications.
-
Retrieval - SpecRAG enhances retrieval-augmented reasoning by using a hybrid retrieval strategy that combines sparse BM25 ranking with dense TF-IDF embeddings projected into a lower-dimensional latent space via truncated SVD for semantic similarity search. This ensures that the retrieved evidence is both lexically precise and semantically relevant to the query.
-
Integration - SpecReasoning enables complex reasoning by transforming multimodal outputs from SpecRAG into a taxonomy-based knowledge graph (KG) that explicitly models structural and procedural dependencies among 3GPP specification entities, including distinct core and security relation types. This allows the system to perform cross-clause integration necessary for security analysis, as demonstrated by its ability to construct a SpecGraph instance.
-
System capabilities: The CellSpecSec-ARI framework enables systematic analysis for 3GPP specifications by supporting (1) interpreting complex 3GPP specifications written in domain-specific language with multimodal context, and (2) performing the challenging security analysis over cellular network standards.
Abstract
Cellular networks are critical infrastructure supporting billions of worldwide users and safety- and mission-critical services. Vulnerabilities in cellular networks can therefore cause service disruption, privacy breaches, and broad societal harm, motivating growing efforts to analyze 3GPP specifications that define required device and operator behavior. However, existing benchmarks for this domain share three systematic deficiencies: they rely on LLM-generated entries without expert verification, they do not require faithful interpretation of normative language, reasoning across cross-referenced clauses, and verifiable conclusions grounded in multimodal evidence such as tables and figures, and they do not evaluate security reasoning over end-to-end protocol behavior. To address these gaps, we present CellularS2-Bench, a staged benchmark. CellularS2-Bench covers core control-plane specifications, processing over 1,866 paragraphs, 22,033 sentences, 424 figures, and 248 tables. It comprises 13,851 questions and instances across three progressive stages, of which 10,269 (74.14%) are newly constructed from 3GPP specifications and 43 real-world vulnerabilities. Beyond newly constructed tasks, CellularS2-Bench integrates and corrects 3,582 (25.86%) instances from 7 prior datasets to broaden task coverage. To establish a reproducible baseline on CellularS2-Bench, we introduce CellularS2-ARI, a unified Adapt-Retrieve-Integrate framework for systematic understanding and specification-driven security analysis of 3GPP specifications. Together, they establish an accessible and reproducible foundation for quantifying progress in specification understanding and security reasoning in cellular network security.
Sources
- TeleMath: A Benchmark for Large Language Models in Telecom Mathematical Problem Solving
- SPEC5G: A Dataset for 5G Cellular Network Protocol Analysis
- Tele-LLMs: A Series of Specialized Large Language Models for Telecommunications
- TeleQnA: A Benchmark Dataset to Assess Large Language Models Telecommunications Knowledge
- TSpec-LLM: An Open-source Dataset for LLM Understanding of 3GPP Specifications
- Telco-DPR: A Hybrid Dataset for Evaluating Retrieval Models of 3GPP Technical Specifications
- Practical Attacks Against Privacy and Availability in 4G/LTE Mobile Communication Systems
- CellSecInspector: Safeguarding Cellular Networks via Automated Security Analysis on Specifications
Related papers
- From Strategy Search to Policy Learning: MCTS-Guided Adversarial Co-Evolution for LLM Safety Alignment
- Privacy Leakage from Recurrent State in LLM Inference
- Incrementally Verifiable Oracle Aided Computation
- Privilege and confidentiality in generative AI workflows
- BadMoE: Backdooring Mixture-of-Experts LLMs via Routing-Aware Triggers and Task-Underutilized Experts
- CoVault: Secure, Scalable Analytics of Personal Data