Improving Interoperability among Defence and National Security Ontologies: Analysis and Evaluation Tasks

Affiliations: City St George's, University of London; LASIGE, Faculdade de Ciências, Universidade de Lisboa; Defence Science and Technology Laboratory (UK); The Alan Turing Institute (UK)

---

The paper addresses the challenge of achieving interoperability among ontologies and knowledge graphs in the defence and national security domain. The authors note that "The use of ontologies and knowledge graphs is becoming increasingly widespread in the defence and national security domain. Numerous ontologies have been developed through initiatives led by academia, industry, and government. Achieving interoperability across diverse defence and national security ontologies remains a major challenge due to the domain's breadth and specialisation."

The work is conducted within the scope of the GUARD project, which focuses on Ensuring Interoperable and Trustworthy Knowledge Graphs for Defence and National Security AI, funded by The Turing Defence & Security Grand Challenge. The research is motivated by the growing importance of Knowledge Graphs (KGs) in Neurosymbolic (NeSy) AI systems, which aim "to (i) increase the coverage, validity and quality of the available data, (ii) impose constraints during the learning process, (iii) reason about what has been learned (i.e., validity of the predictions), and, ultimately, (iv) enable well-informed decision-making."

The defence and national security domain is particularly challenging because "it may require the interplay of several subdomains to cover, among others, threats (e.g., physical and cyber), relevant actors (e.g., government, terrorist groups, intelligence services), operations (e.g., military, emergency response), infrastructure (e.g., airport, bases), policies (e.g., defence strategy, international law), and geolocations."

The paper's contributions are summarised as: "(i) We have collected and documented 60+ public ontologies relevant to the defence and national security domain. (ii) We have analysed their intersection by performing ontology alignment over 1,653 ontology pairs. (iii) We have designed a new OAEI track including 8 matching tasks where we have compared the outcomes of several state-of-the-art alignment systems, created a consensus-based alignment, and manually verified a silver-standard reference alignment."

DISO (Defence, Intelligence and Security Ontologies) is a collection of 60+ publicly available Web Ontology Language (OWL) ontologies related to the defence and national security domain. The ontologies were identified and obtained "during an ontology search exercise undertaken primarily during Nov/Dec of 2025, drawing on the academic literature (including surveys of cyber-security ontologies), government and standards-body materials, and public ontology registries and repositories."

An ontology was included if: "(i) it is publicly available, (ii) it is provided in (or has an available serialisation in) OWL, and (iii) it is relevant to the defence and national security domain, either directly (i.e., developed by or for defence, intelligence and security organisations) or indirectly, through subdomains, such as 'situation awareness' or 'smart environments', that underpin defence and national security applications."

The collection is organised into 11 clusters (or subdomains): agentic (1 ontology), ambient intelligence (1), context awareness (3), cyber-security (15), digital twins (1), information exchange (2), information security (5), mid-level (12), risk management (1), robotics (2), situation awareness (4), smart environments (8, with subclusters: smart buildings — 4, smart cities — 2, smart homes — 2), and upper-level (8), totalling 63 ontologies.

The authors note that three factors make it difficult to provide precise counts of the number of ontologies in the DISO collection: (1) many ontologies are physically represented as networks of multiple component ontology files (e.g., SOUPA is presented as a network of 18 component ontology files), (2) networked ontologies have frequently been merged such networks into single, all-inclusive ontology files for ease of reuse, and (3) some ontologies within DISO fit naturally into more than one DISO cluster (e.g., SOUPA belongs to both 'situation awareness' and 'context awareness'; BOnSAI is assigned to both 'smart buildings' and 'ambient intelligence').

Regarding FAIRness, the authors created metadata in RDF, extending the VoID and Dublin Core Terms vocabularies and added the repository to the LoD cloud to improve its findability.

The authors conducted a bulk ontology alignment exercise using the ontology alignment system LogMap over the DISO ontologies, resulting in the (potential) generation of alignments for 1,653 ontology pairs. The analysis revealed that "LogMap identifies an intersection between at least 647 ontology pairs (66 about cyber-security), indicating that the DISO ontology network has an important number of connections (i.e., ontology mappings) across its ontologies."

The bulk analysis served as the basis to select a subset of ontology pairs for a new ontology alignment track. The authors "analysed the ontology pairs sharing a larger amount of ontology mappings (as suggested by the LogMap system) and ranked them in terms of complexity of the alignment task; that is, ontology pairs where their entities have fewer identical labels were preferred."

This process identified the top-8 candidate ontology pairs, which cluster into three distinct subdomains: cybersecurity (2 pairs), situation awareness (3 pairs), and smart environments (3 pairs).

The Ontology Alignment Evaluation Initiative (OAEI) is an annual campaign for the systematic evaluation of ontology alignment systems that plays a key role in the benchmarking, comparison and reproducibility of ontology alignment systems.

The eight matching tasks and the ten selected ontologies are:

1. UCO–STIX (cybersecurity): UCO (Unified Cyber Ontology) is a domain-level ontology focusing on concepts such as cyber investigations, network defence, threat intelligence, malware analysis, vulnerability research, and offensive operations, partly funded by the Linux Foundation. STIX (Structured Threat Information eXpression), initially developed by MITRE Corporation, is a language and serialisation format for exchanging cyber threat intelligence, managed by the OASIS Cyber Threat Intelligence Technical Committee.

2. STIX–D3FEND (cybersecurity): D3FEND is a domain-level ontology developed by MITRE Corporation, funded by the National Security Agency (NSA), the U.S. Cyber Warfare Directorate, and the U.S. Office of the Under Secretary of Defense for Research and Engineering.

3. JC3IEDM–mIO! (situation awareness): JC3IEDM (Joint Consultation, Command and Control Information Exchange Data Model), standardised under NATO as STANAG 5525, aims to support the exchange of consultation, command and control (C3) information. mIO! is a Context Ontology for Mobile Environments whose aim is to model context-related knowledge that allows mobile applications to adapt their behaviour based on user context.

4. JC3IEDM–Brick (situation awareness): Brick is a Uniform Metadata Schema for Buildings, an open-source project aiming to standardise descriptions of physical, logical and virtual aspects of buildings.

5. JC3IEDM–Facility (situation awareness): Facility is a CCO (Common Core Ontology) mid-level ontology designed to represent facilities (such as buildings, campuses, etc.) that serve some specific purpose.

6. ThinkHome–Brick (smart environments): ThinkHome is a domain-level ontology intended to support smart systems, especially those with a focus on home control, and particularly on energy efficiency.

7. Brick–SmartEnv (smart environments): SmartEnv is a network of 8 component ontologies modelling various aspects of smart homes: temporal, spatial, objects, events, agents, observations, etc.

8. CityOWL–Brick (smart environments): CityOWL is an OWL rendering of a standard schema defined by the Open Geospatial Consortium called CityGML, representing virtual 3D models (aka digital twins) of cities.

The authors executed several state-of-the-art systems on the DISO-OAEI matching tasks with two objectives: (i) to assess the complexity of the tasks, and (ii) to generate silver-standard alignments for the track. The systems selected were: AML, BertMap (with variations using different BERT-based pre-trained models), BertMapLt, LogMap, LogMapLt, LogMapLLM, Matcha, ALOD2Vec, ATMatcher, Fine-TOM, and KGMatcher.

Notably, LogMapLLM, ALOD2Vec, ATMatcher, Fine-TOM, and KGMatcher were not able to produce mappings on all matching tasks. The systems generated highly disparate sets of mappings: for example, in the JC3IEDM-mIO! task, LogMap produces 234 mappings while AML generates 34.

Consensus alignments were computed of vote 2 and 3 (i.e., mappings suggested by at least two or three systems, respectively). To reduce bias, when there are several systems of the same family (i.e., systems participating with several variants), their (voted) mappings are only counted once. For example, in the JC3IEDM-mIO! task, 77 mappings were suggested by at least 2 system families, while only 11 mappings were recommended by all 8 families.

Manual assessment was performed because consensus alignment can serve as a reference for the agreement across systems; they are, however, not complete and may also include errors, as it only requires two systems to agree on an incorrect mapping. The authors performed a manual revision of the Con-2 consensus mappings and the mappings uniquely generated by a system or system family since unique mappings may include potentially valid correspondences that were not identified by any other system family.

Key results of manual validation include: Consensus mappings are typically precise, with 94% of the mappings in JC3IEDM-Facility classified as correct. Con-2 was less accurate for the JC3IEDM-Brick task. Regarding unique mappings, 80% and 13% of the unique mappings identified by LogMap and AML in ThinkHome-Brick, respectively, were annotated as correct.

The manual revision led to the creation of a silver-standard for the DISO-OAEI tracks by merging the correct unique mappings and the correct Con-2 mappings.

Mapping comparison using Jaccard distances (Figure 2) confirmed a significant disparity among the evaluated systems and that the selected matching task involves the discovery of non-trivial mappings. The authors caution that proximity to the consensus mapping set is not necessarily an indicator of better quality; rather, it reflects a higher agreement with the consensus. They also note that "the silver-standard is derived from the outputs of the participating systems, its recall is inherently bounded by what these systems can collectively discover, and it will favour systems that resemble those used in its construction. Thus, it should be regarded as a partial reference rather than a complete gold standard. For this reason, they prefer not to report the actual precision and recall values of the participating systems with respect to the silver standard, but rather the proximity among the relevant mapping sets."

Ontology compatibility analysis revealed that reasoning with the input ontologies and the mappings may lead to logical errors (e.g., unsatisfiabilities). Even systems implementing (approximate) reasoning techniques like AML and LogMap lead to a large number of unsatisfiabilities (e.g., UCO-STIX and STIX-D3FEND). This highlights a modelling disagreement among the DISO-OAEI ontologies, bringing an interesting challenge to the OAEI from the logical and reasoning point of view. The silver-standard alignment also leads to unsatisfiable classes, emphasising the need for reasoning techniques within the ontology alignment pipeline. For the OAEI 2026 track, the silver standard has been treated to fix logical errors and annotate incoherent mappings.

Resources are made available through a small ecosystem of three repositories:

1. diso — the collection of defence and national security ontologies, mirrored to Zenodo upon every major release, with both a canonical distribution and a compact distribution (the latter merging networked ontologies into single files).

2. diso-mappings — a Python pipeline that consumes the DISO compact distribution and produces pairwise alignments between selected ontology pairs, using a configurable set of ontology matching systems (AML, LogMap, LogMapLt, BERTMap, and BERTMapLt by default; with an extendable Matcher base class). The pipeline aggregates the per-system alignments into a consensus alignment via a family-based voting mechanism.

3. diso-oaei — the home for the new DISO-OAEI track introduced in the OAEI 2026 edition. It contains the datasets, evaluation metrics, participation instructions, and will also host the OAEI benchmarking results.

The authors curated and described a collection of publicly available OWL ontologies covering the defence and national security domain, and proposed a new OAEI track for evaluating ontology matching systems on this collection. The analysis identified eight candidate ontology pairs, spanning three distinct subdomains, supporting non-trivial ontology matching tasks across classes, properties, and instances.

The defence and national security domain presents distinctive challenges for ontology matching: source ontologies span heterogeneous granularities (e.g., upper-level, mid-level, temporal), numerous subdomains, and gold standard ground-truth coverage is limited.

Future plans include: automated benchmarking procedures, an active leaderboard, and a candidate mapping ranking task as in the bio-ml track to attract the participation of machine learning systems. The authors note that DISO is not the first initiative to gather domain-specific ontologies, citing BioPortal as the reference repository for biomedical ontologies, and the National Center for Ontological Research (NCOR)'s National Security Ontology Foundry, with which we intend to explore potential collaborations. They also plan to explore adopting the OntoPortal technology to host and manage its ontologies.

The authors are "actively collaborating with experts from the UK Government intelligence community, the NDTP (National Digital Twin Programme), and the Dstl (Defence Science and Technology Laboratory), who will support the development and maintenance of the DISO ecosystem."

• DISO Repository: https://github.com/city-artificial-intelligence/diso

• OAEI Track: https://city-artificial-intelligence.github.io/diso-oaei/

• Codes: https://github.com/city-artificial-intelligence/DISO-mappings

• Permanent repository: https://doi.org/10.5281/zenodo.20059506

• License: MIT License

• Aligns defence and national security ontologies across heterogeneous subdomains. The improved system can map entities between cyber-security, situation awareness, and smart-environment ontologies (e.g., UCO–STIX, JC3IEDM–Brick, Brick–SmartEnv), handling differences in granularity from upper-level to domain-level and mid-level models.

• Generates more robust consensus alignments using family-aware voting. It aggregates outputs from diverse matcher families (LogMap, AML, BERTMap variants) while counting each family only once, so one dominant approach does not bias the result. This produces higher-confidence mappings for tasks where systems disagree massively, such as JC3IEDM-mIO!.

• Detects and repairs logically incoherent alignments. After mapping, it runs reasoning over ontology pairs to identify unsatisfiable classes, flags modelling disagreements, and fixes the silver-standard reference to remove incoherent mappings. This ensures the resulting knowledge graph remains logically consistent, which is critical for AI decision-making.

• Exploits manually validated silver-standard mappings to improve matching quality. The system can use the manually verified silver standard from the DISO-OAEI track as training data or as a partial reference, while explicitly accounting for its bounded recall. It learns to recover correct unique mappings—like the 80% of LogMap-only mappings judged correct in ThinkHome–Brick—rather than relying solely on consensus.

• Selects and weights matchers based on task complexity. By measuring Jaccard distance to consensus sets and analysing per-task behaviour (e.g., LogMap producing 234 mappings vs. AML producing 34 on JC3IEDM-mIO!), the system adaptively chooses which matchers to trust for a given ontology pair, improving precision and recall on non-trivial cross-domain tasks.

• Builds interoperable defence knowledge graphs from networked ontologies. The system consumes the DISO compact distribution, merges component ontology files (e.g., SOUPA's 18 files), performs pairwise alignment across 1,653 ontology pairs, and outputs a unified knowledge graph with provenance, confidence scores, and consensus metadata.

• Enables neurosymbolic AI with ontology-grounded constraints. Using the aligned ontologies, the system can impose semantic constraints during learning, validate predictions against the ontology network, and reason about the validity of outputs—supporting threat analysis, emergency response, and military decision-making with trustworthy knowledge graphs.

• Recommends relevant ontologies for new defence AI tasks. Leveraging the DISO collection's RDF metadata (VoID and Dublin Core), the system can query the LoD cloud and retrieve the most relevant ontologies for a new subdomain, reducing manual search effort and improving reproducibility.

• Benchmarks and ranks matching systems on defence tasks. The system provides an automated leaderboard for the DISO-OAEI track, with evaluation metrics that avoid over-relying on an incomplete silver standard—preferring proximity measures and manual verification workflows to attract both classical and machine-learning matchers.