FAIR-Compliant Architecture for Heterogeneous Astronomical Data: KazVO Framework

arXiv:2609.39546 · astro-ph.IM · Submitted 2026-09-30 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Astrophysics. The day in astrophysics: observations, models and what the data forces.

Vera: We're looking at the paper: "FAIR-Compliant Architecture for Heterogeneous Astronomical Data".

Jocelyn: Here's the gist of the paper. The paper presents the architecture of KazVO, an International Virtual Observatory Alliance compliant node that unifies heterogeneous observational datasets from the Fesenkov Astrophysical Institute (FAI) to provide open, standardized access to regional astronomical assets globally.

Vera: This framework is crucial because it bridges two fundamentally distinct data types—digitized historical glass-plate heritage and live operational photometric and spectroscopic digital streams—opening FAI’s combined datasets to the global scientific community Architectural Foundation The architectural foundation of the system is built upon a partitioned Data Lake, a German Astrophysical Virtual Observatory (GAVO) Data Center Helper Suite (DaCHS) publishing backend, and a PostgreSQL database that maps heterogeneous metadata to the unified IVOA ObsCore standard The system employs a four-class data embargo mechanism based on metadata-only modeling to follow Open Science principles Machine-to-machine services are deployed via IVOA protocols listed in the global Registry of Registers, enabling analysis of Kazakhstani observational assets within external clients such as TOPCAT, Aladin, and PyVO Data Collection and Digitization The observational basis consists of two main categories: physical analog media, specifically glass photographic plates and films from 1950 to 1997, and incoming digital data streams from modern detectors The historical archive of FAI photographic plates and films spans the period from 1950 to 1997 and serves as the primary physical material for this study These materials were observed using three main instruments: the 50-cm Maksutov meniscus telescope (ASI-2), the 39.7-cm Schmidt camera, and the 70-cm AZT-8 reflector The spectroscopic dataset acquired with the AZT-8 telescope predominantly covers the optical wavelength range (3700–8600 ˚A), forming a unique historical baseline for analyzing spectral variability and kinematics Data Standardization and Processing The digitization of the FAI photographic archive involved converting emulsion optical density profiles into relative intensities, alongside converting handwritten observational logbooks into machine-readable metadata TIFF scans were converted into 16-bit FITS and subsequently FITS formats using Maxim DL Pro and Image Reduction and Analysis Facility (IRAF) Metadata normalization was implemented in strict accordance with IVOA recommendations The pipeline involves sequential transformation of astronomical resources from analog states into full interoperability within the VO ecosystem Interoperability and Access Control The integration into the IVOA ecosystem strictly complies with the IVOA architecture, covering the entire data lifecycle from hardware resources to end-user applications Metadata is mapped to the mandatory relational schema of the ObsCore (v1.1) data model during construction of GAVO DaCHS resource descriptors Functional access is provided via core IVOA components such as TAP, SIAP, and SSAP protocols The policy strictly regulates the legal status of institutional assets through a flexible 4-tier access classification scheme (Classes A, B, B+, C) Operational Data Holdings and Policy Enforcement The KazVO Data Lake is logically partitioned by observatory site, observational modality, and historical epoch The system utilizes a hybrid mode for access: human-centric interactive querying via the web portal and programmatic M2M workflows driven by standardized IVOA protocols The "Scientific Showcase" mechanism uses a zero-filled payload masking engine to protect binary arrays during active embargo periods while preserving searchable WCS metadata This dynamic routing ensures that access is granted only when the current date matches or exceeds the release timestamp, transitioning data from Embargoed to Public automatically The developed architecture integrates historical archives and modern observational data streams into a unified standardized infrastructure, ensuring the incorporation of regional data into the global virtual observatory network in strict compliance with IVOA standards The framework successfully transforms heterogeneous observational data streams and scientific collections into a centralized, fully operational IVOA node This architecture demonstrates that deploying a fully compliant IVOA node relies on the systematic harmonization of heterogeneous archives using standardized protocols and containerized microservices The resulting architecture not only preserves Kazakhstani extensive astronomical heritage and modern observational data but also exposes them as robust, interoperable assets for the global astrophysical community The total volume of published files across all resources is approximately 174.

Title and authors: Vera: So, we've seen how they built this architecture, and now we're looking at what they call the improvements they suggest for this KazVO Framework. They’re focusing on making sure it’s not just a static system but one that handles data dynamically.

Jocelyn: They talk about automating the ingestion pipeline, which means instead of manually moving everything, they want to buffer raw frames locally and have the system automatically check their integrity before sending them out via satellite or ground links.

Subrahmanyan: That sounds like a necessary step for handling that volume of data; ensuring the stream itself is reliable before it hits the main processing pipeline is vital for maintaining scientific value nine.

Vera: And they’re pushing for FAIR compliance through metadata injection, which means they're going to use AI to automatically tag FITS headers with governance keywords like OBS CLSS and RELEASE during the whole processing sequence.

Jocelyn: That addresses the "Veracity" aspect of Big Data, making sure the data is correctly labeled right when it's being made available nine.

Subrahmanyan: From a theoretical standpoint, that automated enrichment is good because it enforces structure at scale; if you can automate the governance keywords, you reduce human error in cataloging the data’s context.

Vera: Then there's the dynamic embargo management; they propose a "Scientific Showcase" mechanism where they replace the actual binary data array with zeros while keeping all the essential astrometric keys so people can still search it.

Jocelyn: That sounds like a smart way to manage access control while still proving that you have the data there, even if it's not fully visible right this second twelve <ref:2609.39546#pg2>.

Subrahmanyan: It’s a clever way to handle the legal status of assets—Classes A through C—by separating the searchable metadata from the heavy binary files until they are ready for public release >

The paper's summary: Vera: So, looking at their main summary, KazVO is designed to bridge that gap between old glass plates and new digital streams, creating a single point of access for FAI’s heritage.

Jocelyn: They emphasize that the core challenge they solved was the structural heterogeneity of the observational materials themselves; it was combining physical analog media with modern digital data seven eleven.

Subrahmanyan: That heterogeneity is what forces you to adopt this data evolution model, where resources sequentially transform from isolated local states into something fully interoperable within the larger VO ecosystem three <ref:2609.39546#pg3>.

Vera: They detail how they digitized the old photographic plates by converting emulsion optical density profiles into relative intensities and turning handwritten logbooks into machine-readable metadata scans.

Jocelyn: The process involves converting those images to sixteen-bit FITS files using tools like Maxim DL Pro and IRAF, then normalizing the metadata strictly following IVOA recommendations > <ref:2609.39546#pg2>

Subrahmanyan: That entire pipeline is essentially a transformation from an analog state to full interoperability within the VO ecosystem, which is what makes the whole structure work three <ref:2609.39546#pg3,full interoperability within the VO ecosystem>.

Vera: They stress that this entire methodological framework relies on strict adherence to FAIR and IVOA standards to ensure the data is Findable, Accessible, Interoperable and Reusable

twelve–fourteen: <ref:2609.39546#pg2>.

Jocelyn: It’s a heavy lift, ensuring that when you use TAP or SIAP sixteen seventeen, you get consistent results regardless of whether the input came from an old plate or a modern detector > <ref:2609.39546#pg2>

The paper's improvements: Vera: Now we're looking at the specific improvements they propose to this KazVO architecture, and they are really focused on making it operational for real users.

Jocelyn: They highlight the cross-epoch multi-domain analysis as a major goal; this means they want to allow direct comparison between archival spectra and modern CCD observations for things like emission line profile evolution five.

Subrahmanyan: That direct comparative analysis is powerful because it allows us to test structural models against both historical context and current measurements, which is what we need for understanding how structure forms over time >

Vera: They also mention achieving high precision astrometric calibration by using automated WCS fitting against Gaia DR3 reference catalogs to get sub-arcsecond accuracy six.

Jocelyn: Getting that kind of precise coordinate information is essential because if the astrometry isn't tight, all the subsequent metadata translation into the ivoa.obscore specification won't be reliable fifteen <ref:2609.39546#pg2>.

Subrahmanyan: That high precision validates the entire pipeline; it proves that you can generate accurate headers and translate them correctly into the required format for global use >

Vera: And they also point to scalable data lake partitioning, using AI to manage that structure by isolating assets based on observatory, instrument, and epoch across over five hundred seventy terabytes of storage eight.

Jocelyn: That isolation is smart for scalability; it prevents the archive from becoming a mess where everything gets mixed up and fragmented across all those years >

Conclusion: Vera: So, to wrap up, this paper on the FAIR-Compliant Architecture for Heterogeneous Astronomical Data: KazVO Framework shows how you can take deeply diverse data—from glass plates to live streams—and systematically harmonize them into a unified IVOA node.

Jocelyn: It really demonstrates that deploying a fully compliant IVOA node isn't just about having the right hardware; it’s about the systematic harmonization of those heterogeneous archives using standardized protocols and containerized microservices >

Subrahmanyan: The resulting architecture successfully transforms these varied observational streams into robust, interoperable assets for the global community, preserving Kazakhstani heritage while exposing it as usable data one <ref:2609.39546#pg1>.

Vera: It shows that you can achieve this by systematically harmonizing archives using standardized protocols and microservices three <ref:2609.39546#pg3>.

Jocelyn: That means we have a centralized, fully operational IVOA node ready to serve the global scientific community >

Subrahmanyan: If you look at the volume, there are approximately one hundred seventy-four published files across all those resources that are now accessible in this way >

Fesenkov Astrophysical Institute

astro-ph.IM

Submitted: 2026-09-30

Updated: 2026-11-12

Comments: 28 pages, 13 figures, 7 tables

DOI: 10.3390/galaxies14060098

Code: https://github.com/fai-kz/ffdb-api-server

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 83/100

The gist: The paper presents the architecture of KazVO, an International Virtual Observatory Alliance compliant node that unifies heterogeneous observational datasets from the Fesenkov Astrophysical Institute…

Key concepts

Data Lake Partitioning
The KazVO Data Lake is logically divided based on observatory site, observing type, and historical time period. This structure ensures high scalability and prevents data fragmentation across the 570 TB archive by isolating different types of observational assets effectively.
IVOA ObsCore Mapping
Metadata from various sources is mapped to the mandatory relational schema of the ObsCore v1.1 data model during system construction. This standardization is crucial for ensuring that all data can be understood and integrated by external VO clients using universal protocols.
Scientific Showcase Mechanism
This mechanism protects binary data arrays during embargo periods by replacing them with zero-filled payloads while keeping the essential astrometric metadata searchable. Data automatically transitions to public access when the current date meets or exceeds the set release timestamp.
M2M Services via IVOA Protocols
Machine-to-machine services are deployed using established IVOA protocols listed in global registries. This allows external clients like TOPCAT and PyVO to programmatically query and analyze Kazakhstani observational assets directly.

Terminology

Summary

The paper presents the architecture of KazVO, an International Virtual Observatory Alliance compliant node that unifies heterogeneous observational datasets from the Fesenkov Astrophysical Institute (FAI) to provide open, standardized access to regional astronomical assets globally. p. 0

Where the paper says it

kz, aimuratov@fai. kz Abstract This paper presents the architecture of the Kazakhstani National Virtual Observatory (KazVO)—an International Virtual Observatory Alliance (IVOA) compliant node that unifies the heterogeneous observational datasets of the Fesenkov Astrophysical Institute (FAI).

This framework is crucial because it bridges two fundamentally distinct data types—digitized historical glass-plate heritage and live operational photometric and spectroscopic digital streams—opening FAI’s combined datasets to the global scientific community

Architectural Foundation

The architectural foundation of the system is built upon a partitioned Data Lake, a German Astrophysical Virtual Observatory (GAVO) Data Center Helper Suite (DaCHS) publishing backend, and a PostgreSQL database that maps heterogeneous metadata to the unified IVOA ObsCore standard The system employs a four-class data embargo mechanism based on metadata-only modeling to follow Open Science principles Machine-to-machine services are deployed via IVOA protocols listed in the global Registry of Registers, enabling analysis of Kazakhstani observational assets within external clients such as TOPCAT, Aladin, and PyVO

Data Collection and Digitization

The observational basis consists of two main categories: physical analog media, specifically glass photographic plates and films from 1950 to 1997, and incoming digital data streams from modern detectors The historical archive of FAI photographic plates and films spans the period from 1950 to 1997 and serves as the primary physical material for this study These materials were observed using three main instruments: the 50-cm Maksutov meniscus telescope (ASI-2), the 39.7-cm Schmidt camera, and the 70-cm AZT-8 reflector The spectroscopic dataset acquired with the AZT-8 telescope predominantly covers the optical wavelength range (3700–8600 ˚A), forming a unique historical baseline for analyzing spectral variability and kinematics

Data Standardization and Processing

The digitization of the FAI photographic archive involved converting emulsion optical density profiles into relative intensities, alongside converting handwritten observational logbooks into machine-readable metadata TIFF scans were converted into 16-bit FITS and subsequently FITS formats using Maxim DL Pro and Image Reduction and Analysis Facility (IRAF) Metadata normalization was implemented in strict accordance with IVOA recommendations The pipeline involves sequential transformation of astronomical resources from analog states into full interoperability within the VO ecosystem

Interoperability and Access Control

The integration into the IVOA ecosystem strictly complies with the IVOA architecture, covering the entire data lifecycle from hardware resources to end-user applications Metadata is mapped to the mandatory relational schema of the ObsCore (v1.1) data model during construction of GAVO DaCHS resource descriptors Functional access is provided via core IVOA components such as TAP, SIAP, and SSAP protocols The policy strictly regulates the legal status of institutional assets through a flexible 4-tier access classification scheme (Classes A, B, B+, C)

Operational Data Holdings and Policy Enforcement

The KazVO Data Lake is logically partitioned by observatory site, observational modality, and historical epoch The system utilizes a hybrid mode for access: human-centric interactive querying via the web portal and programmatic M2M workflows driven by standardized IVOA protocols The Scientific Showcase mechanism uses a zero-filled payload masking engine to protect binary arrays during active embargo periods while preserving searchable WCS metadata This dynamic routing ensures that access is granted only when the current date matches or exceeds the release timestamp, transitioning data from Embargoed to Public automatically

Conclusion and Impact

The developed architecture integrates historical archives and modern observational data streams into a unified standardized infrastructure, ensuring the incorporation of regional data into the global virtual observatory network in strict compliance with IVOA standards The framework successfully transforms heterogeneous observational data streams and scientific collections into a centralized, fully operational IVOA node This architecture demonstrates that deploying a fully compliant IVOA node relies on the systematic harmonization of heterogeneous archives using standardized protocols and containerized microservices The resulting architecture not only preserves Kazakhstani extensive astronomical heritage and modern observational data but also exposes them as robust, interoperable assets for the global astrophysical community The total volume of published files across all resources is approximately 174.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Automated Data Ingestion Pipeline for Heterogeneous Assets: The system can automate data ingestion by buffering raw frames on local observatory servers, undergoing automated integrity verification, and preparing them for transmission via Starlink satellite terminals and terrestrial links.

  2. FAIR Compliance Enforcement via Metadata Injection: The AI system can enrich FITS headers with governance keywords like OBS CLSS and RELEASE during pipeline processing to ensure mandatory FITS header structure is met in accordance with IVOA recommendations.

  3. Dynamic Data Embargo Management: The system can implement a Scientific Showcase mechanism where it generates a protected version of the FITS file by replacing the binary data array with zero-valued elements while preserving all essential WCS astrometric keys for metadata discoverability.

  4. Automated Data Release Control: The system can automatically transition datasets from Embargoed to Public status by dynamically evaluating the current date against the release timestamp, terminating pixel zeroing and streaming original full-scale FITS files when the condition CURRENT DATE >= release date is met.

  5. Cross-Epoch Multi-Domain Analysis: The improved system can perform cross-matching between archival holdings and modern observations to enable direct multi-epoch analysis, such as combining an archival spectrum with a modern CCD spectrum for direct comparative analysis of emission line profile evolution.

  6. High Precision Astrometric Calibration: The system can utilize automated WCS fitting against astrometric reference catalogs like Gaia DR3 to achieve sub-arcsecond accuracy, validating the pipeline's readiness for accurate header generation and subsequent metadata translation into the ivoa.obscore specification.

  7. Programmatic Access via IVOA Protocols: The system can expose operational services through core IVOA protocols such as TAP, SIAP, and SSAP to allow external clients like TOPCAT and PyVO to query KazVO node holdings programmatically.

  8. Scalable Data Lake Partitioning: The AI architecture can manage a partitioned Data Lake structure that isolates assets by observatory, instrument, and observing mode to ensure high scalability and prevent archive fragmentation across over 570 TB of storage.

  9. Administrative Inventory Tracking: The system can maintain an administrative inventory using the FFDB to parse headers across all Data Lake levels, preserving complete administrative control over institutional datasets for fast internal queries.

Abstract

This paper presents the architecture of the Kazakhstani National Virtual Observatory (KazVO) - an International Virtual Observatory Alliance (IVOA) compliant node that unifies the heterogeneous observational datasets of the Fesenkov Astrophysical Institute (FAI). The architectural foundation of the system is built upon a partitioned Data Lake, a German Astrophysical Virtual Observatory (GAVO) Data Center Helper Suite (DaCHS) publishing backend, and a PostgreSQL database that maps heterogeneous metadata to the unified IVOA ObsCore standard. We follow Open Science by introducing metadata-only model featuring based on a four-class data embargo mechanism. Machine-to-machine services deployed via IVOA protocols are listed in the global Registry of Registers, enabling analysis of Kazakhstani observational assets within external clients such as Tool for OPerations on Catalogues And Tables (TOPCAT), Aladin, and PyVO. As a result, KazVO framework provides a scalable platform driven by a developed end-to-end pipeline that bridges two fundamentally distinct data types - digitized historical glass-plate heritage (1950-1997) and live operational photometric and spectroscopic digital streams from telescopes at the Assy-Turgen and Tien-Shan Observatories - opening FAI's combined data datasets to the global scientific and time-domain astrophysics community.

Sources

Related papers