Sky Monitor AI: classifying night-sky conditions from an observatory all-sky camera with a CNN ensemble, label-free sky types and exact evidence maps

arXiv:2611.14987 · astro-ph.IM · Submitted 2026-11-11 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Astrophysics. The day in astrophysics: observations, models and what the data forces.

Vera: I'm Vera.

Jocelyn: And I'm Jocelyn, here with Subrahmanyan.

Vera: We're looking at the paper: "Sky Monitor AI".

Jocelyn: Here's the gist of the paper. Small and remotely operated observatories must decide every night whether the sky is clear enough to observe Abstract>.

Vera: The ensemble of three CNNs reaches 86.7 per cent accuracy with the smallest night-to-night scatter Conclusion>.

Jocelyn: Classification Pipeline and Model Performance The paper presents a complete pipeline that classifies frames of the auxiliary all-sky camera of the HigashiHiroshima Observatory as clear or cloudy Abstract>.

Vera: Hand-crafted sky statistics reach 77.1 per cent accuracy on six test nights, while a small convolutional neural network (CNN) trained from scratch reaches 84.3 per cent, and an ensemble of the small CNN and two ImageNet-pretrained ResNet-18 variants reaches 86.7 per cent (AUC 0.924) with the smallest night-to-night scatter Abstract>.

Paper summary: Vera: So we've been looking at this study called "Sky Monitor AI: classifying night-sky conditions from an observatory all-sky camera with a CNN ensemble, label-free sky types and exact evidence maps" which uses an ensemble of deep learning models to classify what’s in the night sky from a big all-sky camera.

Jocelyn: It’s a pretty comprehensive piece because they aren't just giving you a simple clear or cloudy answer; they're also producing these exact evidence maps and estimates for cloud cover.

Subrahmanyan: The authors used this approach to discover new classes of sky types without needing any pre-labeled data for those specific things, like finding frames with water droplets on the camera dome.

Vera: That’s significant because it shows how you can use unsupervised methods, like Gaussian mixture clustering on those AI embeddings, to find stuff that wasn't even in the initial labels.

Jocelyn: And they found a separate class for things like condensation or precipitation on the optics, which is operationally important for keeping equipment safe.

Subrahmanyan: From a cosmological view, it means we get a better way to filter out instrumental noise or localized atmospheric issues before we start looking at the actual astronomical signals.

Vera: The whole system runs as an open-source application that you can use on single frames or even live cameras, which is really democratizing how astronomers can process this kind of data.

Jocelyn: It also tells us that the ensemble method handles things like moonlight much better than any single model alone, which is a big win for those nights when the sky isn't perfectly dark.

Subrahmanyan: So what this implies for future work is using these models to generate preliminary classifications, giving you a starting point before you dive into more complex physical modeling of the light itself.

Conclusion: Vera: So we've seen this whole process laid out in "Sky Monitor AI," which is essentially using an ensemble of deep learning models to look at images from an all-sky camera and decide what's happening night skyward.

Jocelyn: The authors show that their combined approach hits eighty-six point seven per cent accuracy with a zero point nine two four AUC, while keeping the variation between nights really low.

Subrahmanyan: What’s interesting here is how they found those new sky types without any pre-labeled data, which means they could even identify frames with water droplets on the camera dome.

Vera: That’s significant because it proves you can use techniques like Gaussian mixture clustering on the AI embeddings to find things that weren't in the original labels.

Jocelyn: And they distinguished between different types of wet conditions, which is useful for keeping the equipment safe, not just knowing if it's cloudy or clear.

Subrahmanyan: From a cosmological standpoint, this means we have a better way to filter out localized atmospheric stuff before we try to model the actual light coming from deep space.

Vera: It’s an open-source application now, so anyone can use these exact evidence maps with old frames or even live cameras.

Jocelyn: That makes it a real resource for people trying to interpret data from remote observation sites.

Subrahmanyan: So, we've seen the results and the tool. But what this really means is that automated classification can be a strong first step in observational work, helping you sort through ambiguous data before you spend time on more detailed physical modeling of light itself.

Muhammad Waqas

Department of Astronomy, Institut Teknologi Bandung

astro-ph.IM

Submitted: 2026-11-11

Updated: 2026-11-11

Comments: 9 pages, 8 figures, 5 tables. Code and application: https://github.com/waqas6767/Sky-Monitor-AI-for-Remote-Observatory-Operations

Code: https://github.com/waqas6767/Sky-

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 83/100

The gist: Small and remotely operated observatories must decide every night whether the sky is clear enough to observe Abstract>.

Key concepts

CNN Ensemble
This involves combining the predictions of three different Convolutional Neural Networks (CNNs) to make a final decision. By averaging their results, the system becomes more robust against errors from any single model, leading to higher accuracy and less variation in performance across different observation nights.
Label-Free Sky Type Discovery
The researchers used clustering on embeddings from pre-trained CNNs to automatically find different sky types (like clear or cloudy) without needing pre-labeled data for those specific classes. This method successfully identified a new category, 'water droplets on the camera dome,' which was not present in the original training labels.
Evidence Maps
These maps are generated by analyzing how the final network makes its decision. They show exactly which parts of the sky image influenced the classification (e.g., specific regions indicating cloud cover). This provides a visual explanation for why a model chose a particular outcome, useful for operational understanding.
Moonlight Sensitivity
Moonlight is identified as the primary challenge in classifying sky conditions. The ensemble model demonstrated superior resilience to variations caused by moonlight compared to individual models or simpler statistical methods, making it more reliable when observing under varying lunar illumination.

Terminology

Summary

Small and remotely operated observatories must decide every night whether the sky is clear enough to observe Abstract>. p. 0 The ensemble of three CNNs reaches 86. p. 77 per cent accuracy with the smallest night-to-night scatter Conclusion>. p. 7

Classification Pipeline and Model Performance

The paper presents a complete pipeline that classifies frames of the auxiliary all-sky camera of the HigashiHiroshima Observatory as clear or cloudy Abstract>. Hand-crafted sky statistics reach 77. p. 01 per cent accuracy on six test nights, while a small convolutional neural network (CNN) trained from scratch reaches 84. p. 03 per cent, and an ensemble of the small CNN and two ImageNet-pretrained ResNet-18 variants reaches 86. p. 07 per cent (AUC 0. p. 0924) with the smallest night-to-night scatter Abstract>. p. 0 The final ensemble reaches 86. p. 37 per cent accuracy, 2. p. 04 points above the best single model and 9. p. 36 points above the hand-crafted model on the same nights, with the smallest night-to-night scatter Conclusion>. p. 3

Data Preparation and Masking

The camera records 600×600 pixel colour frames through a fisheye lens Page 2, 2.1 Camera and frames>. p. 0 A circular sky mask removes a printed date stamp that would otherwise act as a shortcut Abstract>. p. 0 This mask removes the captions and margins and keeps the whole sky Page 2, 2. p. 13 Sky mask>. p. 0 The masked fraction of the image is f = 1−π 0. p. 14552 ≈ 0. p. 135 Page 2, Table 3>. p. 1 All frames of a night are assigned to the same subset (Table 1) Page 2, 2. p. 14 Night-based split>. p. 1

Model Selection and Generalization Analysis

The decision to use an ensemble was made on the validation nights, as the three-model ensemble had the highest validation accuracy and was selected before the test results were examined in detail Page 3, 3.3 Ensemble and model selection>. p. 2 The per-night breakdown shows that the models fail on different nights Page 5, Table 3>. p. 3 Moonlight is the main difficulty, and the ensemble is the least sensitive to it Page 9, Discussion>. p. 7 The worst night for the ensemble is still at 0. p. 373 Page 5, Table 3>. p. 1

Label-Free Sky Type Discovery

Without labels, Gaussian-mixture clustering of pretrained CNN embeddings recovers the clear/cloudy split with 87 per cent purity and isolates frames with water droplets on the camera dome, a class absent from the labels Abstract>. The two descriptions behave very differently (Table 5) Page 5, Table 5>. p. 5 CNN embedding GMM agrees with fine/cloudy for 87 per cent of the frames (ARI 0. p. 5548), comparable to the supervised test accuracy although on a different sample without held-out nights Page 5, Table 3>. p. 5

Evidence Maps and Operational Tool

For networks ending in global average pooling the decision decomposes exactly into a map of local logits, which yields gradient-free evidence maps and a cloud-cover estimate Abstract>. Sky Monitor AI is an open-source application that applies the models to single frames, whole nights and live cameras Abstract>. p. 0 The application reports Pens, the evidence map, the cloud cover, the nearest of the five sky types, and an out-of-distribution warning when the embedding lies more than 1. p. 62 times the 99th-percentile within-cluster distance from its nearest centre Page 7, 7. p. 6 Exact evidence maps and the Sky Monitor AI application>. p. 5 p. 5

Interpretation of Network Decisions

Signed Grad-CAM indicates that the networks rely on sky regions rather than on fixed structures Abstract>. For equipment safety this conservative behaviour may be preferable Page 4, 3. p. 34 Moon ephemeris and temporal smoothing>. p. 3 The wet-dome cluster suggests that a third class, precipitation or condensation on the optics, is operationally more important than a finer subdivision of cloud cover Page 8, Discussion>. p. 6 The local logit map is the decision the network would make if the whole image looked like position (x,y) Page 7, 7. p. 5 Exact evidence maps and the Sky Monitor AI application>.

Limitations and Future Work

The data span 32 nights from a single season and camera; the models have not been tested on other cameras, other seasons or in daytime Page 8, Discussion>. Natural extensions are more nights with varied Moon phases, frame-level relabelling around weather transitions, sequence models that use cloud motion (cf. p. 7 Buntin et al. p. 0, 2025), and comparison with the model used operationally at the observatory (Uemura et al. p. 7, 2025) Page 8, Discussion>. p. 6 The tool has not been operated alongside human observers, and should not control equipment unattended Page 8, Discussion>. p. 7 p. 7 The analysis of Moon altitude is strong evidence rather than proof Page 4, 4. p. 32 Moonlight>. p. 0

Conclusion Summary

We have presented an end-to-end study of night-sky classification for the all-sky camera of the Higashi-Hiroshima Observatory Page 9, Conclusion>. The ensemble reaches 86. p. 07 per cent accuracy and AUC 0. p. 1924 with the smallest night-to-night scatter Abstract>. p. 0 Moonlight is the main difficulty, and the ensemble is the least sensitive to it Conclusion>. p. 7 Pretrained CNN embeddings recover the clear/cloudy split without labels and reveal a wet-dome class Abstract>. p. 7 An exact local-logit decomposition provides evidence maps and cloud-cover estimates in an open-source application that can be used with archived frames and live cameras Conclusion>. p. 7 The code, the application and the trained models are available at https://github. p. 7com/waqas6767/Sky- Monitor AI-for-Remote-Observatory-Operations (models in release v1. p. 70) Page 9, Acknowledgements>. p. 7 The all-sky camera images are not part of this release; requests should be directed to the Higashi-Hiroshima Observatory, Hiroshima University Page 9, Data availability>. p. 7 The tool has not been operated alongside human observers, and should not control equipment unattended Page 8, Discussion>. The code and documentation are public Page 9, Acknowledgements>. p. 6

References

Astropy Collaboration. The astropy project: Sustaining and growing a community-oriented open-source project and the latest major release (v5. p. 70) of the core package Page 9, References>. p. 7 Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li FeiFei. p. 8 ImageNet: A large-scale hierarchical image database Page 9, References>. p. 8 Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. p. 8 Deep residual learning for image recognition Page 9, References>. p. 8 Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, et al. p. 8 PyTorch: An imperative style, high-performance deep learning library Page 9, References>. p. 8 F. Pedregosa, G. p. 3 Varoquaux, A. p. 8 Gramfort, V. p. 8 Michel, B. Thirion, O. p. 8 Grisel, et al. p. 8 Scikit-learn: Machine learning in Python Page 9, References>. p. 8 Peter J. Rousseeuw. Silhouettes: a graphical aid to the interpretation and validation of cluster analysis Page 9, References>. p. 8 Ramprasaath R. p. 8 Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. p. 8 GradCAM: Visual explanations from deep networks via gradientbased localization Page 9, References>. p. 8 Lior Shamir and Robert J. p. 8 Nemiroff. All-sky relative opacity mapping using night time panoramic images Page 9, References>. p. 8 Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. p. 8 Rethinking the inception architecture for computer vision Page 9, References>. p. 8 Makoto Uemura, Y. p. 7 Koga, R. Sazaki, T. Yukino, T. Nakaoka, et al. p. 8 Smart kanata: A framework for autonomous decision-making in rapid follow-up observations of cataclysmic variables Page 9, References>. p. 8 Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. p. 8 Learning deep features for discriminative localization Page 9, References>. p. 8

--- Page 1 ---

Sky Monitor AI: classifying night-sky conditions from an observatory all-sky camera with a CNN ensemble, label-free sky types and exact evidence maps Muhammad Waqas Department of Astronomy, Institut Teknologi Bandung, Jl. Ganesha 10, Bandung 40132, Indonesia mdwaqas767@gmail. p. 0 p. 0 p. 0 p. 0 p. 0 p. 0 p. 0 p. 0 p. 0com Page 1. p. 0

--- Page 2 ---

Sky Monitor AI: classifying night-sky conditions from an observatory all-sky camera with a CNN ensemble, label-free sky types and exact evidence maps Muhammad Waqas Department of Astronomy, Institut Teknologi Bandung, Jl. Ganesha 10, Bandung 40132, Indonesia mdwaqas767@gmail.com Page 2. p. 0

--- Page 3 ---

Sky Monitor AI: classifying night-sky conditions from an observatory all-sky camera with a CNN ensemble, label-free sky types and exact evidence maps Muhammad Waqas Department of Astronomy, Institut Teknologi Bandung, Jl. Ganesha 10, Bandung 40132, Indonesia mdwaqas767@gmail.com Page 3. p. 0

--- Page 4 ---

Sky Monitor AI: classifying night-sky conditions from an observatory all-sky camera with a CNN ensemble, label-free sky types and exact evidence maps Muhammad Waqas Department of Astronomy, Institut Teknologi Bandung, Jl. Ganesha 10, Bandung 40132, Indonesia mdwaqas767@gmail.com Page 4. p. 0

--- Page 5 ---

Sky Monitor AI: classifying night-sky conditions from an observatory all-sky camera with a CNN ensemble, label-free sky types and exact evidence maps Muhammad Waqas Department of Astronomy, Institut Teknologi Bandung, Jl. Ganesha 10, Bandung 40132, Indonesia mdwaqas767@gmail.com Page 5. p. 0

--- Page 6 ---

Sky Monitor AI: classifying night-sky conditions from an observatory all-sky camera with a CNN ensemble, label-free sky types and exact evidence maps Muhammad Waqas Department of Astronomy, Institut Teknologi Bandung, Jl. Ganesha 10, Bandung 40132, Indonesia mdwaqas767@gmail.com Page 6. p. 0

--- Page 7 ---

Sky Monitor AI: classifying night-sky conditions from an observatory all-sky camera with a CNN ensemble, label-free sky types and exact evidence maps Muhammad Waqas Department of Astronomy, Institut Teknologi Bandung, Jl. Ganesha 10, Bandung 40132, Indonesia mdwaqas767@gmail.com Page 7. p. 0

--- Page 8 ---

Sky Monitor AI: classifying night-sky conditions from an observatory all-sky camera with a CNN ensemble, label-free sky types and exact evidence maps Muhammad Waqas Department of Astronomy, Institut Teknologi Bandung, Jl. Ganesha 10, Bandung 40132, Indonesia mdwaqas767@gmail.com Page 8. p. 0

--- Page 9 ---

Sky Monitor AI: classifying night-sky conditions from an observatory all-sky camera with a CNN ensemble, label-free sky types and exact evidence maps Muhammad Waqas Department of Astronomy, Institut Teknologi Bandung, Jl. Ganesha 10, Bandung 40132, Indonesia mdwaqas767@gmail.com Page 9. p. 0

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Evaluation of model capacity should be based on independent nights rather than total frames, as The effective sample size is of order the number of nights (∼24 for training), not the number of frames. This prevents overfitting by matching model capacity to available data variability.

  2. Implement a stream processing pipeline incorporating temporal smoothing to improve accuracy, as A centred rolling median of the ensemble probability raises the test accuracy from 0.867 to 0.872, 0.872, 0.877 and 0.878 for windows of 3, 5, 9 and 15 frames. This captures systematic errors rather than isolated frame noise.

  3. Develop a label-free sky type classifier using Gaussian mixture models on pretrained CNN embeddings to discover novel classes such as a wet-dome class absent from the labels, allowing operators to label new data by naming clusters.

  4. Utilize an exact, gradient-free decomposition of global average pooling networks into local logits to generate evidence maps and cloud cover estimates for real-time use, as this is the class activation mapping of Zhou et al. (2016) applied to the logit difference.

  5. Integrate a rule-based layer that maps model outputs to operational conditions like clear, overcast, mostly clear or mostly cloudy, and wet dome based on derived quantities such as c = 1/2(1-P) + 1/2 cover. This translates probabilistic predictions into actionable decisions for observatory operations.

Related papers