Object-independent evaluation of small-data astronomical image classifiers: architecture, representation, and acquisition shortcuts
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Astrophysics. The day in astrophysics: observations, models and what the data forces.
Vera: I'm Vera.
Jocelyn: And I'm Jocelyn, here with Subrahmanyan.
Vera: We're looking at the paper: "Object-independent evaluation of small-data astronomical image classifiers".
Jocelyn: Here's the gist of the paper. Reported performance in small astronomical image datasets can reflect object reuse, field-of-view differences, acquisition settings, and catalogue-derived properties as well as astronomical morphology <ref:2611.06484#pg6> The AstroClassifier project began by asking whether a hybrid convolutional neural network (CNN) and Transformer could classify astronomical images with few labeled examples <ref:2611.06484#pg4> The central question became: how much class information transfers to previously unseen physical objects when object identity, representation, and acquisition cues are explicitly examined <ref:2611.06484#pg4> Dataset Composition and Design The dataset contains 370 cataloguesupported physical objects, each represented by three Digitized Sky Survey FITS bands, and uses a group-constrained split of 222 training, 75 validation, and 73 test objects <ref:2611.06484#pg2> The resulting eligible pool contains 370 physical objects: 98 galaxies, 92 globular clusters, 85 nebulae, and 95 open clusters <ref:2611.06484#pg4> Each has one observation bundle containing DSS2 blue, red, and near-infrared FITS products retrieved through the CDS HiPS2FITS service <ref:2611.06484#pg4> The pool therefore contains 1,110 FITS files but only 370 distinct astronomical targets <ref:2611.06484#pg4> Quality eligibility was fixed before model training <ref:2611.06484#pg4> Representation Variants and Controls Four deterministic variants isolate different information: Adaptive full field (A), Common field (B), Central-region-masked controls, and metadata-only controls <ref:2611.06484#pg4> The adaptive full field (A) is retained and resized by exact area-overlap averaging to preserve context and class-dependent FOV <ref:2611.06484#pg4> The common field (B) resamples each target to a 4-arcmin tangent-plane field with 15 arcsec pixels <ref:2611.06484#pg4> Central-region-masked controls set the central region to zero, and stretch percentiles use unmasked pixels only <ref:2611.06484#pg4> Two fixed controls use no pixels, measuring available non-image signal like log source FOV and catalogue extent <ref:2611.06484#pg4> Model Comparison and Training The model families reflect the earlier exploratory history of AstroClassifier, including a custom CNN–Transformer trained from scratch, ResNet-50 pretrained on ImageNet-1K weights, and ViT-B/16 pretrained on ImageNet-1K weights <ref:2611.06484#pg4> For pretrained models, the new head uses ten times the backbone learning rate and all layers are trainable from the first epoch <ref:2611.06484#pg4> The custom model is trained from scratch, using four convolutional stages producing 1,024 channels and followed by two Transformer encoder layers <ref:2611.06484#pg4> Evaluation Outcomes and Interpretation The test establishes a limited but meaningful result: all three families generalized to catalogue-supported physical objects absent from model development <ref:2611.06484#pg4> The controls show that class labels covary with acquisition choices, cataloguederived object extent, and image content outside the central mask <ref:2611.06484#pg4> The strongest methodological finding is that target identity is not the only source of predictive information <ref:2611.06484#pg4> Combined acquisition and catalogue metadata reached balanced accuracy above 0.72 <ref:2611.06484#pg4> High accuracy on unseen objects is evidence of generalization within the collection, but it is not, by itself, evidence that a model learned the intended astronomical morphology <ref:2611.06484#pg4> The comparison does not isolate architecture from pretraining because the near-zero ResNet–ViT differences provide no evidence that convolution or global attention is intrinsically preferable here <ref:2611.06484#pg4> The test establishes a limited but meaningful result: all three families generalized to catalogue-supported physical objects absent from model development <ref:2611.06484#pg4> The custom hybrid did not realize its motivating expectation of an advantage in this small-data setting <ref:2611.06484#pg4> The controls change the scientific interpretation more than the architecture ranking <ref:2611.06484#pg4> Reliable smalldata evaluation therefore requires physical-object split units, explicit shortcut controls, and pre-specified one-pass testing <ref:2611.06484#pg4> The experiment provides no evidence that this evaluated hybrid system is preferable to the two pretrained references <ref:2611.06484#pg4> Reproducibility Measures Each phase records data provenance, lineage, split membership, configurations, checkpoints, and execution records <ref:2611.06484#pg4> Independent reconstruction of all 1,480 tensors (370 objects times four variants) verified identical numerical representations from the source FITS products <ref:2611.06484#pg4> Test inference was performed once with a label-isolated procedure, and all 38 model evaluations recorded complete predictions before the scoring process accessed test labels <ref:2611.06484#pg4> The curated AstroClassifier reproducibility archive for this study is publicly available at https://github.com/epmarinho/astroclassifier-repo <ref:2611.06484#pg4> References Baron, D., 2019.
Vera: Machine learning in astronomy: A practical overview <ref:2611.06484#pg4> Boch, T., Fernique, P., Bonnarel, F., Chaitra, Bot, C., Pineau, F.X., Baumann, M., Michel, L., 2020.
Jocelyn: HiPS2FITS: Fast generation of FITS cutouts from HiPS image datasets <ref:2611.06484#pg4> Dieleman, S., Willett, K.W., Dambre, J., 2015.
Vera: Rotation-invariant convolutional neural networks for galaxy morphology prediction <ref:2611.06484#pg4> Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N.
Paper summary: Vera: So, we're looking at this paper right now: "Object-independent evaluation of small-data astronomical image classifiers: architecture, representation, and acquisition shortcuts." It starts by pointing out that performance on small astronomy datasets can be fooled by things like object reuse or how the data was acquired.
Jocelyn: Exactly. The authors designed a dataset called AstroClassifier to test these specific effects across four types of objects: galaxies, globular clusters, nebulae, and open clusters. They set up a split with two hundred twenty-two training objects and seventy-three test objects for this evaluation <ref:2611.06484#pg1> <ref:2611.06484#pg1>.
Subrahmanyan: That's the setup they used to measure things like object reuse or field-of-view differences on those small samples <ref:2611.06484#pg1>. I wonder how much of that observed performance actually comes from learning the target shape versus just knowing where it came from, and we'll see what this paper says about that later.
Vera: Well, what this paper claims is that they compared three different model families: a custom CNN Transformer trained from scratch, a ResNet-fifty model pretrained on ImageNet weights, and a ViT B/sixteen model also pretrained on ImageNet weights <ref:2611.06484#pg1> <ref:2611.06484#pg1,a custom CNN Transformer trained from scratch>.
Jocelyn: They tested these models under different visual settings, specifically an adaptive full-field representation, a common angular field, and two controls that just mask the center region <ref:2611.06484#pg1>.
Subrahmanyan: The main result they highlight is that in their one-pass object-independent test on the adaptive view, the mean balanced accuracy for all three families was zero point seven seven three plus or minus zero point fourteen for the custom model, and zero point eight hundred seventy-four plus or minus zero point fifty-four for the ResNet-fifty and another one at another one thousand sixteen <ref:2611.06484#pg1>.
Vera: That's a pretty high number for small data, so they go on to check if the architecture really matters by looking at their comparisons <ref:2611.06484#pg1>. They found no primary contrast between the custom model and the pre-trained ones after applying Holm correction, meaning ResNet-fifty and ViT B/sixteen only differed by zero point seven balanced accuracy points, which is a very small difference compared to their uncertainty of plus minus ninety-five percent cluster bootstrap interval of negative zero point forty eight to positive zero point sixty two <ref:2611.06484#pg1>.
Jocelyn: And they also looked at the controls—those based on acquisition and catalogue metadata—which reached a balanced accuracy between zero point seven hundred twenty-one and plus or minus zero point seven hundred thirty-five and the central-region-masked adaptive views hit between zero point seven hundred thirty-one and plus or minus one.eight hundred <ref:2611.06484#pg1>.
Subrahmanyan: So what this really tells us is that while they could generalize to unseen physical objects, it also shows that image context, acquisition settings, and catalogue derived object properties carry substantial class information <ref:2611.06484#pg1>. It's not just the object itself driving the prediction.
Vera: I think that’s a critical point, Subrahmanyan. The paper says this means high accuracy on unseen objects isn't proof that a classifier learned the intended astronomical morphology <ref:2611.06484#pg1>]. It’s more about what you feed it and where it comes from.
Jocelyn: Right, so the comparison doesn't isolate the architecture from pretraining because there is no evidence that convolution or global attention is intrinsically preferable here <ref:2611.06484#pg1>]. It suggests that in this small-data regime, we have to be careful about what we conclude about the model structure itself.
Subrahmanyan: I agree. The paper also states that augmentation can improve generalization, but derivatives of one object remain statistically dependent, which means you can't just rely on those derived examples for strong results <ref:2611.06484#pg3>]. It’s a reminder that augmentation isn't a magic fix for small datasets because the underlying dependency stays there.
Vera: And they also point out that leakage and inappropriate split units are recognized threats to reproducibility in machine learning based science, which is something we all have to watch out for when building these systems <ref:2611.06484#pg9>]. It’s not just about the model weights; it’s about how you set up the test.
Jocelyn: So, this paper sets a pretty clear standard for how we should evaluate these hybrid CNN Transformer models when they're applied to astronomical images with limited labeled examples <ref:2611.06484#pg2>]. It forces us to look beyond just the accuracy number and consider all those input variables.
Subrahmanyan: What this means for the bigger picture is that when we try to build models for cosmology or structure formation, we need to be extremely careful about whether our training set contains enough diversity in acquisition settings and object types so that the model isn't just learning artifacts from those specific conditions <ref:2611.06484#pg3>]. We need more robust ways to test generalization beyond just accuracy on held-out objects.
Vera: It does suggest that for now, high accuracy in this setup is evidence of generalization within this specific collection, but it doesn't confirm the model learned the actual target morphology <ref:2611.06484#pg1>]. It keeps us grounded in reality about what these models are actually doing.
Jocelyn: So to wrap up, this paper shows that object-independent evaluation requires explicitly examining object identity and acquisition cues to see where the class information is coming from <ref:2611.06484#pg4>]. It’s a methodological warning for anyone trying to use deep learning on small astronomical samples.
Subrahmanyan: Indeed, it suggests that reliable small data evaluation needs physical-object split units and explicit shortcut controls before we can make strong claims about how well an architecture performs <ref:2611.06484#pg4>]. The comparison of architectures alone isn't enough; you have to look at the whole group.
Conclusion: Vera: So, we're wrapping up this look at the paper "Object-independent evaluation of small-data astronomical image classifiers." It basically shows how we can test if a model is actually learning the shape of an object or just memorizing where it came from.
Jocelyn: Right, it’s all about those shortcuts—like the way you take a picture or which catalog you use—and whether that information helps the AI guess correctly on new objects.
Subrahmanyan: The authors are showing that we need to look beyond just the model structure, like whether we use a CNN or a Transformer, because those differences are often tiny in these small datasets.
Vera: They did this by setting up four different visual conditions—an adaptive field, a common field, and two control methods based on acquisition settings and catalogue data.
Jocelyn: The results show that class labels are really tied to things like the extent of the object in the catalogue and even what parts of the image you decide to look at.
Subrahmanyan: The main point is that high accuracy on unseen objects doesn't automatically mean a model learned how galaxies or clusters look; it just means it learned those background cues.
Vera: So, this means for small data science, we have to treat the acquisition settings and the catalogue info as just as important as the pixels themselves when we’re checking if a model generalizes.
Jocelyn: It's a bit of a reality check for anyone building these systems; you can’t just look at one number and assume everything is fine.
Subrahmanyan: And this leads to the bigger question—if we want models that work for real cosmology, we need training sets that cover more diversity in those acquisition choices so they don't just learn artifacts from a specific survey.
Vera: So the paper concludes that reliable small data evaluation needs physical-object splits and explicit controls before you can really trust a model’s performance on new stuff.
Jocelyn: It’s less about which architecture is superior and more about understanding what information is actually being transferred when the object identity changes.
Subrahmanyan: This opens up a lot of work for future research into how we can design training protocols that truly separate morphology from those acquisition shortcuts you mentioned earlier.
Eraldo Pereira Marinhoa
Institute of Geoscience and Exact Sciences, São Paulo State University (UNESP)
astro-ph.IM, cs.CV
Submitted: 2026-11-04
Updated: 2026-11-04
Comments: 17 pages and 3 figures
Code: https://github.com/epmarinho/astroclassifier-r
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Importance score: 90/100
The gist: Reported performance in small astronomical image datasets can reflect object reuse, field-of-view differences, acquisition settings…
Key concepts
- Object-independent evaluation
- This means testing a model on physical objects it has never seen before. The goal is to see if the model truly learned the object's nature or just learned shortcuts related to how the image was taken or cataloged.
- Representation Variants
- These are different ways of preparing an astronomical image for a computer, such as using an adaptive full field (preserving context), a common field (resampling to a standard size), or central-region masks. They test how much information is lost or preserved during image preparation.
- Acquisition and Catalogue Metadata
- These are pieces of information about the image that are not part of the pixels themselves, like the telescope settings (acquisition) or where the object is listed in a catalogue (metadata). The study found these factors strongly correlate with class labels, suggesting they act as powerful shortcuts for classification.
Summary
Reported performance in small astronomical image datasets can reflect object reuse, field-of-view differences, acquisition settings, and catalogue-derived properties as well as astronomical morphology <ref:2611. p. 006484#pg6> The AstroClassifier project began by asking whether a hybrid convolutional neural network (CNN) and Transformer could classify astronomical images with few labeled examples <ref:2611. p. 106484#pg4> The central question became: how much class information transfers to previously unseen physical objects when object identity, representation, and acquisition cues are explicitly examined <ref:2611. p. 1 p. 106484#pg4>
Dataset Composition and Design
The dataset contains 370 cataloguesupported physical objects, each represented by three Digitized Sky Survey FITS bands, and uses a group-constrained split of 222 training, 75 validation, and 73 test objects <ref:2611.06484#pg2> The resulting eligible pool contains 370 physical objects: 98 galaxies, 92 globular clusters, 85 nebulae, and 95 open clusters <ref:2611. p. 306484#pg4> Each has one observation bundle containing DSS2 blue, red, and near-infrared FITS products retrieved through the CDS HiPS2FITS service <ref:2611. p. 306484#pg4> The pool therefore contains 1,110 FITS files but only 370 distinct astronomical targets <ref:2611. p. 306484#pg4> Quality eligibility was fixed before model training <ref:2611. p. 306484#pg4>
Representation Variants and Controls
Four deterministic variants isolate different information: Adaptive full field (A), Common field (B), Central-region-masked controls, and metadata-only controls <ref:2611.06484#pg4> The adaptive full field (A) is retained and resized by exact area-overlap averaging to preserve context and class-dependent FOV <ref:2611. p. 406484#pg4> The common field (B) resamples each target to a 4-arcmin tangent-plane field with 15 arcsec pixels <ref:2611. p. 406484#pg4> Central-region-masked controls set the central region to zero, and stretch percentiles use unmasked pixels only <ref:2611. p. 406484#pg4> Two fixed controls use no pixels, measuring available non-image signal like log source FOV and catalogue extent <ref:2611. p. 606484#pg4>
Model Comparison and Training
The model families reflect the earlier exploratory history of AstroClassifier, including a custom CNN–Transformer trained from scratch, ResNet-50 pretrained on ImageNet-1K weights, and ViT-B/16 pretrained on ImageNet-1K weights <ref:2611.06484#pg4> For pretrained models, the new head uses ten times the backbone learning rate and all layers are trainable from the first epoch <ref:2611. p. 706484#pg4> The custom model is trained from scratch, using four convolutional stages producing 1,024 channels and followed by two Transformer encoder layers <ref:2611. p. 606484#pg4>
Evaluation Outcomes and Interpretation
The test establishes a limited but meaningful result: all three families generalized to catalogue-supported physical objects absent from model development <ref:2611.06484#pg4> The controls show that class labels covary with acquisition choices, cataloguederived object extent, and image content outside the central mask <ref:2611. p. 006484#pg4> The strongest methodological finding is that target identity is not the only source of predictive information <ref:2611. p. 606484#pg4> Combined acquisition and catalogue metadata reached balanced accuracy above 0. p. 072 <ref:2611. p. 006484#pg4> High accuracy on unseen objects is evidence of generalization within the collection, but it is not, by itself, evidence that a model learned the intended astronomical morphology <ref:2611. p. 206484#pg4> The comparison does not isolate architecture from pretraining because the near-zero ResNet–ViT differences provide no evidence that convolution or global attention is intrinsically preferable here <ref:2611. p. 106484#pg4>
Conclusion on Generalization
The test establishes a limited but meaningful result: all three families generalized to catalogue-supported physical objects absent from model development <ref:2611.06484#pg4> The custom hybrid did not realize its motivating expectation of an advantage in this small-data setting <ref:2611.06484#pg4> The controls change the scientific interpretation more than the architecture ranking <ref:2611.06484#pg4> Reliable smalldata evaluation therefore requires physical-object split units, explicit shortcut controls, and pre-specified one-pass testing <ref:2611. p. 206484#pg4> The experiment provides no evidence that this evaluated hybrid system is preferable to the two pretrained references <ref:2611.06484#pg4>
Reproducibility Measures
Each phase records data provenance, lineage, split membership, configurations, checkpoints, and execution records <ref:2611.06484#pg4> Independent reconstruction of all 1,480 tensors (370 objects times four variants) verified identical numerical representations from the source FITS products <ref:2611.06484#pg4> Test inference was performed once with a label-isolated procedure, and all 38 model evaluations recorded complete predictions before the scoring process accessed test labels <ref:2611. p. 806484#pg4> The curated AstroClassifier reproducibility archive for this study is publicly available at https://github.com/epmarinho/astroclassifier-repo <ref:2611. p. 0 p. 006484#pg4>
References
Baron, D., 2019. Machine learning in astronomy: A practical overview <ref:2611. p. 206484#pg4> Boch, T. p. 3, Fernique, P. p. 3, Bonnarel, F., Chaitra, Bot, C., Pineau, F.X., Baumann, M., Michel, L., 2020. HiPS2FITS: Fast generation of FITS cutouts from HiPS image datasets <ref:2611. p. 306484#pg4> Dieleman, S. p. 1, Willett, K.W., Dambre, J., 2015. Rotation-invariant convolutional neural networks for galaxy morphology prediction <ref:2611. p. 106484#pg4> Dosovitskiy, A. p. 2, Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N. 2021. An image is worth 16x16 words: Transformers for image recognition at scale <ref:2611. p. 206484#pg4> Fernique, P. p. 3, Allen, M.G., Boch, T., Oberto, A., Pineau, F.X., Durand, D., Bot, C., Cambrésy, L., Derriere, S., Genova, F., Bonnarel, F. 2015. Hierarchical progressive surveys: Multi-resolution HEALPix data structures for astronomical images <ref:2611. p. 106484#pg4> Geirhos, R. p. 1, Jacobsen, J.H., Michaelis, C., Zemel, R., Brendel, W., Bethge, M., Wichmann, F.A. 2020. Shortcut learning in deep neural networks <ref:2611. p. 106484#pg4> Harris, W. p. 3E., 1996. A catalog of parameters for globular clusters in the milky way <ref:2611. p. 306484#pg4> He, K., Zhang, X., Ren, S., Sun, J., 2016. Deep residual learning for image recognition <ref:2611. p. 106484#pg4> Huertas-Company, M. p. 1, Lanusse, F. p. 1, 2018. Deep learning for galaxy morphology classification <ref:2611. p. 106484#pg4> Kapoor, S. p. 1, Narayanan, A. p. 1, 2023. Leakage and the reproducibility crisis in machine-learning-based science <ref:2611. p. 206484#pg4> Khan, S. p. 2, Naseer, M., Hayat, M., Zamir, S.W., Khan, F.S., Shah, M., 2022. Transformers in vision: A survey <ref:2611. p. 106484#pg4> Shorten, C. p. 2, Khoshgoftaar, T. p. 2M., 2019. A survey on image data augmentation for deep learning <ref:2611. p. 106484#pg4> Vaswani, A. p. 2, Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I. 2017. Attention is all you need <ref:2611. p. 006484#pg4> Wenger, M. p. 3, Ochsenbein, F., Egret, D., Dubois, P., Bonnarel, F., Borde, S., Genova, F., Jasniewicz, G., Laloë, S., Lesteven, S., Monier, R. 2000. The SIMBAD astronomical database: The CDS reference database for astronomical objects <ref:2611. p. 106484#pg4>
--- Page 1 ---
Object-independent evaluation of small-data astronomical image classifiers: architecture, representation, and acquisition shortcuts Eraldo Pereira Marinhoa,∗ a Institute of Geoscience and Exact Sciences, São Paulo State University (UNESP), Av. 24 A, 1515, Rio Claro, 13506-900, São Paulo, Brazil Abstract Reported performance in small astronomical image datasets can reflect object reuse, field-of-view differences, acquisition settings, and catalogue-derived properties as well as astronomical morphology <ref:2611. p. 006484#pg4> We designed AstroClassifier to measure these effects in a four-class task comprising galaxies, globular clusters, nebulae, and open clusters <ref:2611. p. 006484#pg4> The dataset contains 370 cataloguesupported physical objects, each represented by three Digitized Sky Survey FITS bands, and uses a group-constrained split of 222 training, 75 validation, and 73 test objects <ref:2611. p. 006484#pg4> We compared a custom CNN–Transformer trained from scratch with ImageNet-pretrained ResNet-50 and ViT-B/16 under an adaptive full-field representation, a common angular field, and two central-region-masked controls, using three fixed seeds <ref:2611. p. 006484#pg4> In the one-pass object-independent test, mean balanced accuracy on the adaptive view was 0. p. 0773±0.014, 0.874±0.054, and 0. p. 0866±0.023 for the custom model, ResNet50, and ViT-B/16, respectively <ref:2611. p. 006484#pg4> No primary architecture contrast rejected its null after Holm correction; ResNet-50 and ViT-B/16 differed by 0. p. 0007 balanced-accuracy points (95% cluster-bootstrap interval [−0. p. 0048, 0.062]) <ref:2611. p. 006484#pg4> Controls based on acquisition and catalogue metadata reached 0. p. 0721–0.735, and central-region-masked adaptive views reached 0. p. 0731–0.800 <ref:2611. p. 006484#pg4> These results demonstrate generalization to unseen physical objects, but also show that image context, acquisition settings, and catalogue-derived object properties carry substantial class information <ref:2611. p. 006484#pg4> In this regime, high accuracy cannot be interpreted as evidence that a classifier learned target morphology <ref:2611. p. 106484#pg4>
--- Page 2 ---
Keywords astronomical image classification, small datasets, object-independent evaluation, shortcut learning, convolutional neural networks, vision transformers Keywords: astronomical image classification <ref:2611.06484#pg4>
--- Page 3 ---
- Introduction Deep learning is widely used for astronomical image analysis, from galaxy morphology to survey triage Dieleman et al. p. 1 (2015); Huertas-Company and Lanusse (2018); Baron (2019) Its apparent success can be difficult to interpret when datasets contain many files but few independent astronomical objects <ref:2611. p. 106484#pg4> Multiple bands, renders, crops, or observations of the same target do not create independent physical examples <ref:2611. p. 106484#pg4> Related products that cross data splits can yield high scores without demonstrating generalization to an unseen object <ref:2611. p. 106484#pg4> Class-dependent acquisition choices, including field of view (FOV), pixel scale, and target extent, can also become shortcuts Geirhos et al. p. 1 (2020); Kapoor and Narayanan (2013)
--- Page 4 ---
The AstroClassifier project began by asking whether a hybrid convolutional neural network (CNN) and Transformer could classify astronomical images with few labeled examples <ref:2611.06484#pg4> An audit of the original experiment show that a credible answer required a different statistical design <ref:2611. p. 106484#pg4> The central question became: how much class information transfers to previously unseen physical objects when object identity, representation, and acquisition cues are explicitly examined <ref:2611.06484#pg4>
--- Page 5 ---
We make four contributions: 1 a catalogue-supported dataset in which the physical object, rather than a band image or derivative, is the observational unit <ref:2611.06484#pg4> 2 a prospectively specified Dataset-v2 comparison of a custom CNN–Transformer, ResNet-50, and ViT-B/16 <ref:2611. p. 106484#pg4> 3 matched common-field, central-region-masked, and metadata-only controls that expose class information outside the nominal target morphology <ref:2611. p. 106484#pg4> 4 a one-pass, label-isolated test procedure that records predictions before an independent scoring process accesses the labels <ref:2611. p. 106484#pg4>
--- Page 7 ---
This manuscript reports the prospectively specified Dataset-v2 evaluation on 73 previously unseen objects <ref:2611.06484#pg4> The label-isolated results separate evidence of objectindependent prediction from the stronger claim that the models learned target morphology <ref:2611. p. 206484#pg4>
--- Page 8 ---
Small-data comparisons must distinguish architectural capacity from pretraining, augmentation, and model selection <ref:2611.06484#pg4> Augmentation can improve generalization Shorten and Khoshgoftaar (2019), but derivatives of one object remain statistically dependent <ref:2611. p. 2 p. 2 p. 206484#pg4> High validation accuracy may also arise from shortcuts that remain stable within a collection but fail under acquisition or identity shifts <ref:2611. p. 2 p. 2 p. 206484#pg4> We therefore treat the architecture comparison as one component of a group-aware evaluation <ref:2611. p. 206484#pg4>
--- Page 9 ---
Small-data comparisons must distinguish architectural capacity from pretraining, augmentation, and model selection <ref:2611.06484#pg4> Augmentation can improve generalization Shorten and Khoshgoftaar (2019), but derivatives of one object remain statistically dependent <ref:2611.06484#pg4> High validation accuracy may also arise from shortcuts that remain stable within a collection but fail under acquisition or identity shifts <ref:2611.06484#pg4> Leakage and inappropriate split units are recognized threats to reproducibility in machine-learning-based science Kapoor and Narayanan (2023) We therefore treat the architecture comparison as one component of a group-aware evaluation <ref:2611. p. 2 p. 206484#pg4>
--- Page 10 ---
Small-data comparisons must distinguish architectural capacity from pretraining, augmentation, and model selection <ref:2611.06484#pg4> Augmentation can improve generalization Shorten and Khoshgoftaar (2019), but derivatives of one object remain statistically dependent <ref:2611.06484#pg4> High validation accuracy may also arise from shortcuts that remain stable within a collection but fail under acquisition or identity shifts <ref:2611.06484#pg4> Leakage and inappropriate split units are recognized threats to reproducibility in machine-learning-based science Kapoor and Narayanan (2023) We therefore treat the architecture comparison as one component of a group-aware evaluation <ref:2611.06484#pg4>
--- Page 11 ---
The controls change the scientific interpretation more than the architecture ranking <ref:2611.06484#pg4> Combined acquisition and catalogue metadata and centralregion-masked images retained substantial predictive accuracy, showing that acquisition settings, catalogue-derived properties, and contextual image content are informative even under object-independent testing <ref:2611. p. 006484#pg4>
--- Page 12 ---
The common-field representation reduces explicit FOV variation but is not a clean causal intervention on FOV <ref:2611.06484#pg4> Its effective 16 × 16 sampling blurs compact targets, can truncate extended structure, and is then repeated to 224 × 224 <ref:2611. p. 406484#pg4> Consequently, A–B differences combine angular context, scale normalization, and information loss <ref:2611.06484#pg4>
--- Page 13 ---
The comparison is also deliberately practical rather than architecturepure <ref:2611.06484#pg4> The custom model was trained from scratch, whereas ResNet-50 and ViT-B/16 received ImageNet initialization <ref:2611. p. 606484#pg4> Their advantage combines architecture, parameterization, optimization, and transfer learning <ref:2611. p. 706484#pg4> The two metadata-only controls also generalized: the depth-3 decision tree obtained 0. p. 9735 balanced accuracy, 0. p. 9 p. 9732 macro-F1, and 0. p. 9 p. 9740 accuracy <ref:2611. p. 9 p. 906484#pg4>
--- Page 14 ---
The two metadata-only controls also generalized: the depth-3 decision tree obtained 0.735 balanced accuracy, 0.732 macro-F1, and 0.740 accuracy <ref:2611.06484#pg4>
--- Page 15 ---
The curated AstroClassifier reproducibility archive for this study is publicly available at https://github.com/epmarinho/astroclassifier-repo <ref:2611.06484#pg4> The version 1.0.0 of the software and reproducibility archive is preserved at https://doi. p. 2org/10.5281/zenodo.23815956 <ref:2611. p. 006484#pg4>
--- Page 16 ---
The original AstroClassifier was independently developed by the author <ref:2611.06484#pg4> Subsequent improvements to the original implementation, extensive experimental testing, and dataset audits were carried out with the assistance of OpenAI Codex <ref:2611. p. 2
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Generalization to Unseen Objects: The system can be evaluated on physical objects absent from model development, as shown by
all three families generalized to catalogue-supported physical objects absent from model development.
This demonstrates thathigh accuracy cannot be interpreted as evidence that a classifier learned target morphology.
-
Shortcut Identification and Control: The system can identify when performance is due to non-morphological cues. Specifically, it can detect when
image context, acquisition settings, and catalogue-derived object properties carry substantial class information
by comparing results against controls likeacquisition and catalogue metadata,
which are shown to be informative even under object-independent testing. -
Architectural Comparison Robustness: The system can perform a more nuanced comparison of architectures. Instead of claiming one architecture is intrinsically superior, the system can recognize that
the comparison does not isolate architecture from pretraining
because the differences between ResNet-50 and ViT-B/16 were statistically insignificant after Holm correction. -
Representation Sensitivity Analysis: The system can quantify how different image representations impact performance. It can determine that
Adaptive full field (A)
yields high accuracy, whileCommon field (B)
reduces mean validation performance for every family, showing thatA–B differences combine angular context, scale normalization, and information loss.
-
Contextual Masking Efficacy: The system can assess the predictive signal retained outside of nominal target morphology. It can show that
central-region-masked images retained substantial predictive signal,
with Aouter views achieving high balanced accuracy between 0.731 and 0.800, even when the central region was zeroed out.
Sources
Related papers
- On-sky demonstration of a vector Zernike wavefront sensor in a cascaded adaptive optics system
- Euclid Data Release 1 (DR1): Calibration strategy and observations
- The General Antiparticle Spectrometer (GAPS) Antarctic Balloon Payload
- Detection and excision of radio-frequency interference in radio arrays
- Unified matched-filter and excess-power pipeline for gravitational-wave detector characterization
- Ushering in sub-arcsecond angular resolution observatories in the far-infrared: The case for a double-Fourier interferometer