Learning What's Real: Disentangling Signal and Measurement Artifacts in Multi-Sensor Data, with Applications to Astrophysics

summary

The gist

Data collected from the physical world is always a combination of multiple sources: an underlying signal from the physical process of interest and a signal from measurement-dependent artifacts from…

In short

The paper proposes a dual-encoder architecture with a counterfactual generation objective to separate intrinsic physical signals from sensor-specific measurement artifacts in multi-instrument data. Trained on overlapping observations, this method learns representations that are invariant to instrumental distortions. This allows for instrument-independent similarity search and robust parameter inference, providing cleaner physical insights from noisy astronomical images.

Key concepts

Dual-Encoder Architecture
This architecture uses two specialized neural networks: a physics encoder and an instrument encoder. The physics encoder learns the intrinsic properties of the source while ignoring instrumental noise, and the instrument encoder learns the specific distortions introduced by each sensor.
Counterfactual Generation Objective
This training objective leverages overlapping observations to create 'training triplets' (anchor, different instrument observation, different source observation). This forces the model to learn representations that are invariant to measurement artifacts by generating synthetic views that differ only in the instrumental or source context.
Physics Latent Space
This latent space is designed specifically to capture underlying physical properties of astronomical objects, such as galaxy morphology. It is trained to be robust against sensor-specific distortions, meaning it recovers accurate physical characteristics even when data comes from different instruments.

Terminology used across episodes

This episode discusses

The paper

Learning What's Real: Disentangling Signal and Measurement Artifacts in Multi-Sensor Data, with Applications to Astrophysics · Read on arXiv

Massachusetts Institute of Technology · Flatiron Institute, Simons Foundation · AstroAI, CfA | Harvard & Smithsonian

Data collected from the physical world is always a combination of multiple sources: an underlying signal from the physical process of interest and a signal from measurement-dependent artifacts from the sensor or instrument. This secondary signal acts as a confounding factor, limiting our ability to extract information about the physics underlying the phenomena we observe. Furthermore, it complicates the combination of observations in heterogeneous or multi-instrument settings. We propose a deep learning framework that leverages overlapping observations, a dual-encoder architecture, and a counterfactual generation objective to disentangle these factors of variation. The resulting representations explicitly separate intrinsic signals from sensor-specific distortions and noise, and can be used for counterfactual view generation, parameter inference unconfounded by measurement distortions, and instrument-independent similarity search. We demonstrate the effectiveness of our approach on astrophysical galaxy images from the DESI Legacy Imaging Survey (Legacy) and the Hyper Suprime-Cam (HSC) Survey as a representative multi-instrument setting. This framework provides a general recipe for scientific and multi-modal self-supervised pretraining: construct training pairs from overlapping observations of the same physical system, treat sensor- or modality-specific effects as augmentations, and learn invariant representations through counterfactual generation.

Transcript

Introduction to the show: ident: Astrophysics. The day in astrophysics: observations, models and what the data forces.

Vera: We're looking at the paper: "Learning What's Real".

Jocelyn: Here's the gist of the paper. Data collected from the physical world is always a combination of multiple sources: an underlying signal from the physical process of interest and a signal from measurement-dependent artifacts from the sensor or instrument.

Vera: This secondary signal acts as a confounding factor, limiting our ability to extract information about the underlying physics.

Jocelyn: To disentangle these factors of variation, we propose a dual-encoder architecture with a counterfactual generation objective that leverages overlapping observations.

Vera: The resulting representations explicitly separate intrinsic signals from sensor-specific distortions and noise, and can be used for counterfactual view generation, parameter inference, and instrumentindependent similarity search—all unconfounded by measurement artifacts.

Title and authors: Vera: So, to summarize what they’ve done in "Learning What's Real: Disentangling Signal and Measurement Artifacts in Multi-Sensor Data, with Applications to Astrophysics," they propose a dual-encoder architecture with a counterfactual generation objective. This framework is designed specifically to separate the intrinsic physical signal from measurement artifacts caused by different sensors or instruments.

Jocelyn: They achieve this by using overlapping observations across multiple instruments—like combining data from the Legacy Survey and Hyper Suprime-Cam—to create training triplets that force the encoders to learn distinct representations for physics and instrument effects.

Subrahmanyan: The paper’s summary focuses heavily on how they construct these training triplets: an anchor observation, an instrument-augmented observation, and a physics-augmented observation. They then use the physics encoder to capture invariant features while the instrument encoder isolates the measurement distortions.

Vera: They show that by using this setup, you get separate latent variables where one space focuses on physical properties and the other on instrumental characteristics. This is what allows for downstream tasks like outlier detection or parameter inference without being confused by instrument-specific noise patterns.

Jocelyn: What’s really interesting about the summary is that they treat sensor- or modality-specific effects as augmentations during training, which helps build those robust, invariant representations through the counterfactual generation objective. It’s a recipe for self-supervised pretraining in science.

Subrahmanyan: From a theory standpoint, it moves away from relying on hand-engineered loss functions and instead enforces structural disentanglement through this specific architecture and the flow matching mechanism, which connects it to ideas about recovering independent causal mechanisms three <ref:2604.09787#pg1>.

The paper's summary: Vera: Looking at the improvements they suggest in "Learning What's Real: Disentangling Signal and Measurement Artifacts in Multi-Sensor Data, with Applications to Astrophysics," they focus on the fact that this approach is architecture-driven rather than relying on pre-defined loss functions. They’re enforcing structural disentanglement through their specific design.

Jocelyn: The improvements highlight how they use multi-view observations—seeing the same underlying content under different conditions or at different times—as a natural source of structure for disentanglement, which is something we see everywhere in real observation data.

Subrahmanyan: I think the most concrete improvement is the asymmetry in property prediction they found. The physics latent space substantially outperforms instrument latents when predicting physical properties, while the pattern reverses when it comes to instrumental properties. This asymmetry proves that their architecture actually succeeds in separating those factors efficiently.

Vera: That’s what they mean by achieving a more efficient separation of physics and instrumental properties compared to monolithic foundation models. The physics latent space recovers physical properties at a level comparable to AION-one which is a strong benchmark for this kind of feature recovery <ref:2604.09787#pg1>.

Jocelyn: And they also show that the instrument latent space does a better job at predicting those instrument-related properties than baseline models, which is useful for understanding the systematics themselves.

Subrahmanyan: If we think about the implications, this means we can trust inferred galaxy properties more across different surveys because our method is explicitly designed to be robust to those instrumental variations, even if those variations are complex.

The paper's improvements: Vera: So, wrapping up "Learning What's Real: Disentangling Signal and Measurement Artifacts in Multi-Sensor Data, with Applications to Astrophysics," the main implication is that we can now build a framework for scientific self-supervised pretraining. It’s about constructing training pairs from overlapping observations of the same physical system.

Jocelyn: This means we treat sensor or modality effects as augmentations when training and learn invariant representations through counterfactual generation, which lets us do things like instrument-independent similarity search without being confounded by artifacts.

Subrahmanyan: For cosmology, this suggests that we can more reliably use these latent spaces to streamline the discovery of rare astrophysical phenomena because the physics latent space effectively isolates and groups complex physical features.

Vera: They also confirm that their generative decoder acts like a learned survey simulator, performing instrument-aware super-resolution and denoising. This is useful for reducing search space for follow-up observations by generating informed predictions rather than just noisy pixels.

Jocelyn: It’s a general recipe here: construct training pairs from overlapping data, treat system effects as augmentations, and learn invariant representations this way. It’s applicable beyond just galaxies, to other areas of astrophysics and even tens of millions of light curves from missions like TESS or Kepler.

Subrahmanyan: I just want to say that the physics latent space recovering properties at a level comparable to AION-one is a solid piece of evidence for how effective this architecture is at isolating intrinsic features <ref:2604.09787#pg1>. It shows the method has real power for uncovering those complex structures we’re looking for in the sky.

Vera: Exactly, and it confirms that by explicitly disentangling the factors, we get a more efficient separation than monolithic foundation models. We have to keep an eye on how they extend this to settings where you don't have such perfect overlap yet.

Conclusion: Vera: So we've been looking at "Learning What's Real: Disentangling Signal and Measurement Artifacts in Multi-Sensor Data, with Applications to Astrophysics," and essentially what they did is build a dual-encoder system that separates the actual physical signal from all the noise caused by different instruments.

Jocelyn: Right, so it uses overlapping observations—like combining data from different surveys—to create these training sets where you have the same physical thing seen through different lenses.

Subrahmanyan: The core idea is to force two encoders to learn separate things: one encoder learns the physics, and the other learns how each specific instrument messes up the data.

Vera: Exactly. The results show that if you look at those latent spaces, you get a physics space that recovers real physical properties pretty well, even matching models like AION-one.

Jocelyn: And the instrument space is surprisingly good at capturing those measurement distortions themselves, which is useful for understanding the noise conditions of different instruments.

Subrahmanyan: From a cosmic view, this means we can now trust inferred galaxy properties more across different surveys because they aren't just affected by the specific sensor they were taken with.

Vera: It also gives us tools like counterfactual generation, which lets you predict what an observation would look like under different conditions or even generate a high-resolution image from a low-resolution one.

Jocelyn: That means we can use it to simulate observations or even do some kind of instrument-independent similarity search, which is pretty neat for finding similar sources across different datasets.

Subrahmanyan: The limitation they mention is that this method currently needs those overlapping observations across instruments, so you're mostly limited to regions where the sky coverage from those surveys overlaps.

Vera: It’s a powerful tool for self-supervised pretraining because it gives us a general recipe: take overlapping data, treat the sensor effects as augmentations during training, and learn these invariant representations.

Jocelyn: It’s a solid framework for moving forward, especially since they show how this architecture can be applied to tens of millions of light curves from missions like TESS or Kepler.

Subrahmanyan: That's where the real potential lies, applying it to those massive datasets to find patterns in structure formation that we just can't see otherwise.

Vera: So, for anyone listening, this paper is about building a way to clean up messy observational data by learning what's physically real versus what the equipment is adding on top of it.

Jocelyn: It shows how combining different instruments can actually help us understand the underlying physics better than using just one telescope alone.

Subrahmanyan: We'll keep an eye on how they extend this to unpaired settings, because that's where the real challenge in observational science often lies.

More episodes

← Home