ViViViT: a Video-to-Video Vision Transformer for Cosmological Lightcone Reconstruction

summary

The gist

Moving along the line of sight of a cosmological lightcone also moves through cosmic time, as in a video of cosmic structure traced by an evolving observable The reconstruction of the underlying…

In short

The ViViViT architecture translates observed 21 cm radio lightcones into underlying matter fields, treating cosmic evolution as a video sequence. It achieves high fidelity by factorizing attention into spatial and temporal parts, preserving spectral power and phase alignment of the reconstructed matter field with ground truth. This allows for direct reconstruction of the matter distribution from observable signals.

Key concepts

Lightcone Reconstruction
This is the core task where AI maps observed radio signals (like 21 cm neutral hydrogen data) along a line of sight into a direct map of the underlying cosmic matter density field. The lightcone geometry links distance and time, allowing the reconstruction to be viewed as a video of cosmic structure evolving over time.
ViViViT Architecture
This is a deterministic encoder-decoder transformer designed specifically for this task. It uses factorized attention—splitting attention into spatial (where) and temporal (when)—to efficiently process the 3D spacetime data, which prevents the computational cost from becoming too large for complex cosmic volumes.
Differential Brightness Temperature ($\delta T_{21}$)
This is the observable signal measured by radio telescopes. It is not a direct measure of matter density because it is heavily influenced by many factors like gas temperature, neutral hydrogen fraction, and line-of-sight velocities. The goal of the model is to reverse this modulation to find the true matter distribution.
Video-to-Video Translation
This frames the problem as a video processing task. The input is a video of cosmic structure traced by an evolving observable (the lightcone), and the output is a reconstructed 'video' of the underlying matter field. This approach leverages transformer models to learn how to translate between these two representations.

Terminology used across episodes

This episode discusses

The paper

ViViViT: a Video-to-Video Vision Transformer for Cosmological Lightcone Reconstruction · Read on arXiv

Federico Semenzato, Alvise Raccanelli, Michele Liguori

Dipartimento di Fisica e Astronomia “G. Galilei”, Universitaçy degli Studi di Padova · INFN, Sezione di Padova · Department of Physics and Astronomy, Johns Hopkins University · INAFOsservatorio Astronomico di Padova

Moving along the line of sight of a cosmological lightcone also moves through cosmic time, as in a video of cosmic structure traced by an evolving observable. We recast the reconstruction of the underlying matter field from observed lightcones as a video-to-video translation task by introducing ViViViT, a vision transformer that factorizes attention across transverse frames and radial evolution. On reconstructed matter fields from 21cm neutral hydrogen simulations over z in[7.66,16.00], ViViViT preserves spectral power and phase alignment within a few percent of the ground truth up to the Nyquist scale, and three-point correlations for all triangle configurations with sides up to k 0.77,h/Mpc, outperforming a convolutional 3D-U-Net baseline across all measured statistics. By encoding the evolving geometry of the cosmological observable itself, ViViViT marks a step toward field-level architectures that natively respect cosmic time, opening the path to time-aware reconstruction of cosmic tracers and targeted searches for redshift-dependent signatures in 21cm data.

Transcript

Introduction to the show: ident: Astrophysics. The day in astrophysics: observations, models and what the data forces.

Vera: I'm Vera.

Jocelyn: And I'm Jocelyn, here with Subrahmanyan.

Vera: We're looking at the paper: "ViViViT: a Video-to-Video Vision Transformer for Cosmological Lightcone Reconstruction".

Jocelyn: Here's the gist of the paper. Moving along the line of sight of a cosmological lightcone also moves through cosmic time, as in a video of cosmic structure traced by an evolving observable The reconstruction of the underlying matter field from observed lightcones is recast as a video-to-video translation task by introducing ViViViT On reconstructed matter fields from 21 cm neutral hydrogen simulations over z ∈

7.66, 16.00: , ViViViT preserves spectral power and phase alignment within a few percent of the ground truth up to the Nyquist scale The Problem Addressed The differential brightness temperature δT21 is not a direct image of the matter density δm because it is modulated through several factors including the neutral hydrogen fraction, gas and spin temperatures, radiation backgrounds, and line-of-sight velocities The matter distribution shapes this signal through both the gas density and the abundance and clustering of galaxies whose radiation heats and ionizes the surrounding intergalactic medium Reconstructing the matter distribution would give direct access to the matter field shared by all cosmological tracers The ViViViT Architecture ViViViT is a deterministic encoder–decoder transformer based on the ViViT architecture of

35: that factorizes attention into spatial and temporal components By factorizing attention into spatial and temporal components, our approach breaks the quadratic cost of full spacetime attention, allowing us to scale the model to larger volumes The encoder forms a global contextual representation of the input tokens The decoder uses self- and cross-attention, followed by a local convolutional upsampling head that restores the full grid Performance Metrics and Results ViViViT outperforms a convolutional U-Net baseline across all measured statistics ViViViT retains finer density structure than the tested U-Net on a held-out longitudinal slice ViViViT improves both small-scale power recovery and cross-correlation with the target field relative to the tested U-Net ViViViT reconstructs the ground-truth bispectrum to within about 5% for most retained triangle configurations, while the U-Net reconstruction error grows toward smaller scales and lower redshift Key Advantages and Limitations The geometry of the lightcone implies that the radial axis is simultaneously comoving distance and look-back time, allowing the sequence to be interpreted as a video of cosmic evolution ViViViT retains sharper features and better preserves the texture of the density field compared to a baseline tending to produce overly smooth fields The model can exploit long-range radial information when it is physically present, as demonstrated in a synthetic benchmark where it reconstructs the target at the 10−3 level at every τ A limitation is that the mapping is a deterministic reconstruction and does not preserve posterior uncertainty Future Directions In future work, our aim is to adapt this infrastructure to conditional sampling, accessing the conditional distribution rather than its median Applications to real datasets must include observational and astrophysical systematics and project the physical field to an instrumental measurement ViViViT extends naturally to multi-channel input, exploiting cross-correlations between tracers to potentially break the foreground degeneracy Beyond this reconstruction, a natural possible extension is to target redshift-dependent physical processes whose signature is distributed across the lightcone The authors state that ViViViT is the first video-to-video transformer applied to cosmological matterfield lightcone reconstruction Acknowledgments and Disclosure The authors thank Nicola Bellomo, Jessie de Kruijf, Sarah Libanore, Phoenix, Francesco Spezzati and Benjamin Wandelt for insightful feedback and discussions This work is partly supported by ICSC - Centro Nazionale di Ricerca in High Performance Computing, Big Data and Quantum Computing, funded by European Union - NextGenerationEU ML acknowledges support by the MIUR Progetti di Ricerca di Rilevante Interesse Nazionale (PRIN) Bando 2022 - grant 20228RMX4A AR acknowledges funding from the Italian Ministry of University and Research (MIUR) through the “Dipartimenti di eccellenza” project “Science of the Universe” LLM Usage In preparing this paper, we used large language models as assistive tools The authors take full responsibility for the content of this paper.

Vera: Architecture and Training Details On four A100 GPUs, one complete training-and-validation epoch takes ∼ 52 minutes for ViViViT and ∼ 24 minutes for U-Net The tokenizer applies a 3D convolution with kernel (pt, ps + 2, ps + 2) and stride (pt, ps, ps), where pt denotes the temporal patch size and ph = pw ≡ ps the spatial patch size in voxel units The tokenizer maps the volume to Z(0) ∈ R B×Nt×Ns×d, where B is the batch size, Nt is the number of temporal patches, Ns is the number of spatial patches, and d is the embedding dimension Loss Function and Estimators The training objective is Lpix = 1/V X x∈V yˆ(x) − y(x) The directional term is Lgdl = 1/3V X d∈

t,h,w: X x∈V ∇dyˆ(x) − ∇dy(x) The L1 pixel term measures voxel-wise fidelity without quadratically upweighting locally large residuals The cross-correlation coefficient r(k) is defined as Pδδˆ(k) / sqrt(Pδˆδˆ(k)Pδδ(k)) The bispectrum Bb(k1, k2, k3) is constructed by binning Fourier modes into spherical shells and averaging over all triangle orientations We split each 384 × 1282 reconstructed volume into three contiguous 1283 regions Before transforming to Fourier space, we then multiply both reconstruction and target ground-truth fields by the same one-dimensional cosine-tapered window along the radial axis The taper fraction is set to α = 0.5, i.e unity over the central half of the region and half-cosine ramps over the outer quarters The power spectrum Pbδδ′(kb) is estimated using Eq.

Jocelyn: (8) where W2 factors cancel The cross-correlation coefficient r(k) is defined as Pδδˆ(k) / sqrt(Pδˆδˆ(k)Pδδ(k)) The bispectrum Bb(k1, k2, k3) is constructed by binning Fourier modes into spherical shells and averaging over all triangle orientations The estimator defined in Eq.

Title and authors: Vera: Let's talk about "ViViViT: a Video-to-Video Vision Transformer for Cosmological Lightcone Reconstruction" and what it means for our work today. Essentially, they are taking the observed lightcone data and treating it like a video of cosmic structure evolving over time.

Jocelyn: The title highlights that this is a vision transformer approach designed specifically to handle the temporal evolution inherent in how we see things along that line of sight.

Subrahmanyan: What’s important is that they are not just fitting some simple model; they are framing the problem as a video-to-video translation task, which implies they are looking for a direct mapping between what we observe and the true matter distribution.

Vera: They use neutral hydrogen simulations over redshifts from seven point sixty-six to sixteen point zero zero to generate their input data, which gives them a real testbed for how well this reconstruction works in realistic cosmological scenarios.

Jocelyn: The paper shows that this technique preserves the spectral power and phase alignment of the underlying matter field within a few percent of the ground truth, even up to the Nyquist scale.

Subrahmanyan: That preservation up to the Nyquist scale is significant because it means they are recovering fine details in the structure rather than just getting a smooth approximation, which is hard to achieve otherwise.

Vera: So for us on the ground, this suggests that we might be able to use these lightcones not just as simple probes but as structured sequences that can be analyzed in three dimensions, with spatial and temporal context built right in.

The paper's summary: Jocelyn: To summarize what they’ve accomplished with "ViViViT: a Video-to-Video Vision Transformer for Cosmological Lightcone Reconstruction," the authors are using that deterministic encoder decoder transformer to perform the translation task from lightcone to matter field.

Vera: The main technical move is factorizing attention into spatial and temporal components within that transformer, which helps them break the quadratic cost associated with processing every point in spacetime simultaneously.

Subrahmanyan: Because they factorize the attention this way, they can actually scale the model up to handle larger volumes of data, which is something really practical when you are dealing with huge cosmological simulations or large observational surveys.

Jocelyn: They separate transverse and radial attention, providing a natural framework to capture signatures that have different spatial morphology and different time dependencies across the entire lightcone sequence.

Vera: They compare this directly to a convolutional U-Net baseline, and their results show that the transformer performs accurate field-to-field translation across the whole lightcone, improving how well they reconstruct those measured two-point and three-point statistics relative to that tested convolutional model.

Subrahmanyan: This improved reconstruction of the two and three point statistics is what matters for us because it means we get a cleaner measurement of structure formation signals, which is what we need to test models of dark matter physics.

The paper's improvements: Vera: When we look at the specific improvements they suggest in "ViViViT: a Video-to-Video Vision Transformer for Cosmological Lightcone Reconstruction," it seems like they are really focusing on how much finer structure they can recover within that reconstruction.

Jocelyn: They state that ViViViT retains finer density structure than the tested U-Net when you look at a held out longitudinal slice of the data, which is a measurable improvement in detail we can actually see in the results.

Subrahmanyan: That retention of fine structure is significant because it means we might be able to measure smaller scale power in those reconstructed fields, and that small scale power is crucial for testing models of dark matter physics.

Vera: They also mention that ViViViT improves both the recovery of small scale power and the cross correlation with the target field when compared to that U-Net baseline.

Jocelyn: On the statistical side, they report that ViViViT reconstructs the ground truth bispectrum to within about five percent for most retained triangle configurations, and this holds true for sides up to k equal zero point seven seven h over M.

Conclusion: Vera: So wrapping things up on "ViViViT: a Video-to-Video Vision Transformer for Cosmological Lightcone Reconstruction," the main implication is that we have a new way to translate those observed twenty-one centimeter lightcones directly into the underlying matter field using this video translation technique <ref:2611.14617#pg1,ViViViT: a Video-to-Video Vision Transformer for Cosmological Lightcone Reconstruction>.

Jocelyn: It’s important to remember that this is a deterministic reconstruction, meaning it doesn't introduce uncertainty in the same way some other methods do, but they also have to be careful about what they can actually reconstruct.

Subrahmanyan: They did flag that because it is a deterministic reconstruction, it doesn't preserve posterior uncertainty, which is something we need to keep in mind when we use this for making cosmological inferences about the universe.

Vera: So the paper suggests that if we want to use these lightcones as a probe for cosmic acceleration or dark matter physics, ViViViT gives us a powerful tool for getting clean n-point measurements and cross correlations with other tracers.

Jocelyn: They also point out that they can exploit long range temporal structure when reconstruction requires it, saying ViViViT can reconstruct the target at the one point minus three level at every tau in a synthetic benchmark where models with bounded reach fail.

Subrahmanyan: It’s interesting that they also show how this framework extends naturally to multi-channel input by exploiting cross correlations between different tracers, which could potentially break foreground degeneracy and improve reconstruction accuracy.

Vera: That’s the big picture for us—using these lightcones not just as a simple probe, but as a structured sequence that can be analyzed in three dimensions. And that’s what this paper on ViViViT is all about.

More episodes

← Home