ViViViT: a Video-to-Video Vision Transformer for Cosmological Lightcone Reconstruction
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Astrophysics. The day in astrophysics: observations, models and what the data forces.
Vera: I'm Vera.
Jocelyn: And I'm Jocelyn, here with Subrahmanyan.
Vera: We're looking at the paper: "ViViViT: a Video-to-Video Vision Transformer for Cosmological Lightcone Reconstruction".
Jocelyn: Here's the gist of the paper. Moving along the line of sight of a cosmological lightcone also moves through cosmic time, as in a video of cosmic structure traced by an evolving observable The reconstruction of the underlying matter field from observed lightcones is recast as a video-to-video translation task by introducing ViViViT On reconstructed matter fields from 21 cm neutral hydrogen simulations over z ∈
7.66, 16.00: , ViViViT preserves spectral power and phase alignment within a few percent of the ground truth up to the Nyquist scale The Problem Addressed The differential brightness temperature δT21 is not a direct image of the matter density δm because it is modulated through several factors including the neutral hydrogen fraction, gas and spin temperatures, radiation backgrounds, and line-of-sight velocities The matter distribution shapes this signal through both the gas density and the abundance and clustering of galaxies whose radiation heats and ionizes the surrounding intergalactic medium Reconstructing the matter distribution would give direct access to the matter field shared by all cosmological tracers The ViViViT Architecture ViViViT is a deterministic encoder–decoder transformer based on the ViViT architecture of
35: that factorizes attention into spatial and temporal components By factorizing attention into spatial and temporal components, our approach breaks the quadratic cost of full spacetime attention, allowing us to scale the model to larger volumes The encoder forms a global contextual representation of the input tokens The decoder uses self- and cross-attention, followed by a local convolutional upsampling head that restores the full grid Performance Metrics and Results ViViViT outperforms a convolutional U-Net baseline across all measured statistics ViViViT retains finer density structure than the tested U-Net on a held-out longitudinal slice ViViViT improves both small-scale power recovery and cross-correlation with the target field relative to the tested U-Net ViViViT reconstructs the ground-truth bispectrum to within about 5% for most retained triangle configurations, while the U-Net reconstruction error grows toward smaller scales and lower redshift Key Advantages and Limitations The geometry of the lightcone implies that the radial axis is simultaneously comoving distance and look-back time, allowing the sequence to be interpreted as a video of cosmic evolution ViViViT retains sharper features and better preserves the texture of the density field compared to a baseline tending to produce overly smooth fields The model can exploit long-range radial information when it is physically present, as demonstrated in a synthetic benchmark where it reconstructs the target at the 10−3 level at every τ A limitation is that the mapping is a deterministic reconstruction and does not preserve posterior uncertainty Future Directions In future work, our aim is to adapt this infrastructure to conditional sampling, accessing the conditional distribution rather than its median Applications to real datasets must include observational and astrophysical systematics and project the physical field to an instrumental measurement ViViViT extends naturally to multi-channel input, exploiting cross-correlations between tracers to potentially break the foreground degeneracy Beyond this reconstruction, a natural possible extension is to target redshift-dependent physical processes whose signature is distributed across the lightcone The authors state that ViViViT is the first video-to-video transformer applied to cosmological matterfield lightcone reconstruction Acknowledgments and Disclosure The authors thank Nicola Bellomo, Jessie de Kruijf, Sarah Libanore, Phoenix, Francesco Spezzati and Benjamin Wandelt for insightful feedback and discussions This work is partly supported by ICSC - Centro Nazionale di Ricerca in High Performance Computing, Big Data and Quantum Computing, funded by European Union - NextGenerationEU ML acknowledges support by the MIUR Progetti di Ricerca di Rilevante Interesse Nazionale (PRIN) Bando 2022 - grant 20228RMX4A AR acknowledges funding from the Italian Ministry of University and Research (MIUR) through the “Dipartimenti di eccellenza” project “Science of the Universe” LLM Usage In preparing this paper, we used large language models as assistive tools The authors take full responsibility for the content of this paper.
Vera: Architecture and Training Details On four A100 GPUs, one complete training-and-validation epoch takes ∼ 52 minutes for ViViViT and ∼ 24 minutes for U-Net The tokenizer applies a 3D convolution with kernel (pt, ps + 2, ps + 2) and stride (pt, ps, ps), where pt denotes the temporal patch size and ph = pw ≡ ps the spatial patch size in voxel units The tokenizer maps the volume to Z(0) ∈ R B×Nt×Ns×d, where B is the batch size, Nt is the number of temporal patches, Ns is the number of spatial patches, and d is the embedding dimension Loss Function and Estimators The training objective is Lpix = 1/V X x∈V yˆ(x) − y(x) The directional term is Lgdl = 1/3V X d∈
t,h,w: X x∈V ∇dyˆ(x) − ∇dy(x) The L1 pixel term measures voxel-wise fidelity without quadratically upweighting locally large residuals The cross-correlation coefficient r(k) is defined as Pδδˆ(k) / sqrt(Pδˆδˆ(k)Pδδ(k)) The bispectrum Bb(k1, k2, k3) is constructed by binning Fourier modes into spherical shells and averaging over all triangle orientations We split each 384 × 1282 reconstructed volume into three contiguous 1283 regions Before transforming to Fourier space, we then multiply both reconstruction and target ground-truth fields by the same one-dimensional cosine-tapered window along the radial axis The taper fraction is set to α = 0.5, i.e unity over the central half of the region and half-cosine ramps over the outer quarters The power spectrum Pbδδ′(kb) is estimated using Eq.
Jocelyn: (8) where W2 factors cancel The cross-correlation coefficient r(k) is defined as Pδδˆ(k) / sqrt(Pδˆδˆ(k)Pδδ(k)) The bispectrum Bb(k1, k2, k3) is constructed by binning Fourier modes into spherical shells and averaging over all triangle orientations The estimator defined in Eq.
Title and authors: Vera: Let's talk about "ViViViT: a Video-to-Video Vision Transformer for Cosmological Lightcone Reconstruction" and what it means for our work today. Essentially, they are taking the observed lightcone data and treating it like a video of cosmic structure evolving over time.
Jocelyn: The title highlights that this is a vision transformer approach designed specifically to handle the temporal evolution inherent in how we see things along that line of sight.
Subrahmanyan: What’s important is that they are not just fitting some simple model; they are framing the problem as a video-to-video translation task, which implies they are looking for a direct mapping between what we observe and the true matter distribution.
Vera: They use neutral hydrogen simulations over redshifts from seven point sixty-six to sixteen point zero zero to generate their input data, which gives them a real testbed for how well this reconstruction works in realistic cosmological scenarios.
Jocelyn: The paper shows that this technique preserves the spectral power and phase alignment of the underlying matter field within a few percent of the ground truth, even up to the Nyquist scale.
Subrahmanyan: That preservation up to the Nyquist scale is significant because it means they are recovering fine details in the structure rather than just getting a smooth approximation, which is hard to achieve otherwise.
Vera: So for us on the ground, this suggests that we might be able to use these lightcones not just as simple probes but as structured sequences that can be analyzed in three dimensions, with spatial and temporal context built right in.
The paper's summary: Jocelyn: To summarize what they’ve accomplished with "ViViViT: a Video-to-Video Vision Transformer for Cosmological Lightcone Reconstruction," the authors are using that deterministic encoder decoder transformer to perform the translation task from lightcone to matter field.
Vera: The main technical move is factorizing attention into spatial and temporal components within that transformer, which helps them break the quadratic cost associated with processing every point in spacetime simultaneously.
Subrahmanyan: Because they factorize the attention this way, they can actually scale the model up to handle larger volumes of data, which is something really practical when you are dealing with huge cosmological simulations or large observational surveys.
Jocelyn: They separate transverse and radial attention, providing a natural framework to capture signatures that have different spatial morphology and different time dependencies across the entire lightcone sequence.
Vera: They compare this directly to a convolutional U-Net baseline, and their results show that the transformer performs accurate field-to-field translation across the whole lightcone, improving how well they reconstruct those measured two-point and three-point statistics relative to that tested convolutional model.
Subrahmanyan: This improved reconstruction of the two and three point statistics is what matters for us because it means we get a cleaner measurement of structure formation signals, which is what we need to test models of dark matter physics.
The paper's improvements: Vera: When we look at the specific improvements they suggest in "ViViViT: a Video-to-Video Vision Transformer for Cosmological Lightcone Reconstruction," it seems like they are really focusing on how much finer structure they can recover within that reconstruction.
Jocelyn: They state that ViViViT retains finer density structure than the tested U-Net when you look at a held out longitudinal slice of the data, which is a measurable improvement in detail we can actually see in the results.
Subrahmanyan: That retention of fine structure is significant because it means we might be able to measure smaller scale power in those reconstructed fields, and that small scale power is crucial for testing models of dark matter physics.
Vera: They also mention that ViViViT improves both the recovery of small scale power and the cross correlation with the target field when compared to that U-Net baseline.
Jocelyn: On the statistical side, they report that ViViViT reconstructs the ground truth bispectrum to within about five percent for most retained triangle configurations, and this holds true for sides up to k equal zero point seven seven h over M.
Conclusion: Vera: So wrapping things up on "ViViViT: a Video-to-Video Vision Transformer for Cosmological Lightcone Reconstruction," the main implication is that we have a new way to translate those observed twenty-one centimeter lightcones directly into the underlying matter field using this video translation technique <ref:2611.14617#pg1,ViViViT: a Video-to-Video Vision Transformer for Cosmological Lightcone Reconstruction>.
Jocelyn: It’s important to remember that this is a deterministic reconstruction, meaning it doesn't introduce uncertainty in the same way some other methods do, but they also have to be careful about what they can actually reconstruct.
Subrahmanyan: They did flag that because it is a deterministic reconstruction, it doesn't preserve posterior uncertainty, which is something we need to keep in mind when we use this for making cosmological inferences about the universe.
Vera: So the paper suggests that if we want to use these lightcones as a probe for cosmic acceleration or dark matter physics, ViViViT gives us a powerful tool for getting clean n-point measurements and cross correlations with other tracers.
Jocelyn: They also point out that they can exploit long range temporal structure when reconstruction requires it, saying ViViViT can reconstruct the target at the one point minus three level at every tau in a synthetic benchmark where models with bounded reach fail.
Subrahmanyan: It’s interesting that they also show how this framework extends naturally to multi-channel input by exploiting cross correlations between different tracers, which could potentially break foreground degeneracy and improve reconstruction accuracy.
Vera: That’s the big picture for us—using these lightcones not just as a simple probe, but as a structured sequence that can be analyzed in three dimensions. And that’s what this paper on ViViViT is all about.
Federico Semenzato, Alvise Raccanelli, Michele Liguori
Dipartimento di Fisica e Astronomia “G. Galilei”, Universitaçy degli Studi di Padova · INFN, Sezione di Padova · Department of Physics and Astronomy, Johns Hopkins University · INAFOsservatorio Astronomico di Padova
astro-ph.IM, astro-ph.CO
Submitted: 2026-11-10
Updated: 2026-11-10
Comments: 15 pages, 6 figures, Accepted to the Sim2Science Workshop (NeurIPS 2026)
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 86/100
The gist: Moving along the line of sight of a cosmological lightcone also moves through cosmic time, as in a video of cosmic structure traced by an evolving observable The reconstruction of the underlying…
Key concepts
- Lightcone Reconstruction
- This is the core task where AI maps observed radio signals (like 21 cm neutral hydrogen data) along a line of sight into a direct map of the underlying cosmic matter density field. The lightcone geometry links distance and time, allowing the reconstruction to be viewed as a video of cosmic structure evolving over time.
- ViViViT Architecture
- This is a deterministic encoder-decoder transformer designed specifically for this task. It uses factorized attention—splitting attention into spatial (where) and temporal (when)—to efficiently process the 3D spacetime data, which prevents the computational cost from becoming too large for complex cosmic volumes.
- Differential Brightness Temperature ($\delta T_{21}$)
- This is the observable signal measured by radio telescopes. It is not a direct measure of matter density because it is heavily influenced by many factors like gas temperature, neutral hydrogen fraction, and line-of-sight velocities. The goal of the model is to reverse this modulation to find the true matter distribution.
- Video-to-Video Translation
- This frames the problem as a video processing task. The input is a video of cosmic structure traced by an evolving observable (the lightcone), and the output is a reconstructed 'video' of the underlying matter field. This approach leverages transformer models to learn how to translate between these two representations.
Summary
Moving along the line of sight of a cosmological lightcone also moves through cosmic time, as in a video of cosmic structure traced by an evolving observable The reconstruction of the underlying matter field from observed lightcones is recast as a video-to-video translation task by introducing ViViViT On reconstructed matter fields from 21 cm neutral hydrogen simulations over z ∈ [7. p. 0 it
Abstract
Moving along the line of sight of a cosmological lightcone also moves through
cosmic time, as in a video of cosmic structure traced by an evolving observable. We
recast the reconstruction of the underlying matter field from observed lightcones as
a video-to-video translation task by introducing ViViViT, a vision transformer that
factorizes attention across transverse frames and radial evolution.Where the paper says it
The Problem Addressed
The differential brightness temperature δT21 is not a direct image of the matter density δm because it is modulated through several factors including the neutral hydrogen fraction, gas and spin temperatures, radiation backgrounds, and line-of-sight velocities The matter distribution shapes this signal through both the gas density and the abundance and clustering of galaxies whose radiation heats and ionizes the surrounding intergalactic medium Reconstructing the matter distribution would give direct access to the matter field shared by all cosmological tracers
The ViViViT Architecture
ViViViT is a deterministic encoder–decoder transformer based on the ViViT architecture of [35] that factorizes attention into spatial and temporal components By factorizing attention into spatial and temporal components, our approach breaks the quadratic cost of full spacetime attention, allowing us to scale the model to larger volumes The encoder forms a global contextual representation of the input tokens The decoder uses self- and cross-attention, followed by a local convolutional upsampling head that restores the full grid
Performance Metrics and Results
ViViViT outperforms a convolutional U-Net baseline across all measured statistics ViViViT retains finer density structure than the tested U-Net on a held-out longitudinal slice ViViViT improves both small-scale power recovery and cross-correlation with the target field relative to the tested U-Net ViViViT reconstructs the ground-truth bispectrum to within about 5% for most retained triangle configurations, while the U-Net reconstruction error grows toward smaller scales and lower redshift
Key Advantages and Limitations
The geometry of the lightcone implies that the radial axis is simultaneously comoving distance and look-back time, allowing the sequence to be interpreted as a video of cosmic evolution ViViViT retains sharper features and better preserves the texture of the density field compared to a baseline tending to produce overly smooth fields The model can exploit long-range radial information when it is physically present, as demonstrated in a synthetic benchmark where it reconstructs the target at the 10−3 level at every τ A limitation is that the mapping is a deterministic reconstruction and does not preserve posterior uncertainty
Future Directions
In future work, our aim is to adapt this infrastructure to conditional sampling, accessing the conditional distribution rather than its median Applications to real datasets must include observational and astrophysical systematics and project the physical field to an instrumental measurement ViViViT extends naturally to multi-channel input, exploiting cross-correlations between tracers to potentially break the foreground degeneracy Beyond this reconstruction, a natural possible extension is to target redshift-dependent physical processes whose signature is distributed across the lightcone The authors state that ViViViT is the first video-to-video transformer applied to cosmological matterfield lightcone reconstruction
Acknowledgments and Disclosure
The authors thank Nicola Bellomo, Jessie de Kruijf, Sarah Libanore, Phoenix, Francesco Spezzati and Benjamin Wandelt for insightful feedback and discussions This work is partly supported by ICSC - Centro Nazionale di Ricerca in High Performance Computing, Big Data and Quantum Computing, funded by European Union - NextGenerationEU ML acknowledges support by the MIUR Progetti di Ricerca di Rilevante Interesse Nazionale (PRIN) Bando 2022 - grant 20228RMX4A AR acknowledges funding from the Italian Ministry of University and Research (MIUR) through the “Dipartimenti di eccellenza” project “Science of the Universe” LLM Usage In preparing this paper, we used large language models as assistive tools The authors take full responsibility for the content of this paper.
Architecture and Training Details
On four A100 GPUs, one complete training-and-validation epoch takes ∼ 52 minutes for ViViViT and ∼ 24 minutes for U-Net The tokenizer applies a 3D convolution with kernel (pt, ps + 2, ps + 2) and stride (pt, ps, ps), where pt denotes the temporal patch size and ph = pw ≡ ps the spatial patch size in voxel units The tokenizer maps the volume to Z(0) ∈ R B×Nt×Ns×d, where B is the batch size, Nt is the number of temporal patches, Ns is the number of spatial patches, and d is the embedding dimension
Loss Function and Estimators
The training objective is Lpix = 1/V X x∈V yˆ(x) − y(x) The directional term is Lgdl = 1/3V X d∈[t,h,w] X x∈V ∇dyˆ(x) − ∇dy(x) The L1 pixel term measures voxel-wise fidelity without quadratically upweighting locally large residuals The cross-correlation coefficient r(k) is defined as Pδδˆ(k) / sqrt(Pδˆδˆ(k)Pδδ(k)) The bispectrum Bb(k1, k2, k3) is constructed by binning Fourier modes into spherical shells and averaging over all triangle orientations
Summary-statistic Estimators
We split each 384 × 1282 reconstructed volume into three contiguous 1283 regions Before transforming to Fourier space, we then multiply both reconstruction and target ground-truth fields by the same one-dimensional cosine-tapered window along the radial axis The taper fraction is set to α = 0.5, i.e unity over the central half of the region and half-cosine ramps over the outer quarters The power spectrum Pbδδ′(kb) is estimated using Eq. p. 8 Before transforming to Fourier space, we then multiply both reconstruction and target ground-truth
fields by the same one-dimensional cosine-tapered window along the radial axis, where t represents
the radial frame index within a given redshift bin, with N = 128 frames per bin,
w(t) =
1
2
h
1 + cos
π
2t
α(N−1) − 1
i , t < α
2
(N − 1),
1,
α
2
(N − 1) ≤ t ≤
1 −
α
2
(N − 1),
w(N − 1 − t), otherwise,
(6)
with taper fraction α = 0. unity over the central half of the region and half-cosine ramps over
the outer quarters. (8) to the reconstruction ˆδ and the target δ, we define the cross-correlation coefficient
r(k) = Pδδˆ (k)
p
Pδˆδˆ(k)Pδδ(k)
, (9)
in which the W2 factors cancel. Bispectrum.Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Video-to-video translation for cosmological field reconstruction allows AI systems to map observed 21 cm lightcones to underlying matter fields, preserving spectral power and phase alignment within a few percent of ground truth up to the Nyquist scale.
-
Factorized attention breaks down the quadratic cost of full spacetime attention into spatial and temporal components, allowing the model to scale to larger volumes while capturing signatures with different morphology and time dependence.
-
The system can perform clean n-point statistical measurements, cross-correlations with other tracers, and a powerful probe to test the nature of cosmic acceleration, dark-matter physics, and departures from general relativity by inverting the mapping between observed lightcones and matter density fields.
-
The model can exploit long-range temporal structure when reconstruction requires it; specifically, ViViViT can reconstruct the target at the 10 minus 3 level at every tau in a synthetic benchmark where bounded-reach models fail.
-
The framework is naturally extended to multi-channel input by exploiting cross-correlations between tracers, which can potentially break foreground degeneracy and improve reconstruction accuracy.
Abstract
Moving along the line of sight of a cosmological lightcone also moves through cosmic time, as in a video of cosmic structure traced by an evolving observable. We recast the reconstruction of the underlying matter field from observed lightcones as a video-to-video translation task by introducing ViViViT, a vision transformer that factorizes attention across transverse frames and radial evolution. On reconstructed matter fields from 21cm neutral hydrogen simulations over z in[7.66,16.00], ViViViT preserves spectral power and phase alignment within a few percent of the ground truth up to the Nyquist scale, and three-point correlations for all triangle configurations with sides up to k 0.77,h/Mpc, outperforming a convolutional 3D-U-Net baseline across all measured statistics. By encoding the evolving geometry of the cosmological observable itself, ViViViT marks a step toward field-level architectures that natively respect cosmic time, opening the path to time-aware reconstruction of cosmic tracers and targeted searches for redshift-dependent signatures in 21cm data.
Sources
Related papers
- On-sky demonstration of a vector Zernike wavefront sensor in a cascaded adaptive optics system
- Euclid Data Release 1 (DR1): Calibration strategy and observations
- The General Antiparticle Spectrometer (GAPS) Antarctic Balloon Payload
- Detection and excision of radio-frequency interference in radio arrays
- Unified matched-filter and excess-power pipeline for gravitational-wave detector characterization
- Ushering in sub-arcsecond angular resolution observatories in the far-infrared: The case for a double-Fourier interferometer