Self-attention summary networks for subsurface velocity-model building from common-image gathers

arXiv:2610.09282 · cs.LG, physics.geo-ph · Submitted 2026-10-07 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: Today's paper: "Self-attention summary networks for subsurface velocity-model building from common-image gathers".

Jane: The gist This work proposes a multiscale self-attention summary network that maps high-dimensional 3D CIG volumes into compact conditioning embeddings for probabilistic subsurface velocity inversion,

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So, to wrap up this discussion on "Self-attention summary networks for subsurface velocity-model building from common-image gathers", the authors have essentially built a system that uses multiscale self-attention to map high dimensional three dee CIG volumes into compact conditioning embeddings <ref:2610.09282#pg1,Self-attention summary networks for subsurface velocity-model building from common-image>.

Jane: And then they condition a flow matching model on those embeddings to learn the posterior distribution of plausible velocity fields, which they demonstrate is better than just conditioning on raw CIGs.

Lu: The key contribution seems to be that this learned embedding preserves offset dependent kinematic structure and spatial coherence while actively reducing variability caused by background velocity mismatch.

Meng: From a production standpoint, this means we’re looking at a more stable inference pipeline because the summary network acts as a powerful filter for the input data before it hits the flow model.

Lalam: And they showed that moving from raw CIG conditioning to this multiscale summary network conditioning progressively reduces the posterior mean root mean square error and increases SSIM, with their multiscale representation yielding the most concentrated uncertainty field.

Tom: The implication is that we can get better velocity samples and more reliable uncertainty estimates by using these learned embeddings instead of just using the raw data directly for conditioning.

Jane: It’s a framework that combines self-attention for feature extraction with conditional flow matching to make subsurface velocity inversion more robust under background model errors.

Conclusion: Tom: So we've looked at how they use these self-attention networks to squeeze those big three dee seismic CIG volumes into smaller conditioning embeddings for velocity inversion.

Jane: Right, and the main authors here are working on making subsurface velocity inference more robust when the background model is actually wrong.

Lu: The paper proposes combining a multiscale self-attention summary encoder with a conditional flow-matching model to generate those posterior velocity fields.

Meng: From my side, I'm curious how compact that latent representation actually is, because if it’s too compressed, we might lose the signal we need for real deployment.

Lalam: The core idea is that this summary network learns a distribution over plausible velocity models conditioned on those compact embeddings from the CIGs.

Tom: It seems they're mapping high-dimensional data to something low-dimensional, and then using that structure to guide the generation process via flow matching.

Jane: So, what does this actually mean for someone who just listens to the show about seismic imaging? It suggests a way to get more stable velocity samples even when we don't have a perfect starting model.

Lu: The results show that increasing the number of sampled offsets from four up to ten leads to a big drop in the singular value ratio, which is good because it means the information is more concentrated.

Meng: And they found that this summary network approach beats just conditioning on raw CIGs when you look at metrics like posterior mean root mean square error and SSIM.

Lalam: Specifically, using this multiscale summary representation gives the most concentrated uncertainty field, which is a big deal for knowing what we don't know about the subsurface.

Tom: So they’ve shown that combining those two techniques—the attention summary and the flow matching—improves both accuracy and how well we understand our errors.

Jane: It sounds like this is moving us closer to a system where inversion isn't so sensitive to initial guesses for the background model.

Lu: The authors acknowledge that future work involves testing generalization across different initial background-velocity models, which is definitely the next step for them.

Shiqin Zeng, Yunlin Zeng, Abhinav Prakash Gahlot, Zijun Deng, Felix J. Herrmann

Georgia Institute of Technology

cs.LG, physics.geo-ph

Submitted: 2026-10-07

Updated: 2026-10-07

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 83/100

The gist: The gist This work proposes a multiscale self-attention summary network that maps high-dimensional 3D CIG volumes into compact conditioning embeddings for probabilistic subsurface velocity inversion,

Key concepts

3D CIG Representation: The input data consists of 3D common-image gathers (CIGs), represented as a tensor where each spatial location has an offset-dependent response vector. This structure encodes local kinematic and structural information relative to the existing background model, providing rich input for velocity estimation.
Multiscale Self-Attention Summary Network: This network flattens the CIG grid into tokens and uses local windowed self-attention across multiple sizes. This allows spatial locations to aggregate information from neighbors with similar offset responses, effectively summarizing the high-dimensional data into a low-dimensional latent embedding while preserving spatially coherent structures.
Conditional Flow Matching: This technique models the conditional distribution of velocity fields by learning a path between a simple Gaussian noise distribution and the target velocity model. This allows for efficient generation of plausible subsurface velocity models conditioned on the learned compact latent representation from the summary network.

Summary

The gist

This work proposes a multiscale self-attention summary network that maps high-dimensional 3D CIG volumes into compact conditioning embeddings for probabilistic subsurface velocity inversion, improving posterior velocity inference by providing greater robustness to background-model mismatch.

Method Overview

The proposed framework combines a self-attention summary encoder for high-dimensional CIG representations with a conditional flow-matching model for posterior velocity generation. The goal is to learn a distribution over plausible velocity fields conditioned on the compact latent representation of the CIG.

3D CIG Representation

The input is defined as subsurface-offset common-image gathers (CIGs), which are discretized as a real-valued tensor y ∈ R D×H×W, where D is the number of sampled offsets and (H, W) are the spatial dimensions. This representation associates each spatial location with an offset-dependent response vector that encodes local kinematic and structural information relative to the background model.

Multiscale Self-Attention Summary Network

The summary encoder Φψ(·) maps the input CIGs to a low-dimensional latent embedding. This is achieved by flattening the spatial grid into tokens indexed by i ∈ 1,..., HW, where each index corresponds to a spatial location and is associated with an offset-response vector from the raw CIGs. Local windowed self-attention is applied at multiple window sizes to aggregate information from neighboring locations with similar offset-dependent responses. The attended feature at token i is then y˜i = P j∈W(i) αij vj, which allows each spatial location to aggregate information from nearby locations with similar offset-dependent responses, reinforcing spatially coherent reflector structure while suppressing locally inconsistent artifacts. Features are compressed through a learned channel-mixing projection to reduce redundancy while preserving physically relevant information.

Velocity-Model Generation via Conditional Flow Matching

The conditional distribution of subsurface velocity models is modeled using conditional flow matching by minimizing Eq. (3): L(ψ, ϕ) = Et, xt, dCIGseis h∥uϕ(xt, t, Φψ(y)) − ut(xt Φψ(y))∥2. The model uses the rectified-flow probability path defined by the linear interpolation xt = (1 - t)x0 + tx1, where x0 ∼ N (0, I) and x1 ∼ pm of subsurface velocity modeling. At inference, the velocity model is generated by integrating the learned conditional ODE in Eq. (4): x1 = x0 + Z 1 0 uϕ(xt, t, Φψ(y)) dt.

Experimental Results and Findings

The study evaluates the impact of latent compactness on performance, showing that increasing C from 4 to 10 leads to a monotonic decrease in the singular value ratio (σmin/σmax), from 7.2% to 0.22%, with downstream performance improving relative to C = 4 for C = 6, 8, and 10. The summary network demonstrates robustness under different initial background-velocity models; when the initial background is changed from a smoothed model to a 1D initial background, the attention features shift their emphasis away from near-zero-offset-dominated responses and redistribute it across other offset-related components. Furthermore, comparing conditioning strategies, moving from raw CIG conditioning to multiscale summary network conditioning progressively reduces the posterior mean RMSE and increases SSIM. The multiscale summary representation yields the most concentrated and best-structured uncertainty field, indicating that combining multiple attention scales improves both posterior accuracy and uncertainty control.

Conclusion

The proposed probabilistic framework effectively combines a multiscale self-attention summary network with conditional flow matching to improve robustness under background model errors and yield more accurate posterior samples and more concentrated uncertainty fields than direct raw CIG conditioning. Future work will test generalization across diverse initial background-velocity models by training on CIGs with different velocity errors and evaluating on unseen backgrounds.

Acknowledgments

This research was carried out with the support of the Georgia Research Alliance and the partners of the ML4Seismic Center.

References

Michael S. Albergo, Nicholas M. Boffi, and Eric Vanden-Eijnden. Stochastic interpolants: A unifying framework for flows and diffusions. Journal of Machine Learning Research, 26(209):1–80, 2025.

Biondo L Biondi. 3D seismic imaging. Society of Exploration Geophysicists, 2006.

Biondo L. Biondi and William W. Symes. Angle-domain common-image gathers for migration velocity analysis by wavefield-continuation imaging. Geophysics, 69(5):1283–1298, 2004 doi: 10.1190/1.1801945.

Li Jing, Jure Zbontar, and Yann LeCun. Implicit rank-minimizing autoencoder. In Advances in Neural Information Processing Systems, volume 33, 2020.

Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. In International Conference on Learning Representations, 2023.

Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. In International Conference on Learning Representations, 2023.

Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin Transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10012–10022, 2021.

Olivier Roy and Martin Vetterli. The effective rank: A measure of effective dimensionality. In 15th European Signal Processing Conference (EUSIPCO), 2007.

Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021.

William W. Symes. Migration velocity analysis and waveform inversion. Geophysical Prospecting, 56(6):765–790, 2008 doi: 10.1111/j.1365-2478.2008.00698.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30 pages 5998–6008.

Jean Virieux and Stéphane Operto. An overview of full-waveform inversion in exploration geophysics. Geophysics, 74(6):WCC1–WCC26, 2009.

Ziyi Yin, Rafael Orozco, Mathias Louboutin, and Felix J. Herrmann. WISE: Full-waveform variational inference via subsurface extensions. Geophysics, 89(4):A23–A28 doi: 10.1190/geo2023-0744.

Improvements for AI systems

Attention-based feature aggregation for CIGs will improve robustness to background model errors by learning a mismatch-aware redistribution of offset information, as noted in Section 3.2: the summary network does not directly recover the correct focusing, but instead learns a mismatch-aware redistribution of offset information.
Multiscale attention windows will enhance feature representation by capturing both fine and broad structural context, as described in Section 3.3: Smaller windows preserve sharper local responses, while larger windows provide broader structural context.
The proposed framework can achieve more accurate and concentrated uncertainty quantification for velocity inversion by replacing direct CIG conditioning with a multiscale summary network, as shown in Section 3.4: the multiscale summary representation yields the most concentrated and best-structured uncertainty field.
By utilizing conditional flow matching with the learned embeddings, the system can generate posterior velocity models that are more accurate than those from raw CIG conditioning, leading to more accurate posterior samples and more concentrated uncertainty fields.

Abstract

Common-image gathers (CIGs) contain physically meaningful information about velocity-model errors through reflector focusing and residual moveout, but in conventional imaging workflows they are typically used only as diagnostic tools. In this work, we propose a multiscale self-attention summary network that maps high-dimensional 3D CIG volumes into compact conditioning embeddings for probabilistic subsurface velocity inversion. These learned embeddings preserve offset-dependent kinematic structure and spatial coherence while reducing variability caused by background-velocity mismatch. Conditioned on these summary embeddings, a flow-matching model learns a transport from a Gaussian source distribution to the posterior distribution of plausible velocity fields. Numerical experiments show that, compared with direct conditioning on raw CIGs, the proposed summary network improves posterior velocity inference. In particular, the multiscale attention design provides greater robustness to background-model mismatch, yielding more accurate posterior reconstructions and lower predictive uncertainty.