Decision-Driven Geosteering Under Uncertainty: A Unified Framework for Sequential Decision Optimization

arXiv:2606.17331 · cs.LG · Submitted 2026-06-15 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Decision-Driven Geosteering Under Uncertainty".

Jane: The gist: This work presents an uncertaintyaware geosteering framework that tightly integrates particle filtering for probabilistic subsurface interpretation with value-based reinforcement learning for sequential decision-making,

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So we're looking at this paper today, "Decision-Driven Geosteering Under Uncertainty: A Unified Framework for Sequential Decision Optimization," and it seems like the authors are tackling a really hard problem in oil and gas drilling. It's all about making decisions while you don't actually know what’s underground.

Jane: Exactly, Tom. Think about drilling a well; you can't see the rock layers directly, so every decision you make—like where to steer the bit—has to be based on an educated guess or a probability of what’s happening down there. This paper says they are putting together a way to handle that uncertainty very carefully.

Lu: It’s interesting because they aren't just using one method for everything; they are coupling particle filtering, which is great for figuring out the uncertainty, with value-based reinforcement learning, which is how you make the actual choices. That combination seems like it bridges a big gap between pure science and practical decision-making.

Meng: From an engineering standpoint, I'm curious about how this translates into something we can actually run on a rig. If the system relies on particle filtering for subsurface interpretation, how much computational power are we talking about for real-time adjustments?

Lalam: I see the concept of using a particle filter to represent geological uncertainty as a weighted ensemble of possible subsurface states. That means instead of one guess, you have many possibilities that get more or less weight based on the measurements you get from the drilling tools.

Tom: Right, so they use this particle filter to condition their decision-making. The paper explains that this gives them belief-informed control instead of just trying to correct a path deterministically. That’s a big difference in how you approach the problem.

Jane: That’s right, Tom. Instead of saying "the path is wrong, so adjust by five degrees," they are making decisions based on the probability distribution they have about the subsurface state. It’s about acting on what you believe to be true at that moment.

Lu: They define a compact decision state using a function that takes those particle states, their weights, and even the current inclination of the well, plus some geometry descriptors related to where you are relative to boundaries. That way, the AI knows exactly what information it needs right now.

Title and authors: Meng: So they’re feeding this complex belief state into a value-based reinforcement learning structure to estimate what action will give them the best outcome moving forward. That sounds like a lot of variables to manage simultaneously in real time.

Lalam: The framework also incorporates an optimism mechanism called an Approximate Dynamic Programming approach, which adds a term opt(x) = (x) + beta sigma(x), which reflects what the paper calls classical decision-analytic principles of value of information. That’s essentially telling the AI to consider the value of knowing more information.

Tom: Okay, so they are using three different decision-making options that all operate under this same uncertainty representation, and they test them against each other in a simulator to see which one performs best during drilling. That controlled testing is key for proving their method works.

Jane: And the simulation setup is pretty detailed; they use identical geological realizations, operational limits, and reward definitions across all the methods being tested so you can actually compare the policies fairly rather than comparing two different setups.

Lu: They focus on evaluating how alternative decision policies behave throughout the entire drilling process, not just looking at how straight or accurate the final well trajectory ends up being. That shifts the focus to a more dynamic performance metric.

Meng: I'm paying attention to their performance metrics here because they mention things like JerkRMS, which is used to quantify control smoothness by calculating the root mean square of jerk over time, specifically in true vertical depth. That sounds like they care about how smooth the drilling feels operationally.

Lalam: And they show that this Dual Deep Reinforcement Learning architecture produces smoother and more stable decisions compared to standard reinforcement learning or approximate dynamic programming, which is shown by a substantially lower JerkRMS value.

Tom: So the implication here is that this new framework gives us a more reliable way to steer wells because it seems to produce smoother control actions under uncertainty, especially when the uncertainty ahead of the bit increases.

Jane: That makes sense. It suggests that the quality of decisions isn't just about hitting a final target; it’s about how consistently and smoothly you manage your path while things are still uncertain. This is a more realistic view for anyone running operations on site.

Title and authors: Lu: The paper notes that visually similar well trajectories can come from fundamentally different decision policies, which really highlights the limitation of judging geosteering performance just by looking at the final geometry of the well path.

Meng: That’s a crucial point for practical application. It means two wells could look identical on a map but have been drilled using very different steering strategies, and this paper shows how to tell them apart based on the underlying decision process.

Lalam: For me, it suggests that the way an AI interprets and acts on those evolving particle filter belief states is what truly distinguishes good control, not just the final shape of the trajectory. It’s about that belief state itself.

Tom: So to wrap up this part of our discussion, this paper presents "Decision-Driven Geosteering Under Uncertainty: A Unified Framework for Sequential Decision Optimization," which successfully integrates particle filtering with value-based reinforcement learning to create a belief-informed control system for drilling. It shows that the Dual Deep Reinforcement Learning architecture yields smoother and more stable decisions, particularly when predictive uncertainty is high ahead of the bit.

Jane: It’s a solid piece of work because it moves beyond deterministic correction and lets the system react intelligently to evolving subsurface probabilities. We should keep an eye on how this framework can be adapted for different types of subsurface data.

Lu: I think the way they unified uncertainty quantification with sequential decision optimization is really interesting, and it sets a new structure for how we think about these complex, sequential problems in engineering applications.

Meng: From what I've seen so far, the practical impact seems to be in building more robust and less jittery control systems for autonomous drilling operations. It’s about reducing that operational noise caused by overly aggressive or unstable adjustments.

Lalam: It points toward a future where decision-making systems are fundamentally tied to probabilistic interpretations of the environment, which could improve how we manage risk in complex industrial settings.

Tom: And that's what we're talking about today. We’ve covered the core idea, how they tackle uncertainty with particle filters and reinforcement learning, and what the results show regarding stability and smoothness. Next up, let’s see what other papers are doing to tackle similar problems in AI applications.

The paper's summary: Tom: So, to wrap up this part, they’ve built this whole system that uses particle filtering to guess what’s underground and then uses reinforcement learning to make steering decisions based on those guesses.

Jane: Exactly. It’s not just one method doing everything; it’s a unified framework where the uncertainty from the geology feeds directly into how the AI makes its choices in real time.

Lu: They take this idea of a probability distribution over subsurface states and use it to condition their decision-making process, which they call belief-informed control instead of just correcting a straight line deterministically.

Meng: What I’m hearing is that the AI isn't just reacting to current measurements; it's reacting to its entire picture of what might be down there, which sounds much more robust for actual drilling operations.

Lalam: And they use this structure to build a compact decision state that includes the particle states and their weights, along with the current angle and boundary information, feeding all of that into a value-based reinforcement learning model.

Tom: It’s basically taking all that complex probability data and turning it into a manageable input for an AI to figure out the best next move while respecting physical limits like dogleg severity constraints.

Jane: They also added this optimism mechanism, which is like telling the AI to account for the value of knowing more information, which is a classic idea from decision analysis that they integrated right in there.

Lu: The paper shows that this Dual Deep Reinforcement Learning architecture tends to produce smoother and more stable steering decisions compared to other approaches like standard reinforcement learning or approximate dynamic programming.

Meng: That smoothness is important because it translates directly into less operational noise, meaning fewer sudden, jerky movements on the drill string during steering adjustments.

Lalam: They measure this smoothness using a performance metric called JerkRMS, which is the root mean square of jerk in true vertical depth over time, and they show that this method keeps that value substantially lower.

Tom: So what it means for us on the ground is that when uncertainty gets bigger ahead of the bit, you don't get those wild oscillations or abrupt corrections; you just keep steering smoothly toward your target while still being smart about the risk.

Jane: It’s about how the AI interprets that changing belief state and acts on it intelligently, rather than just following a fixed path correction rule.

Lu: The paper points out that you can get visually similar well trajectories from fundamentally different decision policies, which really shows you shouldn't judge geosteering performance by the final geometry alone.

Meng: That’s a big deal for verification; it means we need to look at the underlying control logic, not just the final map of the wellbore.

Lalam: The core vision here is that this framework gives us a way to handle sequential decisions under uncertainty by tightly linking probabilistic subsurface interpretation with value-based learning for real-time control.

Tom: It’s about building a system that is both informed by what we know and capable of making robust, stable steering choices when things get messy underground.

The paper's improvements: Tom: So, the paper isn't just about setting up this system; they are also showing how to actually improve it and make it better than what we have now with other methods.

Jane: That’s right, Tom. They introduce things like a unified uncertainty quantification and decision framework that combines a data-driven geological generator based on a kernel density estimator, a particle filter for uncertainty, and multiple robust sequential decision optimization options.

Lu: The paper suggests this approach allows you to generate predictions about subsurface layers while also using the particle filter to quantify that uncertainty probabilistically.

Meng: So it means you get not just one forecast, but a range of possibilities for what’s underground, which is better than relying on a single deterministic model.

Tom: And they talk about their Dual Deep Reinforcement Learning architecture specifically, noting that it reduces variance across seeds during training, which means the policy becomes more robust to random starting conditions and exploration noise.

Jane: That robustness is key because in real drilling situations, you don't have a perfect starting point or perfectly consistent exploration; you need something stable.

Lu: They also introduce a new performance metric focused on trajectory smoothness by quantifying the root mean square jerk in true vertical depth to measure control quality more effectively.

Meng: That specific measurement helps engineers see exactly how smooth the steering actions are, which is a much more practical way to assess if the AI is making sensible physical adjustments.

Tom: The implication here is that this framework can generate trajectories that are not just accurate in terms of final placement, but also incredibly stable during the whole process.

Jane: It moves us toward systems where the control isn't just about hitting a target; it’s about maintaining a sensible path while constantly adapting to new information and growing uncertainty ahead of the bit.

Lu: The authors explicitly state that their method can maintain a trajectory closely aligned with the interpreted structure while avoiding excessive oscillations or abrupt corrections when uncertainty unfolds ahead of the bit.

Meng: That sounds like exactly what we want in an autonomous system; it means it can be responsive without being overly aggressive or unstable.

Tom: So, they’re showing that this approach doesn't just solve the initial problem of making a decision under uncertainty; it actually provides a pathway to building a more reliable and coherent overall control policy.

Conclusion: Tom: So we’re wrapping up on "Decision-Driven Geosteering Under Uncertainty: A Unified Framework for Sequential Decision Optimization." Basically, this paper shows how to couple particle filtering with reinforcement learning to build a belief-informed control system for drilling.

Jane: It really highlights that you don't need one perfect model; you can use a probabilistic ensemble of possibilities to drive smarter, smoother steering decisions in real time.

Lu: The authors found that this Dual Deep Reinforcement Learning architecture produces more stable and coherent decisions compared to standard methods, especially when the uncertainty ahead of the bit is increasing.

Meng: For practical work, this means we can expect less jitter on the drill string because the policy stays closer to the target while still being cautious about what’s happening downhole.

Lalam: This kind of work pushes our understanding of sequential decision-making under uncertainty, suggesting that how an AI interprets and acts on evolving probability distributions is a major factor in improving industrial control systems.

Tom: It really shifts the focus away from just looking at the final well trajectory geometry as the only measure of success.

Jane: That’s right, Tom. The real performance is in how consistently and smoothly the system manages its path while things are still probabilistic down there.

Lu: This framework sets a new structure for how we think about combining uncertainty quantification with sequential optimization problems in engineering applications, which is pretty creative.

Meng: I wonder if this approach could be applied to other complex sequential systems where you need to make decisions based on an evolving understanding of the environment.

Lalam: It suggests that future AI development should focus heavily on these types of belief-informed architectures so they can handle real-world complexity better than purely deterministic approaches.

Tom: So, this paper gives us a solid foundation for making drilling operations more intelligent and less reliant on rigid, pre-programmed correction schemes.

Jane: It’s a great piece of research that shows how to make AI not just smart in isolation, but smart in the context of real-time probabilistic uncertainty.

Lu: We’re looking at how this method integrates multiple decision-making options under one unified uncertainty representation, which is a pretty powerful concept.

Meng: It seems like the next step for engineers is figuring out exactly how to deploy these complex belief states on actual hardware efficiently without losing that real-time capability.

Lalam: The vision here is an AI capable of not just predicting outcomes, but actively steering through uncertainty with a level of stability that matches the complexity of the subsurface.

Tom: That’s what we’ve been talking about today with "Decision-Driven Geosteering Under Uncertainty: A Unified Framework for Sequential Decision Optimization."

Jane: It's a significant step toward more reliable, belief-informed control in complex environments.

NORCE Research Centre · University of Stavanger · Stanford University

cs.LG

Submitted: 2026-06-15

Updated: 2026-06-15

Journal ref: Journal of Computational Science 101 (2026) 103042

DOI: 10.1016/j.jocs.2026.103042

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 86/100

The gist: The gist: This work presents an uncertaintyaware geosteering framework that tightly integrates particle filtering for probabilistic subsurface interpretation with value-based reinforcement learning

Key concepts

Partially Observable Markov Decision Process (POMDP): Geosteering is modeled as a POMDP because the actual subsurface state cannot be directly seen while drilling. Decisions must therefore be based on a 'belief state,' which is a probability distribution over all possible subsurface states, accounting for inherent uncertainty.
Particle Filter (PF): The PF represents geological uncertainty using a weighted ensemble of particles. This filter uses incoming measurements, like Gamma Ray data, to update the weights of these particles. This process refines the probability distribution over reservoir boundaries in real-time.
Value-Based Reinforcement Learning: This component uses a Q-function to learn the optimal sequence of actions based on the current belief state and context. It estimates the expected cumulative reward, guiding the agent to select steering actions that maximize performance while considering future uncertainties.
Belief State ($x_t$): The decision state is a compact representation combining particle information, particle weights, and local trajectory context. This belief state captures the current understanding of subsurface geometry and measurement likelihoods, serving as the crucial input for the learning-based control policy.

Terminology

Reinforcement Learning — 2× in this paper · explained in 4 episodes
Deep Reinforcement Learning — 1× in this paper · explained in 3 episodes
Fidelity — 1× in this paper · explained in 2 episodes
Q-learning — 1× in this paper · explained in 3 episodes
Steering — 1× in this paper · explained in 2 episodes
Unified Framework — 1× in this paper · explained in 5 episodes

Summary

The gist: This work presents an uncertaintyaware geosteering framework that tightly integrates particle filtering for probabilistic subsurface interpretation with value-based reinforcement learning for sequential decision-making, enabling belief-informed control rather than deterministic trajectory correction.

Problem Formulation

Geosteering is modeled as a Partially Observable Markov Decision Process (POMDP) where the subsurface state cannot be directly observed while drilling, requiring decisions to be based on a belief state which is a probability distribution over possible subsurface states<ref:2606.17331#pg8>. The POMDP framework is defined by the tuple M = ⟨S, A, O, P, Z, R, γ⟩<ref:2606.17331#pg9>. The subsurface state st evolves according to a Markovian stochastic process st ∼ P(st st−1, at−1), where the transition kernel captures geological variability and modeling uncertainty<ref:2606.17331#pg10>. Observations ot are probabilistically related to the true subsurface state through the observation model ot ∼ Z(ot st), which is subject to measurement error and limited sensing range<ref:2606.17331#pg9>.

Particle Filter for Boundary Inference

Geological uncertainty is represented by a weighted particle ensemble, while the PF conditions this ensemble on incoming measurements (GR measurements) to refine the uncertainty over reservoir boundaries<ref:2606.17331#pg10>. Each particle state at drilling step t is defined as ξ i t = offset i t, angle i t, which captures the dominant geometric uncertainty relevant for real-time geosteering decisions<ref:2606.17331#pg11>. Particle states evolve according to a stochastic transition model, ξ i t = f(ξ i t−1) + ε i t, where ε i t captures geological variability between successive drilling steps<ref:2606.17331#pg11>. Upon receiving a new GR measurement ot, particle weights are updated according to w i t ∝ w i t−1 p(ot ξ i t), where p(ot ξ i t) evaluates the likelihood of the observed GR measurement given the boundary configuration represented by particle i<ref:2606.17331#pg11>.

Decision State Representation and Actions

A compact decision state xt is constructed as xt = Φ(ξ i t, wi t, ot, at−1), which includes a local trajectory-context term given by the current inclination, posterior weights of the five highest-probability particle hypotheses, and boundary-relative geometry descriptors<ref:2606.17331#pg13>. Steering actions are discretized according to drilling phases; during the landing phase, the agent selects inclination adjustment commands ∆θ from a discrete set spanning the range −10◦ to +10◦ with increments of 0.5◦<ref:2606.17331#pg14>. These actions are constrained by physical and operational limitations, specifically the dogleg severity (DLS) between successive actions must satisfy DLS(at, at−1) ≤ DLSmax<ref:2606.17331#pg9>.

Learning-Based Value Estimation

Decision-making is guided by a state–action value function Q(xt, at), which estimates the expected cumulative discounted reward following action at in state xt<ref:2606.17331#pg16>. Learning is performed using Temporal-Difference updates based on observed transitions (xt, at, rt, xt+1), employing experience replay to improve sample efficiency and stabilize training<ref:2606.17331#pg16>. The framework also incorporates an optimism mechanism in the Approximate Dynamic Programming (ADP) approach where the value estimate is augmented as Vˆ opt(x) = Vˆ (x) + β σ(x), reflecting classical decision-analytic principles of value of information<ref:2606.17331#pg20>.

Performance and Stability Metrics

The proposed framework is evaluated using metrics that assess both geological placement accuracy and the stability of sequential steering decisions, such as the root mean square (RMS) jerk<ref:2606.17331#pg28>. Overall control smoothness is summarized using JerkRMS = vuut 1/T − 3 X T t=4 j squared t<ref:2606.17331#pg29>. The comparison of policies reveals that the Dual DRL architecture consistently produces smoother and more stable decisions than ADP or standard DRL, as evidenced by a substantially lower RMS jerk value<ref:2606.17331#pg23>. This stability is particularly evident in regions where forward uncertainty increases, indicating that the policy effectively balances responsiveness with robustness<ref:2606.17331#pg30>.

Conclusion

The results demonstrate that visually similar well trajectories can arise from fundamentally different decision policies, highlighting the limitations of evaluating geosteering performance based on final placement alone<ref:2606.17331#pg25>. The proposed Dual DRL architecture consistently produces more stable and coherent decisions, particularly in regions of increasing predictive uncertainty<ref:2606.17331#pg30>. This illustrates that differences in decision quality are primarily expressed through how policies interpret and act on evolving PF-based belief states, rather than through final trajectory geometry alone<ref:2606.17331#pg25>. Future work will extend this framework to more complex geological settings and additional operational constraints<ref:2606.17331#pg25>.

Acknowledgments

H.E. Djecta, S. Alyaev, K. Fossum, and R.B. Bratvold acknowledge the support from the project DISTINGUISH (Decision support using neural networks to predict geological uncertainties when geosteering), funded by Aker BP, Equinor, and the Research Council of Norway (RCN PETROMAKS2 project no. 344236<ref:2606.17331#pg26>. R.B. Muhammad acknowledges the support from the Center for Researchbased Innovation DigiWells: Digital Well Center for Value Creation, Competitiveness and Minimum Environmental Footprint (NFR SFI project no. 309589), funded by Aker BP, ConocoPhillips, Equinor, Harbour Energy, Petrobras, TotalEnergies, Vår Energi, and the Research Council of Norway<ref:2606.17331#pg26>. The authors thank ROGII Inc. for providing the academic licenses for Solo Cloud and StarSteer and the relevant training<ref:2606.17331#pg26>.

References

[1] K. Kullawan, R. Bratvold, J. Bickel, A decision analytic approach to geosteering operations, SPE Drilling & Completion 29 (03 2014). doi:10.2118/167433-PA.<ref:2606.17331#pg28>.

[2] K. Kullawan, R. Bratvold, J. Bickel, Sequential geosteering decisions for optimization of real-time well placement, Journal of Petroleum Science and Engineering 165 (2018) 90–104.<ref:2606.17331#pg28>.

[3] S. Alyaev, E. Suter, R. B. Bratvold, A. Hong, X. Luo, K. Fossum, A decision support system for multi-target geosteering, Journal of Petroleum Science and Engineering 183 (2019) 106381.<ref:2606.17331#pg28>.

[4] G. Evensen, The ensemble kalman filter: theoretical formulation and practical implementation, Ocean Dynamics 53 (4) (2003) 343–367. doi:10.1007/s10236-003-0036-9.<ref:2606.17331#pg28>.

[5] S. Alyaev, K. Fossum, H. Djecta, J. Tveranger, A. Elsheikh, Distinguish workflow: a new paradigm of dynamic well placement using generative machine learning, in: ECMOR 2024, Vol. 2024, European Association of Geoscientists & Engineers, 2024, pp. 1–16.<ref:2606.17331#pg6>.

[6] R. S. Sutton, A. G. Barto, Reinforcement Learning: An Introduction, 2nd Edition, The MIT Press, 2018.<ref:2606.17331#pg5>.

Improvements for AI systems

The improved system can perform belief-informed control rather than deterministic trajectory correction, as stated in the abstract, by tightly integrating particle filtering for probabilistic subsurface interpretation with value-based reinforcement learning for sequential decision-making.
The system can utilize a unified uncertainty-quantification and decision framework that "combines a datadriven geological generator based on a kernel-density estimator (KDE), a particle filter for uncertainty quantification and probabilistic stratigraphic forecasting and multiple robust sequential decision optimization options."
The system can achieve enhanced policy stability by employing the proposed Dual Deep Reinforcement Learning (Dual DRL) architecture, which is designed to reduce variance across seeds throughout training, indicating improved robustness to stochastic initialization and exploration noise.
The system can generate smoother trajectories by incorporating a stability-oriented performance metric, specifically quantifying the rate of change of curvature in the vertical profile using the discrete jerk of the trajectory in true vertical depth (TVD) to minimize JerkRMS.
The system can exhibit superior decision-making behavior under uncertainty compared to other methods, as shown by its ability to maintain a trajectory that is closely aligned with the interpreted structure while avoiding excessive oscillations or abrupt corrections when uncertainty unfolds ahead of the bit.

Abstract

Geosteering requires navigating a well trajectory through an unknown geological configuration, while sequentially updating decisions based on indirect measurements acquired during drilling. This work presents an uncertainty-aware geosteering framework that tightly integrates particle filtering for probabilistic subsurface interpretation with value-based reinforcement learning for sequential decision-making. Geological uncertainty ahead of the drill bit is represented explicitly through a particle filter (PF), enabling belief-informed control rather than deterministic trajectory correction. The framework couples PF belief updates with belief-informed decision policies and evaluates three decision-making options that operate under identical uncertainty representations: an interpretable Approximate Dynamic Programming (ADP) scheme, a Deep Q-learning baseline, and a Dual Deep Reinforcement Learning (Dual DRL) architecture trained with a target Q-network scheme for stability, using a dueling (value/advantage) decomposition for Q-value parameterization. Beyond final placement performance, we assess policy behavior using stability-oriented metrics that quantify steering smoothness over time, providing additional operational insight into how decision policies respond as uncertainty evolves. The framework is integrated with an API for validation within an industrial geosteering simulator under realistic measurement noise and drilling constraints. Using identical geological realizations, operational limits, and reward definitions across methods, the experiments provide a controlled and high-fidelity evaluation of how alternative decision policies behave throughout the drilling process, rather than evaluating performance solely from the final well trajectory.