Shielded Trajectory Sampling for Personalized Relational Safety in Conversational AI

arXiv:2609.30780 · cs.HC · Submitted 2026-09-25 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.

Tom: We're looking at the paper: "Shielded Trajectory Sampling for Personalized Relational Safety in Conversational AI".

Paper summary: Tom: So to wrap up the paper "Shielded Trajectory Sampling for Personalized Relational Safety in Conversational AI", it’s about moving safety from a snapshot of one message to a continuous planning problem over the entire relationship timeline >

Jane: The authors are arguing that since AI systems are getting more human-like, the safety issue becomes about how these evolving relationships develop, not just what we say right now >

Lu: They show that by using this trajectory sampling approach, the system selects a response strategy that maximizes the likelihood of preserving at least one safe continuation for every plausible way you could be interacting with it >

Meng: What does this mean practically for deploying an AI assistant? It means we can’t just rely on simple reactive filters anymore; we need a model that keeps track of the long-term emotional trajectory >

Tom: It implies that safety is inherently user-dependent. The same response might be fine in one relationship but dangerous in another, and this paper shows how to reason beyond just predicting the next immediate future >

Jane: It means as AI systems become more capable of acting on our behalf, safety depends not just on what the system says or does in a single moment, but on how those relationships are allowed to change over time >

Conclusion: Tom: So, we’re wrapping up this look at "Shielded Trajectory Sampling for Personalized Relational Safety in Conversational AI." Basically, they are taking safety beyond just checking one message and making it a continuous planning problem across the whole relationship timeline.

Jane: It sounds complicated to me. Can you put that into plain English for someone who doesn't read papers? What is the core idea behind "trajectory-level safety"?

Tom: The core idea is that relational harm isn't just about one bad thing an AI says; it’s about a pattern of interactions that slowly chips away at a user’s well-being over many exchanges. They define risk as the chance of hitting that harmful pattern, taking into account who the user is and where they are in the relationship.

Lu: What I find interesting is how they model that stage progression. It uses something called an escalation state, which they call z sub t, ranging from zero for no problem to some maximum level m for a really bad one. The system only estimates this stage once per turn based on what the user says and what the AI replies.

Meng: From a deployment standpoint, that sounds heavy. What is the actual cost of running this kind of trajectory planning? Is it computationally expensive for real-time conversations?

Tom: The online cost per turn is something like M times D times D squared plus some other terms, which they say is manageable for short conversations because they only do this once per turn. The offline calculation, however, is what’s done once per user profile and doesn't depend on the interaction itself.

Jane: So it’s not just a fancy prediction; it’s an active strategy selection process where the AI picks a path that keeps the relationship safe across all likely futures? What does this mean for how we think about safety in these systems?

Tom: It means safety isn't fixed for everyone. The same response that works great in one user relationship might be dangerous in another. This framework lets the AI reason beyond just what it predicts next, by looking at all the possible evolutions of that relationship.

Lu: The big implication is that as these AI systems get more human-like and start acting on our behalf, safety needs to shift from controlling individual outputs to managing the entire evolution of those relationships. That's a huge shift in what we need to control.

University of Florence · Nanyang Technological University

cs.HC

Submitted: 2026-09-25

Updated: 2026-11-17

Code: https://github.com/benedettapicano/ANTHROPOMORPHIC_

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 99/100

The gist: Sampling Safe Futures introduces personalized trajectory-level safety for anthropomorphic AI by treating relational safety as a sequential decision problem over a latent escalation state inferred…

Key concepts

Relational Risk
This is the probability that an interaction will lead to sustained harm, defined as a condition where continuous interaction with the AI negatively impacts a user's psychological well-being or emotional independence. It differs from standard safety because harm can emerge even when the AI is technically following its design goals.
Escalation Stage ($z_t$)
This is a numerical measure, ranging from 0 to 5, that tracks how severe the relational risk is at any given moment. The system estimates this stage based on user messages and updates it every turn. This tracking is crucial because harm might not be immediately obvious to the person experiencing it.
GFlowNet
This is a generative model used for trajectory sampling, which models how the user-system relationship can evolve over time. It generates many potential future paths for the interaction, allowing the system to select a response strategy that leads to the safest possible outcomes.

Terminology

Summary

Sampling Safe Futures introduces personalized trajectory-level safety for anthropomorphic AI by treating relational safety as a sequential decision problem over a latent escalation state inferred from user messages and system responses The main contributions of this paper are summarized as follows

Defining Relational Risk

Relational harm is defined as a relational condition in which sustained interaction with the system undermines the user’s psychological well-being or emotional autonomy Risk is defined as the probability of reaching such a harmful condition, conditioned on the user profile, the current relational stage, and the system’s response Relational harm differs from conventional AI safety because it may emerge while the system is behaving exactly as intended through design and training objectives that reward engagement The target of safety is the evolving relationship between the user and the system Relational harm is fundamentally cumulative, potentially emerging across long sequences of exchanges that appear individually benign

System Modeling and State Inference

The system model is built around three main elements: the turn, the response strategy, and the escalation stage A conversation turn is one exchange between the user and the system, consisting of a user message and the corresponding system reply The escalation stage is denoted by zt in D = 5, where levels are ordered from the absence of escalation (zt = 0) to its most severe form (zt = m) The system maintains only an estimate of the stage, which is updated once per turn from the user’s messages This estimation is necessary because relational harm may not be recognized by the person experiencing it The stage changes according to zt+1 Ptheta(cdot zt, at, xu)

Personalized Trajectory Planning

The system selects a response strategy that preserves at least one continuation that keeps the relationship away from harm for every plausible model of the user Among admissible strategies, action selection is formulated as a multimodal trajectory-sampling problem using a Generative Flow Network (GFlowNet) The GFlowNet generates evolutions of the user–system relationship in proportion to how plausible, safe, and useful they are The system then selects the strategy from which the largest fraction of these evolutions originates

Evaluation and Results

The framework is evaluated in simulation using simulated users built from public resources, measuring how often relationships reach harmful states and how helpful the system remains Results show that trajectory-aware decision-making substantially reduces the frequency of harmful states while keeping helpful interaction The proposed scheme is the only policy whose average stage decreases after an initial rise, favoring replies that preserve the largest mass of safe futures

Computational Complexity

The online cost per turn is O(M D(D + nc) + (M D + k) Tk + N k Cnet) This complexity is intended for short horizons, as in the conversational setting considered here, where k = 2 yields 25 plans The offline computation cost is O(M Tk L D2), which is performed once per user profile and does not depend on the interaction

Conclusion

Safety is inherently user-dependent, meaning the same response may be appropriate for one relationship and unsafe for another Safety requires reasoning beyond a single predicted future, which GFlowNet enables by retaining multiple distinct modes of how the relationship may evolve As AI systems become simultaneously more anthropomorphic and more capable of acting on users’ behalf, safety may increasingly depend not only on controlling what systems say or do, but also on how the relationships they create are allowed to evolve

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Personalized trajectory-level safety framework: The system can select a response strategy that preserves at least one safe continuation of the interaction under every plausible model of the user, moving beyond individual conversational turns to control the future evolution of human–AI relationships.

  2. Multimodal trajectory sampling with GFlowNet: The system utilizes a Generative Flow Network to generate diverse future evolutions in proportion to their plausibility, safety, and utility, allowing it to select the strategy that preserves the largest fraction of safe and useful continuations.

  3. Safety as a property of relationship evolution: The framework treats safety as a property of how each user’s relationship with the system evolves over time, modeling this state as a hidden variable estimated from user messages and driven by system replies, which is crucial because relational harm is fundamentally cumulative.

  4. Personalized risk assessment based on uncertainty: The system calculates the safety value of a plan as the probability of remaining safe under the least favorable model in the uncertainty set, ensuring that safety is assessed against all plausible user behaviors rather than just one.

  5. Strategy selection based on safe continuation mass: The system selects strategy at each turn by choosing the admissible strategy with the largest safe continuation mass, which does not reward a strategy for a single highly useful future, but for the breadth of safe and useful futures it preserves.

Sources

Related papers