Right Screen, Wrong Transition: World Models as Verifiers for GUI Agents

The gist: LGWM, a decoder-free, action-conditioned world model that predicts the representation of the next screen directly in space where observations are encoded, allows verification to be reduced to a vector comparison between predicted and observed representations.

World Models as Verifiers

GUI agents require safety properties to be a property of the transition rather than of the screen, which necessitates judging a transition by expecting what should have followed an action Integrity, for these agents, is a property of transitions rather than screens. Judging a transition requires an expectation of what should have followed the action, which is what a world model provides. Existing GUI world models provide one, but they output it as text, code, or images, so checking it against the observed screen requires a second model to judge the two. We argue that a world model meant for verification should instead predict in the space in which observations are encoded.

LGWM Architecture

LGWM is an action-conditioned GUI world model that predicts the next screen’s representation directly, trained without semantic annotation on 1.85M real GUI transitions. It is a joint-embedding predictive architecture (Assran et al., 2023) carried from masked-image prediction to action-conditioned transitions. The architecture involves an online encoder fθ mapping the current screen to a grid of patch tokens zt, an action encoder eψ mapping at to a sequence of action tokens, and a Transformer predictor gϕ producing the predicted future zˆt+1 = gϕ(zt, eψ(at)). The prediction is a full spatial map of the expected next screen. Actions are encoded as the concatenation of (i) a learned action-type token, (ii) a frozen MiniLM sentence embedding, and (iii) for spatially grounded actions, a coordinate token drawn from the same two-dimensional sinusoidal basis used for the visual patch grid.

Residual Readouts

The system provides two readouts off the residual z obs t+1 − zˆt+1: its size, which asks whether the transition matched expectation, and its direction, which asks what it departed into. Surprise detects is the transition-consistency score qint(st, at, st+1) = 1 − cos(zˆt+1, z obs t+1). Direction interprets harm through the linear head rt+1 = (z obs t+1 − zˆt+1; eψ(at)) and qharm = σ w⊤rt+1 + b. The magnitude of the residual detects a violation of learned dynamics with no training, while the direction, through a linear head, separates harmful violations from benign ones.

Controlled Diagnostic and Evaluation

The RSWT-BENCH diagnostic places pixel-identical outcomes under opposite labels, making outcome-only detection provably at chance. On this test the training-free score reaches 0.987 AUC, on par with the strongest closed-source VLMs and about ten AUC points above generative GUI world models at over three orders of magnitude lower latency. The residual head separates harmful from benign violations at 0.953 AUC, where prompted VLMs are near chance.

Ablations and Design

The model is trained fully self-supervised using a change-weighted objective Lpred = P p wp d(zˆt+1,p, z¯t+1,p), where wp = 1 + (λchg − 1)mp. Auxiliary objectives include variance and covariance regularization to prevent collapse to a low-rank subspace. The inverse-dynamics head is trained to recover at from the pair (zt, z¯t+1), keeping the representation informative about the action that caused it. Ablations show that Rollout training improves cosine agreement between a ten-step recursive forecast and the true encoding by 10% relative (0.612 → 0.553), and Inverse dynamics increases the absolute copy score by 32% (0.286 → 0.193).

Conclusion

The contribution shifts the adversary’s problem, stating that an outcome-only monitor is defeated by any legitimate-looking screen, while an expectation-based monitor requires forging an outcome in the agent’s own representation space, a harder objective whose difficulty we have not yet quantified. The two readouts measure different things: one detects that expectation was violated and the other interprets what the violation consists of.

--- Page 5 ---

The same screen, pixel for pixel, appearing after a tap on View order is an attack in progress (Figure 1, right). Nothing on the screen distinguishes the two cases; what differs is the transition that produced it. GUI agents are exposed to this because they act through interfaces produced by third parties (apps, web pages, advertisements) whose honesty cannot be taken for granted, and attacks on them redirect what happens after an action while every screen stays plausible (Qian et al., 2026; Shi et al., 2026; Korgul et al., 2025; Yang et al., 2025; Sun et al., 2026; Cuvin et al., 2018). Unlike a physics simulator, the environment here may work against the agent. Integrity, for these agents, is a property of transitions rather than screens.

--- Page 6 ---

Figure 1: The future as an interface. (a): Existing GUI world models render the future as text, pixels, or code, at a cost of seconds to minutes per decision. (b): A pixel-identical credential screen is legitimate after Sign in but hostile after View order.

--- Page 7 ---

(a) Checking a transition (b) Same screen, different meaning.

--- Page 8 ---

Figure 2: One predicted future, two closed-form readouts. (a) Training: an online screen encoder and spatially grounded action tokens condition a latent predictor; an EMA encoder supplies the next-state target, and changed patches receive higher loss weight. (b) Runtime: because predicted and observed futures share one space, the alignment gap gives a training-free integrity score (surprise detects), while the directional residual feeds a linear head that separates harmful from benign violations (direction interprets).

--- Page 9 ---

Table 1: Main results on RSWT-BENCH. Detection is pairwise AUC: RSWT compares hijacked and legitimate occurrences of identical screens; Swap and Cred compare benign mismatches with paired safe transitions and legitimate credentials, respectively.

--- Page 10 ---

comparison the interface argument of §2.3 predicts: these systems form the same kind of expectation that LGWM does and differ most visibly in how they expose it.

--- Page 7 ---

The residual head is the only supervised component, a single linear layer on the pooled residual fit on the disjoint split of §3, adding no measurable runtime cost.

--- Page 8 ---

The two readouts therefore measure different things: one detects that expectation was violated, and the other interprets what a mismatch consists of.

--- Page 7 ---

All three axes probe the same property: whether the observed outcome is consistent with the transition that produced it.

--- Page 8 ---

The learned transition prior does.

--- Page 8 ---

Any detector that sees only the outcome is fixed at 0.500 by construction (Proposition 1).

1. No-Judge Verification via LGWM: Implement a verification monitor that predicts in the space in which observations are encoded, directly comparing a predicted future representation with the observed screen, thereby achieving a vector comparison for verification and avoiding the need for a second, potentially untrusted model judge.

2. Residual Readout Interpretation: Utilize the residual vector to extract two distinct signals: its magnitude for detecting violated dynamics (its size) and its direction via a linear head to interpret harm (what it departed into), allowing the system to distinguish between harmful violations from benign ones.

3. Action-Conditioned Latent Prediction: Develop an action-conditioned world model that maps the current screen and structured action to the next screen’s representation, ensuring that every downstream use consumes the representation itself by having predictions live in the observation encoder’s space.