Known By Their Actions: Fingerprinting LLM Browser Agents via UI Traces

arXiv:2605.14786 · cs.CR, cs.AI, cs.HC, cs.LG · Submitted 2026-05-14 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.

Nadia: We're looking at the paper: "Known By Their Actions".

Elias: Here's the gist of the paper. As LLM-based agents increasingly browse the web on users’ behalf, a natural question arises: can websites passively identify which underlying model powers an agent?

Nadia: Agent actions and interaction timings captured via a passive JavaScript tracker are sufficient to identify the underlying model with up to 96% F1 across 14 frontier LLMs.

Elias: This demonstrates that an agent’s actions and interaction timings, captured via a passive JavaScript tracker, are sufficient to identify the underlying model with up to 96% F1.

Nadia: Pruning redundant reasoning tokens The main contributions of this paper include demonstrating that agent actions are a fingerprint of model identity, achieving up to 96% classification Macro F1 across 14 frontier models using only behavioural traces collected via passive JavaScript injection.

Title and authors: Nadia: Looking at the title, "Known By Their Actions: Fingerprinting LLM Browser Agents via UI Traces," it really highlights that we're talking about passive observation of what an AI agent does while interacting with a website.

Elias: And the authors are from Oxford, which suggests they’re looking at a very broad set of frontier models and environments, spanning information retrieval and shopping tasks.

Priya: I’m curious about the scope—does this ninety-six percent accuracy hold up across all kinds of agents or just specific ones <ref:2605.14786#pg1>?

Nadia: The paper claims it achieves that high classification Macro F1 score across fourteen frontier models using only behavioral traces collected through passive JavaScript injection, which is a key part of their methodology <ref:2605.14786#pg1>.

Elias: It’s interesting because they also formalize a threat model with defense analysis, showing that this kind of agent fingerprinting can actually be maintained practically.

Priya: So it's not just about identification; it's about understanding the practical implications for security and how we might defend against this kind of passive tracking.

The paper's summary: Nadia: They formalize the identification task as a supervised classification problem where they try to predict the originating model, denoted by m hat equals f of tau, given an interaction trace tau generated by an unknown agent.

Elias: The features they use are derived from things like Inter-Event Intervals, which is how long it takes between actions, navigation structure, and general interaction patterns designed to catch regularities caused by the underlying model.

Priya: So the core idea is that the timing and structure of an agent’s clicks and scrolls leave a specific signature that maps back to a particular LLM architecture.

Nadia: Exactly. They extracted forty-one behavioral features from each episode's browser event trace and grouped them into seven categories, including temporal dynamics and scroll behavior, which is how they build this fingerprint <ref:2605.14786#pg3>.

Elias: What’s important for me is how they found the signal; timing is the primary factor because mean absolute SHAP values for their XGBoost classifier show that timing features like Inter-Event Interval standard deviation dominate before retraining on delayed traces.

Priya: That sounds like a good detail to focus on, because it suggests that even if you mess with the timing later, the core action fingerprint is still quite strong initially.

The paper's improvements: Nadia: The paper points out several ways this approach can be improved. They suggest making AI systems better by conditioning prompt injection payloads on that inferred model identity after passive fingerprinting is complete.

Elias: That means an attacker could select directly from a known corpus of model-specific jailbreaks or start a targeted optimization with a much smaller search space, which makes the injection much more precise.

Priya: From my view, that shifts the defense from trying to block all attacks to making the target so specific that only known vulnerabilities apply.

Nadia: They also discuss countermeasures against budget exhaustion vectors, where deployed agents could be protected by serving pages calibrated to maximize token consumption or use structures known to trigger extended reasoning.

Elias: That’s a defense against agent-specific access control, trying to disrupt the adversary's ability to exploit high-cost models like GPT-five point four or Claude Opus four point six by making the task too expensive for them <ref:2605.14786#pg2>.

Priya: It sounds like they are suggesting defenses that work on two levels: one level is identifying the agent, and another is manipulating what happens next based on that identity.

Conclusion: Nadia: So to wrap up, the study confirms that UI actions from agent traces can fingerprint the underlying model of an LLM-based web agent with up to ninety-six percent F1 across fourteen frontier models <ref:2605.14786#pg1>.

Elias: The practical consequence is a shift in how we think about threat landscapes because target identification becomes the first step toward exploitation, enabling model-specific prompt injection or access control.

Priya: I think the big takeaway for us is that the operative question for future agent-aware web infrastructure isn't just whether a client is human or automated anymore, but which specific model is actually producing that behavior.

Nadia: Exactly. We've learned that UI actions are a fingerprint, and now we have to figure out how to build defenses around knowing the identity of the agent performing those actions.

Elias: This paper, "Known By Their Actions: Fingerprinting LLM Browser Agents via UI Traces," shows that we can identify agents with high accuracy using behavioral traces collected via passive JavaScript injection.

Oxford Internet Institute, University of Oxford

cs.CR, cs.AI, cs.HC, cs.LG

Submitted: 2026-05-14

Updated: 2026-11-10

Comments: Accepted at NeurIPS 2026

Code: https://github.com/ByteDance-Seed/Seed2.0

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 96/100

The gist: As LLM-based agents increasingly browse the web on users’ behalf, a natural question arises: can websites passively identify which underlying model powers an agent? Agent actions and interaction…

Key concepts

Agent Fingerprinting
This process involves analyzing behavioral traces—like how long an agent pauses between actions or the structure of its navigation—to create a unique signature for the specific LLM driving it. These traces are derived from passive JavaScript injection and are used to classify the agent into one of several known models.
Inter-Event Intervals (IEIs)
IEIs measure the time intervals between consecutive actions taken by an agent while browsing a website. These temporal features are a primary signal for model identification because different LLMs exhibit distinct patterns in how they pace their interactions and navigate pages, providing measurable behavioral data.
Behavioral Traces
These are the raw data collected from an agent's interaction with a web page, encompassing event traces such as navigation structure, click positions, and timing. These traces are formalized as input features for a supervised classification problem designed to predict the originating model based on these observed patterns.

Terminology

Summary

As LLM-based agents increasingly browse the web on users’ behalf, a natural question arises: can websites passively identify which underlying model powers an agent? p. 0

Where the paper says it

Known By Their Actions: Fingerprinting LLM Browser Agents via UI Traces William Lugoloobi1∗ Samuelle Marro2 Jabez Magomere1 Joss Wright1 Chris Russell1 1Oxford Internet Institute, University of Oxford 2Department of Engineering Science, University of Oxford Abstract As LLM-based agents increasingly browse the web on users’ behalf, a natural question arises: can websites passively identify which underlying model powers an agent? Doing so would represent a significant security risk, enabling targeted attacks tailored to known model vulnerabilities.

Agent actions and interaction timings captured via a passive JavaScript tracker are sufficient to identify the underlying model with up to 96% F1 across 14 frontier LLMs. p. 0

Where the paper says it

Doing so would represent a significant security risk, enabling targeted attacks tailored to known model vulnerabilities. Across 14 frontier LLMs and four web environments spanning information retrieval and shopping tasks, we show that an agent’s actions and interaction timings, captured via a passive JavaScript tracker, are sufficient to identify the underlying model with up to 96% F1.

This demonstrates that an agent’s actions and interaction timings, captured via a passive JavaScript tracker, are sufficient to identify the underlying model with up to 96% F1. p. 0
Where the paper says it

Doing so would represent a significant security risk, enabling targeted attacks tailored to known model vulnerabilities. Across 14 frontier LLMs and four web environments spanning information retrieval and shopping tasks, we show that an agent’s actions and interaction timings, captured via a passive JavaScript tracker, are sufficient to identify the underlying model with up to 96% F1.

Pruning redundant reasoning tokens

The main contributions of this paper include demonstrating that agent actions are a fingerprint of model identity, achieving up to 96% classification Macro F1 across 14 frontier models using only behavioural traces collected via passive JavaScript injection. Furthermore, the authors formalize a threat model with defense analysis, showing that agent fingerprinting is practical to maintain and that standard browser normalisation with randomised delays is insufficient to remove the identifying signal when the adversary retrains their classifier on delayed traces. p. 1

Where the paper says it

We characterise agent fingerprinting under a passive co-located adversary, and show that the attack is practical to maintain: new models can be enrolled by routing a small number of sessions through the instrumented site. We further show that standard browser normalisation with randomised delays is insufficient to remove the identifying signal when the adversary retrains their classifier on delayed traces.

How the training signal is built

The identification task is formalized as a supervised classification problem over behavioural traces, where the goal is to predict the originating model mˆ = f(τ), mˆ ∈ M, given an interaction trace τ generated by an unknown agent. Features are derived from Inter-Event Intervals (IEIs), navigation structure, and interaction patterns designed to capture behavioural regularities induced by the underlying model. p. 2

Where the paper says it

Feature creation. We consider feature mappings ϕ(τ ) derived from Inter-Event Intervals (IEIs, the time between two consecutive actions), navigation structure (e.

The 41 behavioral features extracted from each episode’s browser event trace are grouped into seven categories, including temporal dynamics and scroll behavior. p. 4
Where the paper says it

Trace collection We instrument each page with a lightweight JavaScript observer injected at session initialisation. The observer attaches event listeners to the DOM and records every interaction event produced by the agent, including click coordinates and target element type, scroll direction and magnitude, keypress events and inter-keystroke timing, and navigation events with timestamps.

Where the gains come from

Timing is the primary signal, as mean absolute SHAP values for the XGBoost classifier show that timing-based features like IEI standard deviation dominate agent identification before retraining on delayed traces. Actions carry the fingerprint when timing is disrupted, and retraining on delayed traces allows the classifier to rely on action-centred features such as structural key ratio and click position to make predictions. p. 6

Where the paper says it

Each bar shows the change in mean |SHAP| for a feature when XGBoost is retrained on 5-second-delayed traces. Under normal conditions, timing features based on IEI statistics and time to first action dominate agent identification, but after training on delayed traces, our classifier relies on action-centred features like structural key ratio and click position to make predictions.

Strong classifiers can be trained from few observed events, with fewer than one third of traces sufficient to approach peak classification performance across all four datasets. p. 6
Where the paper says it

2 Agent fingerprinting is efficient at both training and test time Strong classifiers can be trained from few observed events. By varying the proportion of training data used to fit our XGBoost classifier, we find that fewer than one third of traces are sufficient to approach peak classification performance across all four datasets (Figure 6A).

Implications for security

Knowing the identity of a target makes an adversary’s job easier by reducing the search space from all possible attacks to those known to be effective against the target. This enables model-specific prompt injection, adversarial cost inflation, and agent-specific access control, converting generic exploits into targeted ones across three attack surfaces. p. 8

Where the paper says it

of threats known as AI Agent Traps: adversarial page content designed to manipulate, deceive, or exploit visiting agents [47]. Passive fingerprinting lets each trap be conditioned on model identity, converting generic exploits into targeted ones across three attack surfaces we characterise below Model-specific prompt injection.

The operative question for the next generation of agent-aware web infrastructure is therefore no longer whether a client is human or automated, but which model is producing the behaviour. p. 8
Where the paper says it

Every page visit thus becomes a potential attribution event on which model-specific injections, content poisoning, or budget exhaustion can be conditioned. The operative question for the next generation of agent-aware web infrastructure is therefore no longer whether a client is human or automated, but which model is producing the behaviour.

p. 8
Where the paper says it

Every page visit thus becomes a potential attribution event on which model-specific injections, content poisoning, or budget exhaustion can be conditioned. The operative question for the next generation of agent-aware web infrastructure is therefore no longer whether a client is human or automated, but which model is producing the behaviour.

Conclusion

The study shows that UI actions from agent traces can fingerprint the underlying model of an LLM-based web agent with up to 96% F1 across 14 frontier models. The practical consequence is a shift in the threat landscape for deployed agents, where target identification is the first step towards exploitation. p. 8

Where the paper says it

The practical consequence is a shift in the threat landscape for deployed agents. Target identification is the first step towards exploitation, and we show that this move can be executed passively during ordinary navigation by any party that controls a page visited by the agent.

The operative question for the next generation of agent-aware web infrastructure is therefore no longer whether a client is human or automated, but which model is producing the behaviour.

Acknowledgements

LW and JM were supported by a Rhodes Scholarship, SM was supported by the EPSRC Centre for Doctoral Training in Autonomous Intelligent Machines and Systems n. EP/Y035070/1, in addition to Microsoft Ltd. p. 9

Where the paper says it

SM was supported by the EPSRC Centre for Doctoral Training in Autonomous Intelligent Machines and Systems n. EP/Y035070/1, in addition to Microsoft Ltd.

The paper is 2605. p. 0
Where the paper says it

uk Preprint. arXiv:2605.

14786.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Targeted Adversarial Defense Against Prompt Injection: AI systems can be improved by conditioning prompt injection payloads on inferred model identity after passive fingerprinting is complete, making them invisible to any visitor whose model does not match the target. This allows adversaries to select directly from a known corpus of model-specific jailbreaks or initialise a targeted optimisation with a substantially reduced search space.

  2. Budget Exhaustion Countermeasures: Deployed agents can be protected against denial-of-service vectors by implementing defenses that prevent site operators who know the visiting model from serving pages calibrated to maximise its token consumption, through navigational ambiguity, redundant content, or structures known to trigger extended reasoning. This targets agent-specific access control by disrupting the adversary's ability to exploit high-cost models like GPT-5.4 or Claude Opus 4.6.

  3. Robust and Adaptive Fingerprinting: Systems can be improved by training classifiers that are robust against timing perturbations, as they retrain on delayed traces largely recovers performance. This ensures agent identification remains effective even when an adversary injects randomised timing delays between actions, maintaining the ability to identify agents even under attack.

  4. Efficient Identification During Interaction: AI systems can be designed to infer model identity early in a session, as the paper shows that identification can occur while the agent is still navigating the page, leaving ample opportunity to condition a subsequent attack on the inferred identity. This allows for proactive defense mechanisms immediately upon initial interaction rather than waiting for session completion.

  5. Site-Conditioned Defense Strategies: Infrastructure can be improved by utilizing pooled-site training strategies, as pooling multiple tasks from the same website recovers strong attribution even when single-task transfer is weak. This means a site operator cannot rely on a fingerprint specific to one task but must manage an identifier that reflects the agent's behavior across all tasks visited on that domain.

Abstract

As LLM-based agents increasingly browse the web on users' behalf, a natural question arises: can websites passively identify which underlying model powers an agent? Doing so would represent a significant security risk, enabling targeted attacks tailored to known model vulnerabilities. Across 14 frontier LLMs and four web environments spanning information retrieval and shopping tasks, we show that an agent's actions and interaction timings, captured via a passive JavaScript tracker, are sufficient to identify the underlying model with up to 96% F1 within a fixed harness. Although fingerprints are harness-specific, with cross-harness transfer falling to near chance, pooling traces across harnesses recovers model identification at 0.74 to 0.81 Macro-F1. We further show that strong classifiers can be trained from roughly one third of the available traces, and that identification improves steadily as more events are observed. Injecting randomised timing delays between actions substantially degrades classifier performance, but does not provide robust protection: a classifier retrained on delayed traces largely recovers performance. We release our code and a labelled corpus of agent traces at https://github.com/KabakaWilliam/known actions.

Sources

Related papers