Known By Their Actions: Fingerprinting LLM Browser Agents via UI Traces
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.
Nadia: We're looking at the paper: "Known By Their Actions".
Elias: Here's the gist of the paper. As LLM-based agents increasingly browse the web on users’ behalf, a natural question arises: can websites passively identify which underlying model powers an agent?
Nadia: Agent actions and interaction timings captured via a passive JavaScript tracker are sufficient to identify the underlying model with up to 96% F1 across 14 frontier LLMs.
Elias: This demonstrates that an agent’s actions and interaction timings, captured via a passive JavaScript tracker, are sufficient to identify the underlying model with up to 96% F1.
Nadia: Pruning redundant reasoning tokens The main contributions of this paper include demonstrating that agent actions are a fingerprint of model identity, achieving up to 96% classification Macro F1 across 14 frontier models using only behavioural traces collected via passive JavaScript injection.
Title and authors: Nadia: Looking at the title, "Known By Their Actions: Fingerprinting LLM Browser Agents via UI Traces," it really highlights that we're talking about passive observation of what an AI agent does while interacting with a website.
Elias: And the authors are from Oxford, which suggests they’re looking at a very broad set of frontier models and environments, spanning information retrieval and shopping tasks.
Priya: I’m curious about the scope—does this ninety-six percent accuracy hold up across all kinds of agents or just specific ones <ref:2605.14786#pg1>?
Nadia: The paper claims it achieves that high classification Macro F1 score across fourteen frontier models using only behavioral traces collected through passive JavaScript injection, which is a key part of their methodology <ref:2605.14786#pg1>.
Elias: It’s interesting because they also formalize a threat model with defense analysis, showing that this kind of agent fingerprinting can actually be maintained practically.
Priya: So it's not just about identification; it's about understanding the practical implications for security and how we might defend against this kind of passive tracking.
The paper's summary: Nadia: They formalize the identification task as a supervised classification problem where they try to predict the originating model, denoted by m hat equals f of tau, given an interaction trace tau generated by an unknown agent.
Elias: The features they use are derived from things like Inter-Event Intervals, which is how long it takes between actions, navigation structure, and general interaction patterns designed to catch regularities caused by the underlying model.
Priya: So the core idea is that the timing and structure of an agent’s clicks and scrolls leave a specific signature that maps back to a particular LLM architecture.
Nadia: Exactly. They extracted forty-one behavioral features from each episode's browser event trace and grouped them into seven categories, including temporal dynamics and scroll behavior, which is how they build this fingerprint <ref:2605.14786#pg3>.
Elias: What’s important for me is how they found the signal; timing is the primary factor because mean absolute SHAP values for their XGBoost classifier show that timing features like Inter-Event Interval standard deviation dominate before retraining on delayed traces.
Priya: That sounds like a good detail to focus on, because it suggests that even if you mess with the timing later, the core action fingerprint is still quite strong initially.
The paper's improvements: Nadia: The paper points out several ways this approach can be improved. They suggest making AI systems better by conditioning prompt injection payloads on that inferred model identity after passive fingerprinting is complete.
Elias: That means an attacker could select directly from a known corpus of model-specific jailbreaks or start a targeted optimization with a much smaller search space, which makes the injection much more precise.
Priya: From my view, that shifts the defense from trying to block all attacks to making the target so specific that only known vulnerabilities apply.
Nadia: They also discuss countermeasures against budget exhaustion vectors, where deployed agents could be protected by serving pages calibrated to maximize token consumption or use structures known to trigger extended reasoning.
Elias: That’s a defense against agent-specific access control, trying to disrupt the adversary's ability to exploit high-cost models like GPT-five point four or Claude Opus four point six by making the task too expensive for them <ref:2605.14786#pg2>.
Priya: It sounds like they are suggesting defenses that work on two levels: one level is identifying the agent, and another is manipulating what happens next based on that identity.
Conclusion: Nadia: So to wrap up, the study confirms that UI actions from agent traces can fingerprint the underlying model of an LLM-based web agent with up to ninety-six percent F1 across fourteen frontier models <ref:2605.14786#pg1>.
Elias: The practical consequence is a shift in how we think about threat landscapes because target identification becomes the first step toward exploitation, enabling model-specific prompt injection or access control.
Priya: I think the big takeaway for us is that the operative question for future agent-aware web infrastructure isn't just whether a client is human or automated anymore, but which specific model is actually producing that behavior.
Nadia: Exactly. We've learned that UI actions are a fingerprint, and now we have to figure out how to build defenses around knowing the identity of the agent performing those actions.
Elias: This paper, "Known By Their Actions: Fingerprinting LLM Browser Agents via UI Traces," shows that we can identify agents with high accuracy using behavioral traces collected via passive JavaScript injection.
Oxford Internet Institute, University of Oxford
cs.CR, cs.AI, cs.HC, cs.LG
Submitted: 2026-05-14
Updated: 2026-11-10
Comments: Accepted at NeurIPS 2026
Code: https://github.com/ByteDance-Seed/Seed2.0
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 96/100
The gist: As LLM-based agents increasingly browse the web on users’ behalf, a natural question arises: can websites passively identify which underlying model powers an agent? Agent actions and interaction…
Key concepts
- Agent Fingerprinting
- This process involves analyzing behavioral traces—like how long an agent pauses between actions or the structure of its navigation—to create a unique signature for the specific LLM driving it. These traces are derived from passive JavaScript injection and are used to classify the agent into one of several known models.
- Inter-Event Intervals (IEIs)
- IEIs measure the time intervals between consecutive actions taken by an agent while browsing a website. These temporal features are a primary signal for model identification because different LLMs exhibit distinct patterns in how they pace their interactions and navigate pages, providing measurable behavioral data.
- Behavioral Traces
- These are the raw data collected from an agent's interaction with a web page, encompassing event traces such as navigation structure, click positions, and timing. These traces are formalized as input features for a supervised classification problem designed to predict the originating model based on these observed patterns.
Terminology
Summary
As LLM-based agents increasingly browse the web on users’ behalf, a natural question arises: can websites passively identify which underlying model powers an agent? p. 0 Known By Their Actions: Fingerprinting LLM
Browser Agents via UI Traces
William Lugoloobi1∗ Samuelle Marro2 Jabez Magomere1 Joss Wright1 Chris Russell1
1Oxford Internet Institute, University of Oxford
2Department of Engineering Science, University of Oxford
Abstract
As LLM-based agents increasingly browse the web on users’ behalf, a natural
question arises: can websites passively identify which underlying model powers
an agent? Doing so would represent a significant security risk, enabling targeted
attacks tailored to known model vulnerabilities.Where the paper says it
Agent actions and interaction timings captured via a passive JavaScript tracker are sufficient to identify the underlying model with up to 96% F1 across 14 frontier LLMs. p. 0 Doing so would represent a significant security risk, enabling targeted
attacks tailored to known model vulnerabilities. Across 14 frontier LLMs and four
web environments spanning information retrieval and shopping tasks, we show
that an agent’s actions and interaction timings, captured via a passive JavaScript
tracker, are sufficient to identify the underlying model with up to 96% F1. Doing so would represent a significant security risk, enabling targeted
attacks tailored to known model vulnerabilities. Across 14 frontier LLMs and four
web environments spanning information retrieval and shopping tasks, we show
that an agent’s actions and interaction timings, captured via a passive JavaScript
tracker, are sufficient to identify the underlying model with up to 96% F1.Where the paper says it
Where the paper says it
Pruning redundant reasoning tokens
The main contributions of this paper include demonstrating that agent actions are a fingerprint of model identity, achieving up to 96% classification Macro F1 across 14 frontier models using only behavioural traces collected via passive JavaScript injection. Furthermore, the authors formalize a threat model with defense analysis, showing that agent fingerprinting is practical to maintain and that standard browser normalisation with randomised delays is insufficient to remove the identifying signal when the adversary retrains their classifier on delayed traces. p. 1 We characterise agent fingerprinting
under a passive co-located adversary, and show that the attack is practical to maintain: new
models can be enrolled by routing a small number of sessions through the instrumented site. We further show that standard browser normalisation with randomised delays is insufficient
to remove the identifying signal when the adversary retrains their classifier on delayed
traces.Where the paper says it
How the training signal is built
The identification task is formalized as a supervised classification problem over behavioural traces, where the goal is to predict the originating model mˆ = f(τ), mˆ ∈ M, given an interaction trace τ generated by an unknown agent. Features are derived from Inter-Event Intervals (IEIs), navigation structure, and interaction patterns designed to capture behavioural regularities induced by the underlying model. p. 2 Feature creation. We consider feature mappings ϕ(τ ) derived from Inter-Event Intervals (IEIs,
the time between two consecutive actions), navigation structure (e. Trace collection We instrument each page with a lightweight JavaScript observer injected at session
initialisation. The observer attaches event listeners to the DOM and records every interaction event
produced by the agent, including click coordinates and target element type, scroll direction and
magnitude, keypress events and inter-keystroke timing, and navigation events with timestamps.Where the paper says it
Where the paper says it
Where the gains come from
Timing is the primary signal, as mean absolute SHAP values for the XGBoost classifier show that timing-based features like IEI standard deviation dominate agent identification before retraining on delayed traces. Actions carry the fingerprint when timing is disrupted, and retraining on delayed traces allows the classifier to rely on action-centred features such as structural key ratio and click position to make predictions. p. 6 Each bar shows the change in mean |SHAP| for a feature when XGBoost is retrained on
5-second-delayed traces. Under normal conditions, timing features based on IEI statistics and time to
first action dominate agent identification, but after training on delayed traces, our classifier relies on
action-centred features like structural key ratio and click position to make predictions. 2 Agent fingerprinting is efficient at both training and test time
Strong classifiers can be trained from few observed events. By varying the proportion of training
data used to fit our XGBoost classifier, we find that fewer than one third of traces are sufficient to
approach peak classification performance across all four datasets (Figure 6A).Where the paper says it
Where the paper says it
Implications for security
Knowing the identity of a target makes an adversary’s job easier by reducing the search space from all possible attacks to those known to be effective against the target. This enables model-specific prompt injection, adversarial cost inflation, and agent-specific access control, converting generic exploits into targeted ones across three attack surfaces. p. 8 of threats known as AI Agent Traps: adversarial page content designed to manipulate, deceive, or
exploit visiting agents [47]. Passive fingerprinting lets each trap be conditioned on model identity,
converting generic exploits into targeted ones across three attack surfaces we characterise below
Model-specific prompt injection. Every page visit thus
becomes a potential attribution event on which model-specific injections, content poisoning, or budget
exhaustion can be conditioned. The operative question for the next generation of agent-aware web infrastructure is therefore no longer
whether a client is human or automated, but which model is producing the behaviour. Every page visit thus
becomes a potential attribution event on which model-specific injections, content poisoning, or budget
exhaustion can be conditioned. The operative question for the next generation of agent-aware web infrastructure is therefore no longer
whether a client is human or automated, but which model is producing the behaviour.Where the paper says it
Where the paper says it
Where the paper says it
Conclusion
The study shows that UI actions from agent traces can fingerprint the underlying model of an LLM-based web agent with up to 96% F1 across 14 frontier models. The practical consequence is a shift in the threat landscape for deployed agents, where target identification is the first step towards exploitation. p. 8 The practical consequence is a shift in the threat landscape for deployed agents. Target identification
is the first step towards exploitation, and we show that this move can be executed passively during
ordinary navigation by any party that controls a page visited by the agent.Where the paper says it
Acknowledgements
LW and JM were supported by a Rhodes Scholarship, SM was supported by the EPSRC Centre for Doctoral Training in Autonomous Intelligent Machines and Systems n. EP/Y035070/1, in addition to Microsoft Ltd. p. 9 SM was supported by the EPSRC Centre for
Doctoral Training in Autonomous Intelligent Machines and Systems n. EP/Y035070/1, in addition to
Microsoft Ltd. uk
Preprint. arXiv:2605.Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Targeted Adversarial Defense Against Prompt Injection: AI systems can be improved by conditioning prompt injection payloads on inferred model identity after passive fingerprinting is complete, making them
invisible to any visitor whose model does not match the target.
This allows adversaries toselect directly from a known corpus of model-specific jailbreaks
orinitialise a targeted optimisation with a substantially reduced search space.
-
Budget Exhaustion Countermeasures: Deployed agents can be protected against denial-of-service vectors by implementing defenses that prevent site operators who know the visiting model from serving pages
calibrated to maximise its token consumption, through navigational ambiguity, redundant content, or structures known to trigger extended reasoning.
This targetsagent-specific access control
by disrupting the adversary's ability to exploit high-cost models like GPT-5.4 or Claude Opus 4.6. -
Robust and Adaptive Fingerprinting: Systems can be improved by training classifiers that are robust against timing perturbations, as they
retrain on delayed traces largely recovers performance.
This ensures agent identification remains effective even when an adversary injectsrandomised timing delays between actions,
maintaining the ability to identify agents even under attack. -
Efficient Identification During Interaction: AI systems can be designed to infer model identity early in a session, as the paper shows that
identification can occur while the agent is still navigating the page, leaving ample opportunity to condition a subsequent attack on the inferred identity.
This allows for proactive defense mechanisms immediately upon initial interaction rather than waiting for session completion. -
Site-Conditioned Defense Strategies: Infrastructure can be improved by utilizing pooled-site training strategies, as
pooling multiple tasks from the same website recovers strong attribution
even when single-task transfer is weak. This means a site operator cannot rely on a fingerprint specific to one task but must manage an identifier that reflects the agent's behavior across all tasks visited on that domain.
Abstract
As LLM-based agents increasingly browse the web on users' behalf, a natural question arises: can websites passively identify which underlying model powers an agent? Doing so would represent a significant security risk, enabling targeted attacks tailored to known model vulnerabilities. Across 14 frontier LLMs and four web environments spanning information retrieval and shopping tasks, we show that an agent's actions and interaction timings, captured via a passive JavaScript tracker, are sufficient to identify the underlying model with up to 96% F1 within a fixed harness. Although fingerprints are harness-specific, with cross-harness transfer falling to near chance, pooling traces across harnesses recovers model identification at 0.74 to 0.81 Macro-F1. We further show that strong classifiers can be trained from roughly one third of the available traces, and that identification improves steadily as more events are observed. Injecting randomised timing delays between actions substantially degrades classifier performance, but does not provide robust protection: a classifier retrained on delayed traces largely recovers performance. We release our code and a labelled corpus of agent traces at https://github.com/KabakaWilliam/known actions.
Sources
- WebGPT: Browser-assisted question-answering with human feedback
- BeCAPTCHA-Mouse: Synthetic Mouse Trajectories and Improved Bot Detection
- Fast imaging of scattering obstacles from phaseless far-field measurements at a fixed frequency
- WebArena: A Realistic Web Environment for Building Autonomous Agents
- Mind2Web: Towards a Generalist Agent for the Web
- Optimizing Mouse Dynamics for User Authentication by Machine Learning: Addressing Data Sufficiency, Accuracy-Practicality Trade-off, and Model Performance Challenges
- Automated Website Fingerprinting through Deep Learning
- Deep Fingerprinting: Undermining Website Fingerprinting Defenses with Deep Learning
- A Whole New World: Creating a Parallel-Poisoned Web Only AI-Agents Can See
- Exposing LLM User Privacy via Traffic Fingerprint Analysis: A Study of Privacy Risks in LLM Agent Interactions
- Universal and Transferable Adversarial Attacks on Aligned Language Models
- "Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models
- MIP against Agent: Malicious Image Patches Hijacking Multimodal OS Agents
- Preference Redirection via Attention Concentration: An Attack on Computer Use Agents
- Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generation
- WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents
- DeepShop: A Benchmark for Deep Research Shopping Agents
- GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learning
- Qwen3 Technical Report
- UI-TARS: Pioneering Automated GUI Interaction with Native Agents
Related papers
- From Strategy Search to Policy Learning: MCTS-Guided Adversarial Co-Evolution for LLM Safety Alignment
- Privacy Leakage from Recurrent State in LLM Inference
- Incrementally Verifiable Oracle Aided Computation
- Privilege and confidentiality in generative AI workflows
- BadMoE: Backdooring Mixture-of-Experts LLMs via Routing-Aware Triggers and Task-Underutilized Experts
- CoVault: Secure, Scalable Analytics of Personal Data