Known By Their Actions: Fingerprinting LLM Browser Agents via UI Traces

summary

The gist

As LLM-based agents increasingly browse the web on users’ behalf, a natural question arises: can websites passively identify which underlying model powers an agent? Agent actions and interaction…

In short

This study shows that passive JavaScript tracking of an agent's actions and timings can identify which underlying Large Language Model powers it with up to 96% accuracy across 14 models. Agent behavior acts as a unique fingerprint, meaning knowing the model identity is the crucial first step for targeted security exploits.

Key concepts

Agent Fingerprinting
This process involves analyzing behavioral traces—like how long an agent pauses between actions or the structure of its navigation—to create a unique signature for the specific LLM driving it. These traces are derived from passive JavaScript injection and are used to classify the agent into one of several known models.
Inter-Event Intervals (IEIs)
IEIs measure the time intervals between consecutive actions taken by an agent while browsing a website. These temporal features are a primary signal for model identification because different LLMs exhibit distinct patterns in how they pace their interactions and navigate pages, providing measurable behavioral data.
Behavioral Traces
These are the raw data collected from an agent's interaction with a web page, encompassing event traces such as navigation structure, click positions, and timing. These traces are formalized as input features for a supervised classification problem designed to predict the originating model based on these observed patterns.

Terminology used across episodes

This episode discusses

The paper

Known By Their Actions: Fingerprinting LLM Browser Agents via UI Traces · Read on arXiv

Oxford Internet Institute, University of Oxford

As LLM-based agents increasingly browse the web on users' behalf, a natural question arises: can websites passively identify which underlying model powers an agent? Doing so would represent a significant security risk, enabling targeted attacks tailored to known model vulnerabilities. Across 14 frontier LLMs and four web environments spanning information retrieval and shopping tasks, we show that an agent's actions and interaction timings, captured via a passive JavaScript tracker, are sufficient to identify the underlying model with up to 96% F1 within a fixed harness. Although fingerprints are harness-specific, with cross-harness transfer falling to near chance, pooling traces across harnesses recovers model identification at 0.74 to 0.81 Macro-F1. We further show that strong classifiers can be trained from roughly one third of the available traces, and that identification improves steadily as more events are observed. Injecting randomised timing delays between actions substantially degrades classifier performance, but does not provide robust protection: a classifier retrained on delayed traces largely recovers performance. We release our code and a labelled corpus of agent traces at https://github.com/KabakaWilliam/known actions.

Transcript

Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.

Nadia: We're looking at the paper: "Known By Their Actions".

Elias: Here's the gist of the paper. As LLM-based agents increasingly browse the web on users’ behalf, a natural question arises: can websites passively identify which underlying model powers an agent?

Nadia: Agent actions and interaction timings captured via a passive JavaScript tracker are sufficient to identify the underlying model with up to 96% F1 across 14 frontier LLMs.

Elias: This demonstrates that an agent’s actions and interaction timings, captured via a passive JavaScript tracker, are sufficient to identify the underlying model with up to 96% F1.

Nadia: Pruning redundant reasoning tokens The main contributions of this paper include demonstrating that agent actions are a fingerprint of model identity, achieving up to 96% classification Macro F1 across 14 frontier models using only behavioural traces collected via passive JavaScript injection.

Title and authors: Nadia: Looking at the title, "Known By Their Actions: Fingerprinting LLM Browser Agents via UI Traces," it really highlights that we're talking about passive observation of what an AI agent does while interacting with a website.

Elias: And the authors are from Oxford, which suggests they’re looking at a very broad set of frontier models and environments, spanning information retrieval and shopping tasks.

Priya: I’m curious about the scope—does this ninety-six percent accuracy hold up across all kinds of agents or just specific ones <ref:2605.14786#pg1>?

Nadia: The paper claims it achieves that high classification Macro F1 score across fourteen frontier models using only behavioral traces collected through passive JavaScript injection, which is a key part of their methodology <ref:2605.14786#pg1>.

Elias: It’s interesting because they also formalize a threat model with defense analysis, showing that this kind of agent fingerprinting can actually be maintained practically.

Priya: So it's not just about identification; it's about understanding the practical implications for security and how we might defend against this kind of passive tracking.

The paper's summary: Nadia: They formalize the identification task as a supervised classification problem where they try to predict the originating model, denoted by m hat equals f of tau, given an interaction trace tau generated by an unknown agent.

Elias: The features they use are derived from things like Inter-Event Intervals, which is how long it takes between actions, navigation structure, and general interaction patterns designed to catch regularities caused by the underlying model.

Priya: So the core idea is that the timing and structure of an agent’s clicks and scrolls leave a specific signature that maps back to a particular LLM architecture.

Nadia: Exactly. They extracted forty-one behavioral features from each episode's browser event trace and grouped them into seven categories, including temporal dynamics and scroll behavior, which is how they build this fingerprint <ref:2605.14786#pg3>.

Elias: What’s important for me is how they found the signal; timing is the primary factor because mean absolute SHAP values for their XGBoost classifier show that timing features like Inter-Event Interval standard deviation dominate before retraining on delayed traces.

Priya: That sounds like a good detail to focus on, because it suggests that even if you mess with the timing later, the core action fingerprint is still quite strong initially.

The paper's improvements: Nadia: The paper points out several ways this approach can be improved. They suggest making AI systems better by conditioning prompt injection payloads on that inferred model identity after passive fingerprinting is complete.

Elias: That means an attacker could select directly from a known corpus of model-specific jailbreaks or start a targeted optimization with a much smaller search space, which makes the injection much more precise.

Priya: From my view, that shifts the defense from trying to block all attacks to making the target so specific that only known vulnerabilities apply.

Nadia: They also discuss countermeasures against budget exhaustion vectors, where deployed agents could be protected by serving pages calibrated to maximize token consumption or use structures known to trigger extended reasoning.

Elias: That’s a defense against agent-specific access control, trying to disrupt the adversary's ability to exploit high-cost models like GPT-five point four or Claude Opus four point six by making the task too expensive for them <ref:2605.14786#pg2>.

Priya: It sounds like they are suggesting defenses that work on two levels: one level is identifying the agent, and another is manipulating what happens next based on that identity.

Conclusion: Nadia: So to wrap up, the study confirms that UI actions from agent traces can fingerprint the underlying model of an LLM-based web agent with up to ninety-six percent F1 across fourteen frontier models <ref:2605.14786#pg1>.

Elias: The practical consequence is a shift in how we think about threat landscapes because target identification becomes the first step toward exploitation, enabling model-specific prompt injection or access control.

Priya: I think the big takeaway for us is that the operative question for future agent-aware web infrastructure isn't just whether a client is human or automated anymore, but which specific model is actually producing that behavior.

Nadia: Exactly. We've learned that UI actions are a fingerprint, and now we have to figure out how to build defenses around knowing the identity of the agent performing those actions.

Elias: This paper, "Known By Their Actions: Fingerprinting LLM Browser Agents via UI Traces," shows that we can identify agents with high accuracy using behavioral traces collected via passive JavaScript injection.

More episodes

← Home