Beyond the Landing Page: Website Blocking, Tracker Visibility, and Human Alignment in Automated Web Crawls
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.
Nadia: We're looking at the paper: "Beyond the Landing Page".
Elias: Here's the gist of the paper. Web tracking studies largely rely on automated crawlers, yet their measurements can differ from what people encounter while browsing Beyond the Landing Page: Website Blocking, Tracker Visibility, and Human Alignment in Automated Web Crawls.
Nadia: We find that website denials increase beyond landing pages for standard browser automation, while a detection-resistant automation framework largely prevents this escalation without increasing recorded tracking Agentic crawlers record tracking activity from 1.5–1.7 times as many third-party organizations, particularly on consent-managed pages Agentic crawlers recover 82–87% of tracker organizations observed by participants, compared to 58–68% for non-agentic crawlers.
Elias: Comparison of Crawler Types The study compares agentic and non-agentic crawlers across different dimensions to examine how websites respond and what tracking they record.
Nadia: The four non-agentic crawlers use Playwright, a browser automation framework, to drive Chromium or Firefox.
Paper summary: Nadia: Looking at the paper "Beyond the Landing Page: Website Blocking, Tracker Visibility, and Human Alignment in Automated Web Crawls", it seems like there’s a lot to take away about what automated measurements are actually capturing.
Elias: The authors show that agentic crawlers record more tracking data and recover more participant observed trackers, but they also observe activity absent from human visits >
Priya: So the key finding isn't that agents are automatically superior for measurement; it’s that website interventions, tracking visibility, and agreement with human browsing are distinct things we need to measure separately >
Nadia: That’s right. The denial rates on later pages could mostly be reproduced if you used a patched automation framework instead of relying on the agent's navigation style >
Elias: It confirms that agentic crawling doesn't automatically improve web tracking measurements in every way it seems to do, especially when compared to a standard browser automation setup >
Priya: It’s important for us as researchers to stop assuming any single crawler type is inherently better and instead focus on selecting the right crawler based on the specific measurement we are trying to make >
Nadia: The paper gives practical guidance on choosing crawlers according to what you want to measure, rather than just aiming for human likeness or detection avoidance >
Elias: And for agent frameworks, it suggests making consent handling a clear and user controlled choice, rather than treating it as an accidental consequence of clearing navigation obstacles >
Priya: That frames the issue perfectly: we need to be explicit about how our tools interact with the web and users. It moves us toward more intentional design >
Nadia: So what this means for the world is that understanding these differences helps us build better defenses and more accurate privacy measurements for the web as it changes >
Conclusion: Nadia: So we’re wrapping up our take on "Beyond the Landing Page: Website Blocking, Tracker Visibility, and Human Alignment in Automated Web Crawls."
Elias: Basically, this paper looks at how automated tools like crawlers measure tracking when they interact with a real website.
Priya: The main point is that agentic crawlers might seem better for collecting data, but the results don't tell the whole story.
Nadia: Right. They found that agents record more third-party tracking and recover more trackers seen by people, but they also see things humans never saw.
Elias: That’s because website blocks and tracking visibility aren't all about how sneaky the crawler is being.
Priya: The real lesson here is that you need to measure website blocking, which is a different thing from measuring how much tracking you capture on a page.
Nadia: Exactly. If you want to study someone actually browsing, your crawler setup has to match their actual navigation and consent choices on the site.
Elias: The authors stress that this means we shouldn't just pick one type of crawler—agentic or not—and assume it’s automatically the best measurement tool out there.
Priya: It pushes us toward a more thoughtful approach to privacy research, where we define exactly what kind of interaction we are trying to study first.
Nadia: And for anyone building an agent framework, they need to make consent handling a clear choice for the user, not just something the agent does automatically when it gets stuck.
Elias: Because if you treat that as an afterthought, you might end up measuring tracking in ways that don't reflect what a real user is experiencing.
Priya: This distinction between blocking and visibility is crucial because it shows how measurement design impacts the data itself, and we need to be more precise about those designs going forward.
H M Nazmus Sakib, Mahsin Bin Akram, Raveen Wijewickrama, Igor Bilogrevic, Murtuza Jadliwala
University of Texas at San Antonio · Google LLC
cs.CR
Submitted: 2026-11-04
Updated: 2026-11-04
Code: https://github.com/browser-use/browser-use
License: http://creativecommons.org/licenses/by-sa/4.0/
Importance score: 84/100
The gist: Web tracking studies largely rely on automated crawlers, yet their measurements can differ from what people encounter while browsing Beyond the Landing Page: Website Blocking, Tracker Visibility…
Key concepts
- Agentic Crawlers
- These are advanced crawlers built using browser-use frameworks and SuperScraper. They attempt to navigate websites like a human, often handling consent choices explicitly. They perform better at recovering observed trackers from participants compared to standard non-agentic methods.
- Website Interventions
- These are ways websites block or challenge automated crawlers, such as page denials or hard blocks. The study measured how often these interventions occurred on landing pages versus subsequent pages, finding that some crawlers faced much higher blocking rates later in the browsing process.
- Tracker Visibility Metrics
- This measures how many tracking organizations an automated crawler records compared to what human participants observe. Agentic crawlers recorded 1.5–1.7 times more trackers than non-agentic ones, especially on pages where consent was managed by the website.
- Human Alignment and Agreement
- This assesses how well the tracking data collected by an automated crawler matches what human participants actually see or observe during their browsing sessions. Agentic crawlers showed better recovery of observed trackers, but this did not mean they recorded activity that humans never encountered.
Terminology
Summary
Web tracking studies largely rely on automated crawlers, yet their measurements can differ from what people encounter while browsing Beyond the Landing Page: Website Blocking, Tracker Visibility, and Human Alignment in Automated Web Crawls. p. 0 Beyond the Landing Page: Website Blocking, Tracker Visibility,
and Human Alignment in Automated Web Crawls
A H M Nazmus Sakib1
, Mahsin Bin Akram1
, Raveen Wijewickrama1
, Igor Bilogrevic2
, Murtuza
Jadliwala1
1The University of Texas at San Antonio
2Google LLC
Abstract
Web tracking studies largely rely on automated crawlers, yet their
measurements can differ from what people encounter while browsing. Conventional crawlers follow predefined navigation rules,
whereas LLM-driven agents decide how to navigate and interact
with websites, potentially changing the tracking they encounter. We find that website denials increase beyond
landing pages for standard browser automation, while a detectionresistant automation framework largely prevents this escalation
without increasing recorded tracking. On the same URLs, agentic crawlers record tracking activity from 1. 7 times as many
third-party organizations, particularly on consent-managed pages. Agentic crawlers recover 82–87% of tracker organizations observed
by participants, compared with 58–68% for non-agentic crawlers,
yet roughly half of their own observations are absent from human participant visits.Where the paper says it
Where the paper says it
Where the paper says it
Comparison of Crawler Types
The study compares agentic and non-agentic crawlers across different dimensions to examine how websites respond and what tracking they record. The four non-agentic crawlers use Playwright, a browser automation framework, to drive Chromium or Firefox. p. 0 To address these RQs, we study crawling on 2,000 popular
websites (Figure 1), comparing four non-agentic and two agentic
crawlers. The non-agentic crawlers use Playwright, a browser
automation framework, to drive Chromium or Firefox. Their
stealth variants incorporate browser fingerprint spoofing, with the
Chromium variant also using Patchright, a modified version of
Playwright designed to reduce automation detection. The two agentic crawlers are an agent built on the browser-use framework [7]
and SuperScraper, a research crawler we built that perceives pages
arXiv:2611.Where the paper says it
Where the paper says it
Analysis of Website Interventions
Website interventions are measured using page denials, hard blocks, and challenge markers The denial rates were similar for non-agentic (6.0–8.1%) and agentic crawlers (5. p. 0 Conventional crawlers follow predefined navigation rules,
whereas LLM-driven agents decide how to navigate and interact
with websites, potentially changing the tracking they encounter. We compare agentic and non-agentic crawlers to examine how websites respond to them, what tracking they record, and how closely
their observations match human browsing. Across 2,000 popular
websites, we evaluate different navigation strategies, automation
frameworks, and stealth configurations, alongside browsing data
from 26 participants. We find that website denials increase beyond
landing pages for standard browser automation, while a detectionresistant automation framework largely prevents this escalation
without increasing recorded tracking. Differences emerged on subsequent pages (Figure 2). Among sites with at least six pages per
crawler, Chromium’s denial rate increased from 0. Across 2,000 popular
websites, we evaluate different navigation strategies, automation
frameworks, and stealth configurations, alongside browsing data
from 26 participants. We find that website denials increase beyond
landing pages for standard browser automation, while a detectionresistant automation framework largely prevents this escalation
without increasing recorded tracking. Across 2,000 popular
websites, we evaluate different navigation strategies, automation
frameworks, and stealth configurations, alongside browsing data
from 26 participants. We find that website denials increase beyond
landing pages for standard browser automation, while a detectionresistant automation framework largely prevents this escalation
without increasing recorded tracking. First, website denials are similar across configurations on landing pages but increase on subsequent pages for
browsers driven by unmodified Playwright. Replacing Playwright
with Patchright largely eliminates this escalation, whereas browser
fingerprint spoofing and simulated mouse movement do not.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Tracking Visibility Metrics
On matched URLs, the agentic crawlers record trackers from 1.5–1.7 times as many organizations as non-agentic crawls. p. 1 Second, on matched URLs, the agentic crawlers record trackers from
1. 7 times as many organizations as non-agentic crawls. 7 times as many organizations as non-agentic crawls. This
difference is particularly pronounced on consent-managed pages. This
difference is particularly pronounced on consent-managed pages. The interaction-logging crawl shows that our browser-use agent
frequently accepted consent and that confirmed consent actions
were followed by additional third-party scripts.Where the paper says it
Where the paper says it
Where the paper says it
Human Alignment and Agreement
The agents recover 82–87% of the tracker organizations observed by participants on shared URLs, compared with 58–68% for non-agentic crawlers. However, only roughly half of their own observations are absent from human participant visits. p. 0 7 times as many
third-party organizations, particularly on consent-managed pages. Agentic crawlers recover 82–87% of tracker organizations observed
by participants, compared with 58–68% for non-agentic crawlers,
yet roughly half of their own observations are absent from human participant visits. Agentic crawlers recover 82–87% of tracker organizations observed
by participants, compared with 58–68% for non-agentic crawlers,
yet roughly half of their own observations are absent from human participant visits. These findings show that avoiding website
interventions, increasing tracking visibility, and achieving closer
agreement with human observations do not necessarily go hand in
hand, highlighting the need to select and validate crawlers according to the measurement objective.Where the paper says it
Where the paper says it
Practical Guidelines for Crawler Selection
Crawler choice should follow the question being measured For studies of tracking during ordinary browsing, the visit protocol should reflect the target users’ navigation, observation duration, and consent choices. The paper recommends specific crawler configurations based on measurement objectives such as Interaction-triggered Tracking or Fingerprinting. p. 7 Table 4: Crawler selection and configuration parameters by
measurement objective. Objective Recommended Crawler & Parameters
Interaction-triggered
Tracking
Agentic (direct protocol control). 5 Discussion
Different outcomes require different controls. Our findings
show why crawler quality cannot be reduced to a single measure of human likeness or detection avoidance.Where the paper says it
Where the paper says it
Implications for Agentic Web
Agent frameworks should make consent handling an explicit, user-controllable choice, rather than an incidental consequence of clearing obstacles to navigation. The low later-page denial rates of the agents do not establish that agentic navigation is inherently harder to detect. p. 7 Agent frameworks should therefore make consent handling an explicit, user-controllable choice, rather than an incidental
consequence of clearing obstacles to navigation. For websites, the
low later-page denial rates of our agents do not establish that agentic navigation is inherently harder to detect: Patchright produced
a similar reduction without using an agent, and passive challenge
markers did not track denial rates consistently. They do show how a seemingly minor interaction policy can affect the tracking recorded during delegated
browsing. Agent frameworks should therefore make consent handling an explicit, user-controllable choice, rather than an incidental
consequence of clearing obstacles to navigation. Implications for the agentic Web. These findings also matter
when agents browse on users’ behalf rather than as measurement instruments.Where the paper says it
Where the paper says it
Where the paper says it
Limitations of the Study
The limitations include studying one open model with one framework per agent design and running crawls without a proxy from a single US vantage point. SuperScraper lacks network data on most landing pages, and only the browser-use agent’s consent actions were logged. p. 7 The intervention results
reflect stealth and detection tools at one point in time. SuperScraper
reached fewer pages per site than the other crawlers and lacks network data on most landing pages, and only the browser-use agent’s
consent actions were logged. The participant reference is small,
pooled, and descriptive, with short visits and unrecorded consent
decisions. Page denial is a lower bound on blocking, our heuristics
cannot separate tracking from defensive fingerprinting [8], and the
drift over the three-month collection is bounded in Section 3. The participant reference is small,
pooled, and descriptive, with short visits and unrecorded consent
decisions. Page denial is a lower bound on blocking, our heuristics
cannot separate tracking from defensive fingerprinting [8], and the
drift over the three-month collection is bounded in Section 3.Where the paper says it
Where the paper says it
Where the paper says it
Conclusion
Our findings show that agentic crawling does not uniformly improve Web-tracking measurements. Agents recorded more third-party tracking and recovered more participant-observed trackers, but also observed activity absent from human visits. p. 8 7 Conclusion
Our findings show that agentic crawling does not uniformly improve Web-tracking measurements. Agents recorded more thirdparty tracking and recovered more participant-observed trackers,
but also observed activity absent from human visits. Agents recorded more thirdparty tracking and recovered more participant-observed trackers,
but also observed activity absent from human visits. Their lower
denial rates could largely be reproduced with a patched automation
framework, while browser configurations and instrumentation complicated fingerprinting comparisons. Their lower
denial rates could largely be reproduced with a patched automation
framework, while browser configurations and instrumentation complicated fingerprinting comparisons. These findings demonstrate
that website interventions, tracking visibility, and agreement with
human browsing are distinct measurement objectives. These findings demonstrate
that website interventions, tracking visibility, and agreement with
human browsing are distinct measurement objectives. We provide
practical guidance for selecting and validating crawlers according
to these objectives rather than assuming that any one crawler type
is inherently better.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Reduce website denial escalation by replacing Playwright with Patchright when driving Chromium, as
Replacing Playwright with Patchright alone reduced Chromium’s escalation from +10.1 to +1.8 pp and matched internal-page denial by 6.4 pp.
-
Improve tracking visibility on consent-managed pages by designing agent frameworks to make consent handling an explicit, user-controllable choice rather than an incidental consequence of clearing obstacles, as
Agentic crawlers recorded 1.80–2.16 times as many tracker organizations as participants
on these pages compared to non-agentic crawls. -
Enhance fingerprinting robustness by testing instrumentation in every browser configuration rather than relying only on aggregate prevalence, because
Fingerprinting studies should therefore report vector-level measurements and test instrumentation in every browser configuration, rather than relying only on aggregate prevalence.
-
Optimize agent behavior for multi-page access by ensuring agents crawl multiple internal pages per site to avoid denial escalation, as
Lower later-page denial rates of our agents do not establish that agentic navigation is inherently harder to detect.
-
Improve human agreement measurement by focusing analysis on internal pages where both crawlers and participants observed activity, because
On participant-shared internal pages, participants were denied on at most 0.4% of pages and the agents on none.
Abstract
Web tracking studies largely rely on automated crawlers, yet their measurements can differ from what people encounter while browsing. Conventional crawlers follow predefined navigation rules, whereas LLM-driven agents decide how to navigate and interact with websites, potentially changing the tracking they encounter. We compare agentic and non-agentic crawlers to examine how websites respond to them, what tracking they record, and how closely their observations match human browsing. Across 2,000 popular websites, we evaluate different navigation strategies, automation frameworks, and stealth configurations, alongside browsing data from 26 participants. We find that website denials increase beyond landing pages for standard browser automation, while a detection-resistant automation framework largely prevents this escalation without increasing recorded tracking. On the same URLs, agentic crawlers record tracking activity from 1.5-1.7 times as many third-party organizations, particularly on consent-managed pages. Agentic crawlers recover 82-87% of tracker organizations observed by participants, compared with 58-68% for non-agentic crawlers, yet roughly half of their own observations are absent from human participant visits. These findings show that avoiding website interventions, increasing tracking visibility, and achieving closer agreement with human observations do not necessarily go hand in hand, highlighting the need to select and validate crawlers according to the measurement objective.
Sources
- On the Internet, Nobody Knows You're an LLM Bot: Unmasking Web Agents with Multi-Layer Fingerprinting
- Detecting Bot Detection: Prevalence, Techniques, and Implications for Web Measurement Research
- When Handshakes Tell the Truth: Detecting Web Bad Bots via TLS Fingerprints
- OmniParser for Pure Vision Based GUI Agent
- Known By Their Actions: Fingerprinting LLM Browser Agents via UI Traces
- Qwen3-VL Technical Report
- SoK: Advances and Open Problems in Web Tracking
- FP-Agent: Fingerprinting AI Browsing Agents
Related papers
- From Strategy Search to Policy Learning: MCTS-Guided Adversarial Co-Evolution for LLM Safety Alignment
- Privacy Leakage from Recurrent State in LLM Inference
- Incrementally Verifiable Oracle Aided Computation
- Privilege and confidentiality in generative AI workflows
- BadMoE: Backdooring Mixture-of-Experts LLMs via Routing-Aware Triggers and Task-Underutilized Experts
- CoVault: Secure, Scalable Analytics of Personal Data