Beyond the Landing Page: Website Blocking, Tracker Visibility, and Human Alignment in Automated Web Crawls

arXiv:2611.06197 · cs.CR · Submitted 2026-11-04 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.

Nadia: We're looking at the paper: "Beyond the Landing Page".

Elias: Here's the gist of the paper. Web tracking studies largely rely on automated crawlers, yet their measurements can differ from what people encounter while browsing Beyond the Landing Page: Website Blocking, Tracker Visibility, and Human Alignment in Automated Web Crawls.

Nadia: We find that website denials increase beyond landing pages for standard browser automation, while a detection-resistant automation framework largely prevents this escalation without increasing recorded tracking Agentic crawlers record tracking activity from 1.5–1.7 times as many third-party organizations, particularly on consent-managed pages Agentic crawlers recover 82–87% of tracker organizations observed by participants, compared to 58–68% for non-agentic crawlers.

Elias: Comparison of Crawler Types The study compares agentic and non-agentic crawlers across different dimensions to examine how websites respond and what tracking they record.

Nadia: The four non-agentic crawlers use Playwright, a browser automation framework, to drive Chromium or Firefox.

Paper summary: Nadia: Looking at the paper "Beyond the Landing Page: Website Blocking, Tracker Visibility, and Human Alignment in Automated Web Crawls", it seems like there’s a lot to take away about what automated measurements are actually capturing.

Elias: The authors show that agentic crawlers record more tracking data and recover more participant observed trackers, but they also observe activity absent from human visits >

Priya: So the key finding isn't that agents are automatically superior for measurement; it’s that website interventions, tracking visibility, and agreement with human browsing are distinct things we need to measure separately >

Nadia: That’s right. The denial rates on later pages could mostly be reproduced if you used a patched automation framework instead of relying on the agent's navigation style >

Elias: It confirms that agentic crawling doesn't automatically improve web tracking measurements in every way it seems to do, especially when compared to a standard browser automation setup >

Priya: It’s important for us as researchers to stop assuming any single crawler type is inherently better and instead focus on selecting the right crawler based on the specific measurement we are trying to make >

Nadia: The paper gives practical guidance on choosing crawlers according to what you want to measure, rather than just aiming for human likeness or detection avoidance >

Elias: And for agent frameworks, it suggests making consent handling a clear and user controlled choice, rather than treating it as an accidental consequence of clearing navigation obstacles >

Priya: That frames the issue perfectly: we need to be explicit about how our tools interact with the web and users. It moves us toward more intentional design >

Nadia: So what this means for the world is that understanding these differences helps us build better defenses and more accurate privacy measurements for the web as it changes >

Conclusion: Nadia: So we’re wrapping up our take on "Beyond the Landing Page: Website Blocking, Tracker Visibility, and Human Alignment in Automated Web Crawls."

Elias: Basically, this paper looks at how automated tools like crawlers measure tracking when they interact with a real website.

Priya: The main point is that agentic crawlers might seem better for collecting data, but the results don't tell the whole story.

Nadia: Right. They found that agents record more third-party tracking and recover more trackers seen by people, but they also see things humans never saw.

Elias: That’s because website blocks and tracking visibility aren't all about how sneaky the crawler is being.

Priya: The real lesson here is that you need to measure website blocking, which is a different thing from measuring how much tracking you capture on a page.

Nadia: Exactly. If you want to study someone actually browsing, your crawler setup has to match their actual navigation and consent choices on the site.

Elias: The authors stress that this means we shouldn't just pick one type of crawler—agentic or not—and assume it’s automatically the best measurement tool out there.

Priya: It pushes us toward a more thoughtful approach to privacy research, where we define exactly what kind of interaction we are trying to study first.

Nadia: And for anyone building an agent framework, they need to make consent handling a clear choice for the user, not just something the agent does automatically when it gets stuck.

Elias: Because if you treat that as an afterthought, you might end up measuring tracking in ways that don't reflect what a real user is experiencing.

Priya: This distinction between blocking and visibility is crucial because it shows how measurement design impacts the data itself, and we need to be more precise about those designs going forward.

H M Nazmus Sakib, Mahsin Bin Akram, Raveen Wijewickrama, Igor Bilogrevic, Murtuza Jadliwala

University of Texas at San Antonio · Google LLC

cs.CR

Submitted: 2026-11-04

Updated: 2026-11-04

Code: https://github.com/browser-use/browser-use

License: http://creativecommons.org/licenses/by-sa/4.0/

Importance score: 84/100

The gist: Web tracking studies largely rely on automated crawlers, yet their measurements can differ from what people encounter while browsing Beyond the Landing Page: Website Blocking, Tracker Visibility…

Key concepts

Agentic Crawlers
These are advanced crawlers built using browser-use frameworks and SuperScraper. They attempt to navigate websites like a human, often handling consent choices explicitly. They perform better at recovering observed trackers from participants compared to standard non-agentic methods.
Website Interventions
These are ways websites block or challenge automated crawlers, such as page denials or hard blocks. The study measured how often these interventions occurred on landing pages versus subsequent pages, finding that some crawlers faced much higher blocking rates later in the browsing process.
Tracker Visibility Metrics
This measures how many tracking organizations an automated crawler records compared to what human participants observe. Agentic crawlers recorded 1.5–1.7 times more trackers than non-agentic ones, especially on pages where consent was managed by the website.
Human Alignment and Agreement
This assesses how well the tracking data collected by an automated crawler matches what human participants actually see or observe during their browsing sessions. Agentic crawlers showed better recovery of observed trackers, but this did not mean they recorded activity that humans never encountered.

Terminology

Summary

Web tracking studies largely rely on automated crawlers, yet their measurements can differ from what people encounter while browsing Beyond the Landing Page: Website Blocking, Tracker Visibility, and Human Alignment in Automated Web Crawls. p. 0

Where the paper says it

Beyond the Landing Page: Website Blocking, Tracker Visibility, and Human Alignment in Automated Web Crawls A H M Nazmus Sakib1 , Mahsin Bin Akram1 , Raveen Wijewickrama1 , Igor Bilogrevic2 , Murtuza Jadliwala1 1The University of Texas at San Antonio 2Google LLC Abstract Web tracking studies largely rely on automated crawlers, yet their measurements can differ from what people encounter while browsing. Conventional crawlers follow predefined navigation rules, whereas LLM-driven agents decide how to navigate and interact with websites, potentially changing the tracking they encounter.

We find that website denials increase beyond landing pages for standard browser automation, while a detection-resistant automation framework largely prevents this escalation without increasing recorded tracking Agentic crawlers record tracking activity from 1. p. 0
Where the paper says it

We find that website denials increase beyond landing pages for standard browser automation, while a detectionresistant automation framework largely prevents this escalation without increasing recorded tracking. On the same URLs, agentic crawlers record tracking activity from 1.

5–1.7 times as many third-party organizations, particularly on consent-managed pages Agentic crawlers recover 82–87% of tracker organizations observed by participants, compared to 58–68% for non-agentic crawlers. p. 0
Where the paper says it

7 times as many third-party organizations, particularly on consent-managed pages. Agentic crawlers recover 82–87% of tracker organizations observed by participants, compared with 58–68% for non-agentic crawlers, yet roughly half of their own observations are absent from human participant visits.

Comparison of Crawler Types

The study compares agentic and non-agentic crawlers across different dimensions to examine how websites respond and what tracking they record. The four non-agentic crawlers use Playwright, a browser automation framework, to drive Chromium or Firefox. p. 0

Where the paper says it

To address these RQs, we study crawling on 2,000 popular websites (Figure 1), comparing four non-agentic and two agentic crawlers. The non-agentic crawlers use Playwright, a browser automation framework, to drive Chromium or Firefox.

The two agentic crawlers are an agent built on the browser-use framework and SuperScraper. p. 0
Where the paper says it

Their stealth variants incorporate browser fingerprint spoofing, with the Chromium variant also using Patchright, a modified version of Playwright designed to reduce automation detection. The two agentic crawlers are an agent built on the browser-use framework [7] and SuperScraper, a research crawler we built that perceives pages arXiv:2611.

Analysis of Website Interventions

Website interventions are measured using page denials, hard blocks, and challenge markers The denial rates were similar for non-agentic (6.0–8.1%) and agentic crawlers (5. p. 0

Where the paper says it

Conventional crawlers follow predefined navigation rules, whereas LLM-driven agents decide how to navigate and interact with websites, potentially changing the tracking they encounter. We compare agentic and non-agentic crawlers to examine how websites respond to them, what tracking they record, and how closely their observations match human browsing.

3–8.1%) on landing pages. p. 0
Where the paper says it

Across 2,000 popular websites, we evaluate different navigation strategies, automation frameworks, and stealth configurations, alongside browsing data from 26 participants. We find that website denials increase beyond landing pages for standard browser automation, while a detectionresistant automation framework largely prevents this escalation without increasing recorded tracking.

Differences emerged on subsequent pages, where Chromium’s denial rate increased from 0. p. 4
Where the paper says it

Differences emerged on subsequent pages (Figure 2). Among sites with at least six pages per crawler, Chromium’s denial rate increased from 0.

2% on landing pages to 10. p. 0
Where the paper says it

Across 2,000 popular websites, we evaluate different navigation strategies, automation frameworks, and stealth configurations, alongside browsing data from 26 participants. We find that website denials increase beyond landing pages for standard browser automation, while a detectionresistant automation framework largely prevents this escalation without increasing recorded tracking.

4% on pages 2–6, an escalation of +10. p. 0
Where the paper says it

Across 2,000 popular websites, we evaluate different navigation strategies, automation frameworks, and stealth configurations, alongside browsing data from 26 participants. We find that website denials increase beyond landing pages for standard browser automation, while a detectionresistant automation framework largely prevents this escalation without increasing recorded tracking.

1 pp. Replacing Playwright with Patchright largely eliminates this escalation, whereas browser fingerprint spoofing and simulated mouse movement do not. p. 1
Where the paper says it

First, website denials are similar across configurations on landing pages but increase on subsequent pages for browsers driven by unmodified Playwright. Replacing Playwright with Patchright largely eliminates this escalation, whereas browser fingerprint spoofing and simulated mouse movement do not.

Tracking Visibility Metrics

On matched URLs, the agentic crawlers record trackers from 1.5–1.7 times as many organizations as non-agentic crawls. p. 1

Where the paper says it

Second, on matched URLs, the agentic crawlers record trackers from 1. 7 times as many organizations as non-agentic crawls.

This difference is particularly pronounced on consent-managed pages. p. 1
Where the paper says it

7 times as many organizations as non-agentic crawls. This difference is particularly pronounced on consent-managed pages.

The interaction-logging crawl shows that the browser-use agent frequently accepted consent and that confirmed consent actions were followed by additional third-party scripts. p. 1
Where the paper says it

This difference is particularly pronounced on consent-managed pages. The interaction-logging crawl shows that our browser-use agent frequently accepted consent and that confirmed consent actions were followed by additional third-party scripts.

Human Alignment and Agreement

The agents recover 82–87% of the tracker organizations observed by participants on shared URLs, compared with 58–68% for non-agentic crawlers. However, only roughly half of their own observations are absent from human participant visits. p. 0

Where the paper says it

7 times as many third-party organizations, particularly on consent-managed pages. Agentic crawlers recover 82–87% of tracker organizations observed by participants, compared with 58–68% for non-agentic crawlers, yet roughly half of their own observations are absent from human participant visits.

This shows that avoiding website interventions and increasing tracking visibility do not necessarily go hand in hand. p. 0
Where the paper says it

Agentic crawlers recover 82–87% of tracker organizations observed by participants, compared with 58–68% for non-agentic crawlers, yet roughly half of their own observations are absent from human participant visits. These findings show that avoiding website interventions, increasing tracking visibility, and achieving closer agreement with human observations do not necessarily go hand in hand, highlighting the need to select and validate crawlers according to the measurement objective.

Practical Guidelines for Crawler Selection

Crawler choice should follow the question being measured For studies of tracking during ordinary browsing, the visit protocol should reflect the target users’ navigation, observation duration, and consent choices. The paper recommends specific crawler configurations based on measurement objectives such as Interaction-triggered Tracking or Fingerprinting. p. 7

Where the paper says it

Table 4: Crawler selection and configuration parameters by measurement objective. Objective Recommended Crawler & Parameters Interaction-triggered Tracking Agentic (direct protocol control).

The findings demonstrate that crawler quality cannot be reduced to a single measure of human likeness or detection avoidance. p. 7
Where the paper says it

5 Discussion Different outcomes require different controls. Our findings show why crawler quality cannot be reduced to a single measure of human likeness or detection avoidance.

Implications for Agentic Web

Agent frameworks should make consent handling an explicit, user-controllable choice, rather than an incidental consequence of clearing obstacles to navigation. The low later-page denial rates of the agents do not establish that agentic navigation is inherently harder to detect. p. 7

Where the paper says it

Agent frameworks should therefore make consent handling an explicit, user-controllable choice, rather than an incidental consequence of clearing obstacles to navigation. For websites, the low later-page denial rates of our agents do not establish that agentic navigation is inherently harder to detect: Patchright produced a similar reduction without using an agent, and passive challenge markers did not track denial rates consistently.

Agent frameworks should therefore make consent handling an explicit, user-controllable choice, rather than an incidental consequence of clearing obstacles to navigation. p. 7
Where the paper says it

They do show how a seemingly minor interaction policy can affect the tracking recorded during delegated browsing. Agent frameworks should therefore make consent handling an explicit, user-controllable choice, rather than an incidental consequence of clearing obstacles to navigation.

The findings also matter when agents browse on users’ behalf rather than as measurement instruments. p. 7
Where the paper says it

Implications for the agentic Web. These findings also matter when agents browse on users’ behalf rather than as measurement instruments.

Limitations of the Study

The limitations include studying one open model with one framework per agent design and running crawls without a proxy from a single US vantage point. SuperScraper lacks network data on most landing pages, and only the browser-use agent’s consent actions were logged. p. 7

Where the paper says it

The intervention results reflect stealth and detection tools at one point in time. SuperScraper reached fewer pages per site than the other crawlers and lacks network data on most landing pages, and only the browser-use agent’s consent actions were logged.

Page denial is a lower bound on blocking, as heuristics cannot separate tracking from defensive fingerprinting. p. 7
Where the paper says it

The participant reference is small, pooled, and descriptive, with short visits and unrecorded consent decisions. Page denial is a lower bound on blocking, our heuristics cannot separate tracking from defensive fingerprinting [8], and the drift over the three-month collection is bounded in Section 3.

The drift over the three-month collection is bounded in Section 3. p. 7
Where the paper says it

The participant reference is small, pooled, and descriptive, with short visits and unrecorded consent decisions. Page denial is a lower bound on blocking, our heuristics cannot separate tracking from defensive fingerprinting [8], and the drift over the three-month collection is bounded in Section 3.

2.

Conclusion

Our findings show that agentic crawling does not uniformly improve Web-tracking measurements. Agents recorded more third-party tracking and recovered more participant-observed trackers, but also observed activity absent from human visits. p. 8

Where the paper says it

7 Conclusion Our findings show that agentic crawling does not uniformly improve Web-tracking measurements. Agents recorded more thirdparty tracking and recovered more participant-observed trackers, but also observed activity absent from human visits.

Their lower denial rates could largely be reproduced with a patched automation framework. p. 8
Where the paper says it

Agents recorded more thirdparty tracking and recovered more participant-observed trackers, but also observed activity absent from human visits. Their lower denial rates could largely be reproduced with a patched automation framework, while browser configurations and instrumentation complicated fingerprinting comparisons.

These findings demonstrate that website interventions, tracking visibility, and agreement with human browsing are distinct measurement objectives. p. 8
Where the paper says it

Their lower denial rates could largely be reproduced with a patched automation framework, while browser configurations and instrumentation complicated fingerprinting comparisons. These findings demonstrate that website interventions, tracking visibility, and agreement with human browsing are distinct measurement objectives.

The paper provides practical guidance for selecting and validating crawlers according to these objectives rather than assuming that any one crawler type is inherently better. p. 8
Where the paper says it

These findings demonstrate that website interventions, tracking visibility, and agreement with human browsing are distinct measurement objectives. We provide practical guidance for selecting and validating crawlers according to these objectives rather than assuming that any one crawler type is inherently better.

The single most informative sentence about the paper's subject and finding is: Agentic crawlers record more third-party tracking and recover more participant-observed trackers, but also observe activity absent from human visits.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Reduce website denial escalation by replacing Playwright with Patchright when driving Chromium, as Replacing Playwright with Patchright alone reduced Chromium’s escalation from +10.1 to +1.8 pp and matched internal-page denial by 6.4 pp.

  2. Improve tracking visibility on consent-managed pages by designing agent frameworks to make consent handling an explicit, user-controllable choice rather than an incidental consequence of clearing obstacles, as Agentic crawlers recorded 1.80–2.16 times as many tracker organizations as participants on these pages compared to non-agentic crawls.

  3. Enhance fingerprinting robustness by testing instrumentation in every browser configuration rather than relying only on aggregate prevalence, because Fingerprinting studies should therefore report vector-level measurements and test instrumentation in every browser configuration, rather than relying only on aggregate prevalence.

  4. Optimize agent behavior for multi-page access by ensuring agents crawl multiple internal pages per site to avoid denial escalation, as Lower later-page denial rates of our agents do not establish that agentic navigation is inherently harder to detect.

  5. Improve human agreement measurement by focusing analysis on internal pages where both crawlers and participants observed activity, because On participant-shared internal pages, participants were denied on at most 0.4% of pages and the agents on none.

Abstract

Web tracking studies largely rely on automated crawlers, yet their measurements can differ from what people encounter while browsing. Conventional crawlers follow predefined navigation rules, whereas LLM-driven agents decide how to navigate and interact with websites, potentially changing the tracking they encounter. We compare agentic and non-agentic crawlers to examine how websites respond to them, what tracking they record, and how closely their observations match human browsing. Across 2,000 popular websites, we evaluate different navigation strategies, automation frameworks, and stealth configurations, alongside browsing data from 26 participants. We find that website denials increase beyond landing pages for standard browser automation, while a detection-resistant automation framework largely prevents this escalation without increasing recorded tracking. On the same URLs, agentic crawlers record tracking activity from 1.5-1.7 times as many third-party organizations, particularly on consent-managed pages. Agentic crawlers recover 82-87% of tracker organizations observed by participants, compared with 58-68% for non-agentic crawlers, yet roughly half of their own observations are absent from human participant visits. These findings show that avoiding website interventions, increasing tracking visibility, and achieving closer agreement with human observations do not necessarily go hand in hand, highlighting the need to select and validate crawlers according to the measurement objective.

Sources

Related papers