Beyond the Landing Page: Website Blocking, Tracker Visibility, and Human Alignment in Automated Web Crawls
summary
The gist
Web tracking studies largely rely on automated crawlers, yet their measurements can differ from what people encounter while browsing Beyond the Landing Page: Website Blocking, Tracker Visibility…
In short
The study compared agentic and non-agentic web crawlers to see how they interact with websites and record tracking data. Findings show agentic crawlers record more third-party trackers, but also observe activity humans don't see. Website blocking rates were similar initially, but later pages showed significant differences based on the crawler type.
Key concepts
- Agentic Crawlers
- These are advanced crawlers built using browser-use frameworks and SuperScraper. They attempt to navigate websites like a human, often handling consent choices explicitly. They perform better at recovering observed trackers from participants compared to standard non-agentic methods.
- Website Interventions
- These are ways websites block or challenge automated crawlers, such as page denials or hard blocks. The study measured how often these interventions occurred on landing pages versus subsequent pages, finding that some crawlers faced much higher blocking rates later in the browsing process.
- Tracker Visibility Metrics
- This measures how many tracking organizations an automated crawler records compared to what human participants observe. Agentic crawlers recorded 1.5–1.7 times more trackers than non-agentic ones, especially on pages where consent was managed by the website.
- Human Alignment and Agreement
- This assesses how well the tracking data collected by an automated crawler matches what human participants actually see or observe during their browsing sessions. Agentic crawlers showed better recovery of observed trackers, but this did not mean they recorded activity that humans never encountered.
Terminology used across episodes
This episode discusses
- Beyond the Landing Page: Website Blocking, Tracker Visibility, and Human Alignment in Automated Web Crawls · Paper Radio
- On the Internet, Nobody Knows You're an LLM Bot: Unmasking Web Agents with Multi-Layer Fingerprinting
- Detecting Bot Detection: Prevalence, Techniques, and Implications for Web Measurement Research
- When Handshakes Tell the Truth: Detecting Web Bad Bots via TLS Fingerprints
- OmniParser for Pure Vision Based GUI Agent
- Known By Their Actions: Fingerprinting LLM Browser Agents via UI Traces · Paper Radio
- Qwen3-VL Technical Report
- SoK: Advances and Open Problems in Web Tracking · Paper Radio
- FP-Agent: Fingerprinting AI Browsing Agents
The paper
Beyond the Landing Page: Website Blocking, Tracker Visibility, and Human Alignment in Automated Web Crawls · Read on arXiv
H M Nazmus Sakib, Mahsin Bin Akram, Raveen Wijewickrama, Igor Bilogrevic, Murtuza Jadliwala
University of Texas at San Antonio · Google LLC
Web tracking studies largely rely on automated crawlers, yet their measurements can differ from what people encounter while browsing. Conventional crawlers follow predefined navigation rules, whereas LLM-driven agents decide how to navigate and interact with websites, potentially changing the tracking they encounter. We compare agentic and non-agentic crawlers to examine how websites respond to them, what tracking they record, and how closely their observations match human browsing. Across 2,000 popular websites, we evaluate different navigation strategies, automation frameworks, and stealth configurations, alongside browsing data from 26 participants. We find that website denials increase beyond landing pages for standard browser automation, while a detection-resistant automation framework largely prevents this escalation without increasing recorded tracking. On the same URLs, agentic crawlers record tracking activity from 1.5-1.7 times as many third-party organizations, particularly on consent-managed pages. Agentic crawlers recover 82-87% of tracker organizations observed by participants, compared with 58-68% for non-agentic crawlers, yet roughly half of their own observations are absent from human participant visits. These findings show that avoiding website interventions, increasing tracking visibility, and achieving closer agreement with human observations do not necessarily go hand in hand, highlighting the need to select and validate crawlers according to the measurement objective.
Transcript
Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.
Nadia: We're looking at the paper: "Beyond the Landing Page".
Elias: Here's the gist of the paper. Web tracking studies largely rely on automated crawlers, yet their measurements can differ from what people encounter while browsing Beyond the Landing Page: Website Blocking, Tracker Visibility, and Human Alignment in Automated Web Crawls.
Nadia: We find that website denials increase beyond landing pages for standard browser automation, while a detection-resistant automation framework largely prevents this escalation without increasing recorded tracking Agentic crawlers record tracking activity from 1.5–1.7 times as many third-party organizations, particularly on consent-managed pages Agentic crawlers recover 82–87% of tracker organizations observed by participants, compared to 58–68% for non-agentic crawlers.
Elias: Comparison of Crawler Types The study compares agentic and non-agentic crawlers across different dimensions to examine how websites respond and what tracking they record.
Nadia: The four non-agentic crawlers use Playwright, a browser automation framework, to drive Chromium or Firefox.
Paper summary: Nadia: Looking at the paper "Beyond the Landing Page: Website Blocking, Tracker Visibility, and Human Alignment in Automated Web Crawls", it seems like there’s a lot to take away about what automated measurements are actually capturing.
Elias: The authors show that agentic crawlers record more tracking data and recover more participant observed trackers, but they also observe activity absent from human visits >
Priya: So the key finding isn't that agents are automatically superior for measurement; it’s that website interventions, tracking visibility, and agreement with human browsing are distinct things we need to measure separately >
Nadia: That’s right. The denial rates on later pages could mostly be reproduced if you used a patched automation framework instead of relying on the agent's navigation style >
Elias: It confirms that agentic crawling doesn't automatically improve web tracking measurements in every way it seems to do, especially when compared to a standard browser automation setup >
Priya: It’s important for us as researchers to stop assuming any single crawler type is inherently better and instead focus on selecting the right crawler based on the specific measurement we are trying to make >
Nadia: The paper gives practical guidance on choosing crawlers according to what you want to measure, rather than just aiming for human likeness or detection avoidance >
Elias: And for agent frameworks, it suggests making consent handling a clear and user controlled choice, rather than treating it as an accidental consequence of clearing navigation obstacles >
Priya: That frames the issue perfectly: we need to be explicit about how our tools interact with the web and users. It moves us toward more intentional design >
Nadia: So what this means for the world is that understanding these differences helps us build better defenses and more accurate privacy measurements for the web as it changes >
Conclusion: Nadia: So we’re wrapping up our take on "Beyond the Landing Page: Website Blocking, Tracker Visibility, and Human Alignment in Automated Web Crawls."
Elias: Basically, this paper looks at how automated tools like crawlers measure tracking when they interact with a real website.
Priya: The main point is that agentic crawlers might seem better for collecting data, but the results don't tell the whole story.
Nadia: Right. They found that agents record more third-party tracking and recover more trackers seen by people, but they also see things humans never saw.
Elias: That’s because website blocks and tracking visibility aren't all about how sneaky the crawler is being.
Priya: The real lesson here is that you need to measure website blocking, which is a different thing from measuring how much tracking you capture on a page.
Nadia: Exactly. If you want to study someone actually browsing, your crawler setup has to match their actual navigation and consent choices on the site.
Elias: The authors stress that this means we shouldn't just pick one type of crawler—agentic or not—and assume it’s automatically the best measurement tool out there.
Priya: It pushes us toward a more thoughtful approach to privacy research, where we define exactly what kind of interaction we are trying to study first.
Nadia: And for anyone building an agent framework, they need to make consent handling a clear choice for the user, not just something the agent does automatically when it gets stuck.
Elias: Because if you treat that as an afterthought, you might end up measuring tracking in ways that don't reflect what a real user is experiencing.
Priya: This distinction between blocking and visibility is crucial because it shows how measurement design impacts the data itself, and we need to be more precise about those designs going forward.
More episodes
- 2608.27496-ROPE: Routed Origin Policy Enforcement against Indirect Prompt Injection
- Daily Summary for 2026-11-19
- 2607.07085-Is Randomness Necessary for Adaptive Data Analysis?
- The day in conversation for 2026-11-19
- Daily Summary for 2026-11-19
- 2609.10601-Threshold Choice, Not Sample Size, Bounds Trustless Verification of Nondeterministic Compound AI Workflows
- 2610.38786-Adversarial Process Disturbances at a Deployable Alarm Budget: Detector Evaluation on a Simulator and a Real Plant
- 2511.08660-Binary and Multiclass Cyberattack Classification on GeNIS Dataset
- 2604.27302-Static Attribution of Android Residential Proxy Malware Using Graph Kernels
- Daily Summary for 2026-11-18