Revisiting N2DCG: An Empirically Grounded Reformulation of Carousel Recommendation Evaluation
summary
The gist
Carousel interfaces have been widely used in video and music streaming services, yet it remains unclear how to properly evaluate recommender systems in these two-dimensional layouts The authors…
In short
The paper reformulates N2DCG, a metric for evaluating recommender systems in carousel interfaces, to better reflect real user behavior. It addresses two flaws: ignoring category constraints and failing to model mirrored F-pattern examination behavior after horizontal swiping. The new metric, 2DCGref(y), is validated empirically and shows higher accuracy than the original N2DCG.
Key concepts
- N2DCG
- The original metric used to evaluate how well recommendations are laid out in a two-dimensional carousel interface. It was found to be insufficient because it did not account for the specific way users examine content after swiping horizontally.
- Categorical Constraints (Ycat2D)
- This concept addresses the limitation where the original metric treated all positions equally, ignoring that carousels have inherent category structures. The authors restricted the ideal layout to respect these categories, creating a more realistic evaluation space for recommender systems.
- Mirrored F-Pattern Behavior
- This describes a specific user interaction pattern in carousels: after swiping horizontally, users shift their focus to the rightmost position instead of following a standard left-to-right pattern. The new metric incorporates this behavior using specialized discount functions.
Terminology used across episodes
This episode discusses
- Revisiting N2DCG: An Empirically Grounded Reformulation of Carousel Recommendation Evaluation · Paper Radio
- GenPage: Towards End-to-End Generative Homepage Construction at Netflix
The paper
Revisiting N2DCG: An Empirically Grounded Reformulation of Carousel Recommendation Evaluation · Read on arXiv
University of Amsterdam · Brno University of Technology
Carousel interfaces have been widely used in video and music streaming services, yet it remains unclear how to properly evaluate recommender systems in these two-dimensional layouts. N2DCG has been proposed to address this gap by adapting NDCG to carousel-based recommendation, but it relies on unverified assumptions borrowed from the single-list web-search setting that do not transfer well to two-dimensional carousel layouts. We identify two substantial limitations of N2DCG: its ideal ranking, used for normalization, violates carousel constraints, and its discount function does not reflect user browsing behavior observed in empirical data. To address both limitations, we propose a reformulation of N2DCG that normalizes appropriately by respecting constraints and uses an empirically grounded discount function. We validate the proposed metric, showing that it better reflects users' empirical behavior on real-world eye-tracking data and better predicts the comparison results of carousel layouts simulated based on empirical examination patterns.
Transcript
Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.
Tom: We're looking at the paper: "Revisiting N2DCG: An Empirically Grounded Reformulation of Carousel Recommendation Evaluation".
Jane: Here's the gist of the paper. Carousel interfaces have been widely used in video and music streaming services, yet it remains unclear how to properly evaluate recommender systems in these two-dimensional layouts The authors propose a reformulation of N2DCG that normalizes appropriately by respecting constraints and uses an empirically grounded discount function Limitations of the Original Metric The original N2DCG formulation suffers from two substantial limitations in the context of carousel recommendations First, it ignores categorical constraints inherent to carousel interfaces, treating all positions as a single global pool which leads to invalid layouts when items span different categories Second, the discount function fails to model mirrored F-pattern examination behavior observed in user interaction data.
Tom: Proposal for Category-Aware Evaluation To address the categorical constraint issue, the authors propose a category-aware variant where the ideal layout is restricted to respect category structure, defined as Ycat2D.
Jane: This leads to a category-aware I2DCGcat which is the maximum 2DCG over this restricted space.
Tom: Modeling Mirrored F-Pattern Behavior The authors identify a failure of the original N2DCG discount factor to model mirrored F-pattern examination behavior, where attention shifts after horizontal swiping.
Paper summary: Jane: So, wrapping up this discussion on "Revisiting N2DCG: An Empirically Grounded Reformulation of Carousel Recommendation Evaluation." What does this whole paper actually mean for the field?
Tom: It means that for two-dimensional layouts like carousels, we need to be much more careful about how we define and measure recommendation quality. The authors tackled the dual issues of structural constraints and behavioral modeling.
Lu: They showed that by using an empirically grounded discount function, they get a score that is way closer to how a user actually interacts with the interface when they are swiping.
Meng: If we look at it from an engineering standpoint, adopting this reformulated metric means we’re setting a higher bar for what constitutes a good carousel layout. It forces us to think about structure and interaction effort simultaneously during the evaluation phase.
Lalam: The final result is that 2DCGref(y) is presented as the new standard because it seems to capture both the category constraints and that mirrored F-pattern behavior they observed in user data <ref:2608.21877#pg1>.
Tom: So, for anyone working on recommender systems for streaming services, this paper suggests ditching the original N2DCG if you want an accurate evaluation of carousel performance. It’s a concrete step toward building better interfaces.
Conclusion: Tom: So, we’ve looked at how they rebuilt N2DCG for carousels and now we need to talk about what that actually means for recommender systems in general.
Jane: The paper is titled "Revisiting N2DCG: An Empirically Grounded Reformulation of Carousel Recommendation Evaluation." It sounds a bit technical, but they are taking an old metric and making it fit how carousels actually work.
Lu: They took the original N2DCG and fixed two big problems. First, it didn't respect the structure of the carousel—like items belonging to different categories being mixed up.
Meng: And second, it didn't model user behavior correctly. The original discount function just doesn't capture that "mirrored F-pattern" where users look in a different way after they swipe horizontally.
Tom: So what’s the payoff? They propose this new metric, 2DCGref(y), which uses empirically grounded discount functions based on eye-tracking data <ref:2608.21877#pg1>. It’s better at actually measuring what matters for these layouts.
Jane: I think it means that instead of using a rough score, we can get a much more accurate idea of how well our AI is designing those scrolling interfaces for users.
Lu: It suggests that just tweaking the original formula isn't enough; you need to ground those adjustments in real user behavior, which they validated using eye-tracking data on the RecGaze dataset.
Meng: For production, this is good because it gives us a metric that actually reflects interaction patterns we see in real traffic rather than just an abstract mathematical concept.
Tom: Exactly. The conclusion is that this reformulated N2DCG is a much better tool for evaluating carousel recommendations because it accounts for the layout structure and the actual way people look at things while scrolling.
Jane: It suggests that future work should focus on taking this metric-based approach and applying it to simulator-based evaluations, which means testing these models in virtual environments instead of just looking at eye-tracking data.
Lu: That makes sense. Moving from observing behavior to simulating behavior is the natural next step for making these metrics even more robust.
More episodes
- 2505.16254-Human-AI Writing as Distributed Cognition: Reassessing Collaborative Writing Theories in the Age of LLMs
- 2506.10913-Choreographic Quick Changes: First-Class Location (Set) Polymorphism
- 2507.14812-A Black-Box Approach for Exogenous Replenishment in Online Resource Allocation
- 2507.15448-Frames over finite fields with respect to the Galois inner product and their connections to self-orthogonal and LCD codes
- 2508.21572-NewsReX: An Open-Source Multi-Framework for Neural News Recommendation
- 2509.17028-Impact of Packetization on Network Calculus Analysis
- 2510.17342-AoA Services in 5G Networks: A Framework for Real-World Implementation and Systematic Testing
- 2511.04088-Efficient and rate-optimal list-decoding in the presence of minimal feedback
- 2511.17264-Notes on Stack Machines and Quantum Stack Machines
- 2601.15318-On the closest balanced game