Revisiting N2DCG: An Empirically Grounded Reformulation of Carousel Recommendation Evaluation
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.
Tom: We're looking at the paper: "Revisiting N2DCG: An Empirically Grounded Reformulation of Carousel Recommendation Evaluation".
Jane: Here's the gist of the paper. Carousel interfaces have been widely used in video and music streaming services, yet it remains unclear how to properly evaluate recommender systems in these two-dimensional layouts The authors propose a reformulation of N2DCG that normalizes appropriately by respecting constraints and uses an empirically grounded discount function Limitations of the Original Metric The original N2DCG formulation suffers from two substantial limitations in the context of carousel recommendations First, it ignores categorical constraints inherent to carousel interfaces, treating all positions as a single global pool which leads to invalid layouts when items span different categories Second, the discount function fails to model mirrored F-pattern examination behavior observed in user interaction data.
Tom: Proposal for Category-Aware Evaluation To address the categorical constraint issue, the authors propose a category-aware variant where the ideal layout is restricted to respect category structure, defined as Ycat2D.
Jane: This leads to a category-aware I2DCGcat which is the maximum 2DCG over this restricted space.
Tom: Modeling Mirrored F-Pattern Behavior The authors identify a failure of the original N2DCG discount factor to model mirrored F-pattern examination behavior, where attention shifts after horizontal swiping.
Paper summary: Jane: So, wrapping up this discussion on "Revisiting N2DCG: An Empirically Grounded Reformulation of Carousel Recommendation Evaluation." What does this whole paper actually mean for the field?
Tom: It means that for two-dimensional layouts like carousels, we need to be much more careful about how we define and measure recommendation quality. The authors tackled the dual issues of structural constraints and behavioral modeling.
Lu: They showed that by using an empirically grounded discount function, they get a score that is way closer to how a user actually interacts with the interface when they are swiping.
Meng: If we look at it from an engineering standpoint, adopting this reformulated metric means we’re setting a higher bar for what constitutes a good carousel layout. It forces us to think about structure and interaction effort simultaneously during the evaluation phase.
Lalam: The final result is that 2DCGref(y) is presented as the new standard because it seems to capture both the category constraints and that mirrored F-pattern behavior they observed in user data <ref:2608.21877#pg1>.
Tom: So, for anyone working on recommender systems for streaming services, this paper suggests ditching the original N2DCG if you want an accurate evaluation of carousel performance. It’s a concrete step toward building better interfaces.
Conclusion: Tom: So, we’ve looked at how they rebuilt N2DCG for carousels and now we need to talk about what that actually means for recommender systems in general.
Jane: The paper is titled "Revisiting N2DCG: An Empirically Grounded Reformulation of Carousel Recommendation Evaluation." It sounds a bit technical, but they are taking an old metric and making it fit how carousels actually work.
Lu: They took the original N2DCG and fixed two big problems. First, it didn't respect the structure of the carousel—like items belonging to different categories being mixed up.
Meng: And second, it didn't model user behavior correctly. The original discount function just doesn't capture that "mirrored F-pattern" where users look in a different way after they swipe horizontally.
Tom: So what’s the payoff? They propose this new metric, 2DCGref(y), which uses empirically grounded discount functions based on eye-tracking data <ref:2608.21877#pg1>. It’s better at actually measuring what matters for these layouts.
Jane: I think it means that instead of using a rough score, we can get a much more accurate idea of how well our AI is designing those scrolling interfaces for users.
Lu: It suggests that just tweaking the original formula isn't enough; you need to ground those adjustments in real user behavior, which they validated using eye-tracking data on the RecGaze dataset.
Meng: For production, this is good because it gives us a metric that actually reflects interaction patterns we see in real traffic rather than just an abstract mathematical concept.
Tom: Exactly. The conclusion is that this reformulated N2DCG is a much better tool for evaluating carousel recommendations because it accounts for the layout structure and the actual way people look at things while scrolling.
Jane: It suggests that future work should focus on taking this metric-based approach and applying it to simulator-based evaluations, which means testing these models in virtual environments instead of just looking at eye-tracking data.
Lu: That makes sense. Moving from observing behavior to simulating behavior is the natural next step for making these metrics even more robust.
University of Amsterdam · Brno University of Technology
cs.IR, cs.HC
Submitted: 2026-08-22
Updated: 2026-08-22
Journal ref: Proceedings of the 35th ACM International Conference on Information and Knowledge Management (CIKM 2026), Rome, Italy, November 7-11, 2026
Code: https://github.com/jkang98/carousel-metric
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 83/100
The gist: Carousel interfaces have been widely used in video and music streaming services, yet it remains unclear how to properly evaluate recommender systems in these two-dimensional layouts The authors…
Key concepts
- N2DCG
- The original metric used to evaluate how well recommendations are laid out in a two-dimensional carousel interface. It was found to be insufficient because it did not account for the specific way users examine content after swiping horizontally.
- Categorical Constraints (Ycat2D)
- This concept addresses the limitation where the original metric treated all positions equally, ignoring that carousels have inherent category structures. The authors restricted the ideal layout to respect these categories, creating a more realistic evaluation space for recommender systems.
- Mirrored F-Pattern Behavior
- This describes a specific user interaction pattern in carousels: after swiping horizontally, users shift their focus to the rightmost position instead of following a standard left-to-right pattern. The new metric incorporates this behavior using specialized discount functions.
Terminology
Summary
Carousel interfaces have been widely used in video and music streaming services, yet it remains unclear how to properly evaluate recommender systems in these two-dimensional layouts The authors propose a reformulation of N2DCG that normalizes appropriately by respecting constraints and uses an empirically grounded discount function
Limitations of the Original Metric
The original N2DCG formulation suffers from two substantial limitations in the context of carousel recommendations First, it ignores categorical constraints inherent to carousel interfaces, treating all positions as a single global pool which leads to invalid layouts when items span different categories Second, the discount function fails to model mirrored F-pattern examination behavior observed in user interaction data.
Proposal for Category-Aware Evaluation
To address the categorical constraint issue, the authors propose a category-aware variant where the ideal layout is restricted to respect category structure, defined as Ycat2D. This leads to a category-aware I2DCGcat which is the maximum 2DCG over this restricted space. p. 1 Since each category
corresponds to exactly one carousel, Ycat
2𝐷
consists precisely of the
layouts obtained by (i) assigning category-specific carousels to
display rows and (ii) ranking items within each carousel; items
from different categories never share a row. The category-aware
I2DCGcat is the maximum 2DCG over this restricted space:
I2DCGcat = max
𝑦
∗ ∈Ycat
2𝐷
2DCG(𝑦
∗
), (7)
and the category-aware N2DCGcat is defined as 2DCG
I2DCGcat
.Where the paper says it
Modeling Mirrored F-Pattern Behavior
The authors identify a failure of the original N2DCG discount factor to model mirrored F-pattern examination behavior, where attention shifts after horizontal swiping. They propose four novel position-discount formulations, including dmir, which modifies the position discount to account for attention shift after a swipe. p. 2 4 Proposal: Mirrored F-Pattern N2DCG
To capture both mirrored F-pattern attention and the interaction
effort of accessing carousel items, we propose four formulations of
the discount function 𝑑 (𝑖, 𝑗) at row 𝑖 and column 𝑗. We first modify
the position discount to reflect attention shift after a swipe, and
then explore three ways to incorporate user action penalties into
the base discount.Where the paper says it
Empirical Grounding and Validation
The proposed metric is validated using the RecGaze eye-tracking dataset to estimate parameters of the discount functions. The best-performing formulation is found by combining mirrored F-pattern with the row-page discount, yielding higher accuracy than the original metric in simulated carousel layouts The final reformulated 2DCG is presented as 2DCGref(y). p. 3 The best result is
achieved by combining the mirrored F-pattern with the row-page
discount 𝑑𝑅𝑃𝐷 , which yields the highest Spearman’s 𝜌 and Pearson’s
𝑟 correlations and the lowest MSE after max normalization. Accordingly, the next experiment compares the original and reformulated
2DCG using 𝑑𝑎 and 𝑑𝑅𝑃𝐷 , respectivelWhere the paper says it
Conclusion
The reformulation of N2DCG addresses both categorical constraints and misalignment with actual browsing behavior, suggesting that the reformulated N2DCG is a better choice to evaluate carousel recommendation. Future work should extend this metric-based evaluation into behavior-aware simulator-based evaluation. p. 4 This suggests that our reformulated N2DCG is a better
choice to evaluate carousel recommendation. Future work should
extend this metric-based evaluation into behavior-aware simulatorbased evaluation, enabling carousel policies to be evaluated under
explicit models of user browsing and choice behavior.Where the paper says it
--- Page 1 ---
Revisiting N2DCG: An Empirically Grounded Reformulation of Carousel Recommendation Evaluation Jingwei Kang, Santiago de Leon-Martinez, Maarten de Rijke, and Harrie Oosterhuis. 2026. Revisiting N2DCG: An Empirically Grounded Reformulation of Carousel Recommendation Evaluation. p. 0 Revisiting N2DCG: An Empirically Grounded Reformulation
of Carousel Recommendation Evaluation
Jingwei Kang∗
University of Amsterdam
Amsterdam, The Netherlands
j. nl
Santiago de Leon-Martinez∗†
Brno University of Technology
Brno, Czechia
santiago. Revisiting N2DCG: An Empirically Grounded Reformulation of Carousel Recommendation Evaluation. In Proceedings of the 35th
ACM International Conference on Information and Knowledge Management
(CIKM ’26), November 07–11, 2026, Rome, Italy.Where the paper says it
Where the paper says it
Abstract Carousel interfaces have been widely used in video and music streaming services like Netflix [7, 13]. They encourage user exploration by containing multiple lists or rows called carousels, each horizontally swipeable and organized around a single defining theme. p. 0 3839894
1 Introduction: New Insights in Carousels
Carousel interfaces have become ubiquitous in streaming media
services like Netflix [7, 13]. They encourage user exploration by
containing multiple lists or rows called carousels, each horizontally
swipeable and organized around a single defining theme. They encourage user exploration by
containing multiple lists or rows called carousels, each horizontally
swipeable and organized around a single defining theme. Despite
their widespread use, carousels have only recently gained attention
in recommender systems research, with work on user click modeling [3, 12] and on evaluation [5, 6]. Despite
their widespread use, carousels have only recently gained attention
in recommender systems research, with work on user click modeling [3, 12] and on evaluation [5, 6]. However, a recent eye-tracking
study [2] and two follow-up analyses [4, 10] reveal a behavioral
pattern that prior work does not account for. However, a recent eye-tracking
study [2] and two follow-up analyses [4, 10] reveal a behavioral
pattern that prior work does not account for. User examination in
carousels does not follow a single global F-pattern [2, 4, 10]: the
initial page shows a dual-focus F-pattern (with attention peaks at
both the top-left and top-right corners), and after swiping this shifts
to a mirrored F-pattern, with horizontal decay running right-toleft rather than left-to-right. User examination in
carousels does not follow a single global F-pattern [2, 4, 10]: the
initial page shows a dual-focus F-pattern (with attention peaks at
both the top-left and top-right corners), and after swiping this shifts
to a mirrored F-pattern, with horizontal decay running right-toleft rather than left-to-right. This behavior is attributed to users
maintaining their focus on the rightmost position after swiping. This behavior is attributed to users
maintaining their focus on the rightmost position after swiping. However, N2DCG [5, 6], the only evaluation metric for carousels,
does not account for this behavior. However, N2DCG [5, 6], the only evaluation metric for carousels,
does not account for this behavior. Our work addresses this gap by
re-examining and reformulating N2DCG.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
--- Page 2 ---
CIKM ’26, November 07–11, 2026 Jingwei Kang, Santiago de Leon-Martinez, Maarten de Rijke, and Harrie Oosterhuis i-th row and j-th column in a 2D arrangement y: arXiv:2608.21877v1 [cs. p. 0 Let 𝑦𝑖,𝑗 denote the item located at the
arXiv:2608. 21877v1 [cs.Where the paper says it
--- Page 3 ---
Revisiting N2DCG: An Empirically Grounded Reformulation of Carousel Recommendation Evaluation CIKM ’26, November 07–11, 2026 Jingwei Kang, Santiago de Leon-Martinez, Maarten de Rijke, and Harrie Oosterhuis 1.5 10
--- Page 4 ---
CIKM ’26, November 07–11, 2026 Jingwei Kang, Santiago de Leon-Martinez, Maarten de Rijke, and Harrie Oosterhuis P Ei,j = (N(E i,j = 1) / Ntotal), (13)
--- Page 5 ---
CIKM ’26, November 07–11–2026 Jingwei Kang, Santiago de Leon-Martinez, Maarten de Rijke, and Harrie Oosterhuis Table 2: Accuracy comparison between the original and reformulated 2DCG under different ground-truth differences. Setting Diff. p. 4 Validation using empirical examination frequencies and simulated
Table 2: Accuracy comparison between the original and reformulated 2DCG under different ground-truth differences. Setting Diff. Validation using empirical examination frequencies and simulated
Table 2: Accuracy comparison between the original and reformulated 2DCG under different ground-truth differences. Setting Diff. Setting Diff. Δ ≥ Orig. Setting Diff. Δ ≥ Orig.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
--- Page 6 ---
CIKM ’26, November 07–11–2026 Jingwei Kang, Santiago de Leon-Martinez, Maarten de Rijke, and Harrie Oosterhuis Lessons Learned Experiments show the row-page discount dRPD best captures user attention and preferences in carousel interfaces Thus, our reformulated 2DCG with our best found parameters is: 2DCGref(y) = sum i sum j g(y i,j) log2(4i + 9˜j) · 0.651[j>] · 0. p. 4 Thus, our reformulated 2DCG with our best found parameters is:
2𝐷𝐶𝐺ref(𝑦) =
∑︁𝐼
𝑖=1
∑︁
𝐽
𝑗=1
𝑔(𝑦𝑖,𝑗)
log2
(4𝑖 + 9 ˜𝑗)
· 0. 651[𝑗>𝛿 ]
· 0.Where the paper says it
--- Page 7 ---
CIKM ’26, November 07–11–2026 Conclusion We identified two limitations of N2DCG—the neglect of categorical constraints and the misalignment with actual browsing behavior in carousel interfaces—and proposed a reformulation addressing both Validation using empirical examination frequencies and simulated Table 2: Accuracy comparison between the original and reformulated 2DCG under different ground-truth differences. Setting Diff. Δ ≥ Orig. Acc. Our Acc. Corr. Both Err. pairwise comparisons demonstrates the effectiveness of our reformulation This suggests that our reformulated N2DCG is a better choice to evaluate carousel recommendation. p. 4 0%
pairwise comparisons demonstrates the effectiveness of our reformulation. This suggests that our reformulated N2DCG is a better
choice to evaluate carousel recommendation.Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
- Improved Carousel Recommendation Evaluation Metric
The N2DCG is replaced by a reformulated 2DCG metric that accounts for categorical constraints, ensuring that the ideal layout is constrained to respect the category structure of the carousel interface.
This prevents invalid layouts from inflating evaluation scores and ensures comparability across different recommendation scenarios.
- Mirrored F-Pattern Attention Modeling
The position discount function is modified to reflect observed user behavior by replacing the original column index with a re-indexed position ˜j, which captures attention shift after a swipe
leading to a mirrored F-pattern.
This allows the evaluation metric to better predict user preferences between simulated carousel layouts.
- User Action Penalty Integration
The metric incorporates strategies to penalize user interaction effort using four novel discount formulations, including the Row-Page Discount (dRPD) and Multiplicative Swipe Penalties (dmul). This directly models the interaction effort of accessing carousel items
by applying penalties based on the number of horizontal and vertical swipes.
- Category-Aware Layout Optimization
The system can optimize item placement by constraining the ideal layout to respect category boundaries, as defined by Ycat2D = y in Y2D for all i, j, j' where cat(yi,j) = cat(yi,j'). This ensures that the evaluation reflects valid carousel layouts
where items from different categories never share a row.
- Behavior-Grounded Ranking
The recommendation system can be trained to prioritize item placement according to the empirical examination frequencies derived from RecGaze data (Eq. 13). By fitting discount function parameters by maximizing Spearman’s rank correlation with these frequencies, the metric ensures that positions with higher expected attention correctly receive larger discount.
Abstract
Carousel interfaces have been widely used in video and music streaming services, yet it remains unclear how to properly evaluate recommender systems in these two-dimensional layouts. N2DCG has been proposed to address this gap by adapting NDCG to carousel-based recommendation, but it relies on unverified assumptions borrowed from the single-list web-search setting that do not transfer well to two-dimensional carousel layouts. We identify two substantial limitations of N2DCG: its ideal ranking, used for normalization, violates carousel constraints, and its discount function does not reflect user browsing behavior observed in empirical data. To address both limitations, we propose a reformulation of N2DCG that normalizes appropriately by respecting constraints and uses an empirically grounded discount function. We validate the proposed metric, showing that it better reflects users' empirical behavior on real-world eye-tracking data and better predicts the comparison results of carousel layouts simulated based on empirical examination patterns.
Sources
Related papers
- UniRank: Benchmarking Ranking Models for Unified Sequential Modeling and Feature Interaction
- Revisiting Text Ranking in Deep Research
- GateSID: One Gate to Balance Semantic and Collaborative Signals across the Item Lifecycle
- SAM-D2Q: Aligning Multimodal Doc2Query with Search Demand and Conversion for E-commerce
- GINGER: Spicing Up Graph-Based Vector Search
- Subtraction Gets You More: Gap-Aware Retrieval for Multimodal Multi-Hop QA