Revisiting N2DCG: An Empirically Grounded Reformulation of Carousel Recommendation Evaluation

arXiv:2608.21877 · cs.IR, cs.HC · Submitted 2026-08-22 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.

Tom: We're looking at the paper: "Revisiting N2DCG: An Empirically Grounded Reformulation of Carousel Recommendation Evaluation".

Jane: Here's the gist of the paper. Carousel interfaces have been widely used in video and music streaming services, yet it remains unclear how to properly evaluate recommender systems in these two-dimensional layouts The authors propose a reformulation of N2DCG that normalizes appropriately by respecting constraints and uses an empirically grounded discount function Limitations of the Original Metric The original N2DCG formulation suffers from two substantial limitations in the context of carousel recommendations First, it ignores categorical constraints inherent to carousel interfaces, treating all positions as a single global pool which leads to invalid layouts when items span different categories Second, the discount function fails to model mirrored F-pattern examination behavior observed in user interaction data.

Tom: Proposal for Category-Aware Evaluation To address the categorical constraint issue, the authors propose a category-aware variant where the ideal layout is restricted to respect category structure, defined as Ycat2D.

Jane: This leads to a category-aware I2DCGcat which is the maximum 2DCG over this restricted space.

Tom: Modeling Mirrored F-Pattern Behavior The authors identify a failure of the original N2DCG discount factor to model mirrored F-pattern examination behavior, where attention shifts after horizontal swiping.

Paper summary: Jane: So, wrapping up this discussion on "Revisiting N2DCG: An Empirically Grounded Reformulation of Carousel Recommendation Evaluation." What does this whole paper actually mean for the field?

Tom: It means that for two-dimensional layouts like carousels, we need to be much more careful about how we define and measure recommendation quality. The authors tackled the dual issues of structural constraints and behavioral modeling.

Lu: They showed that by using an empirically grounded discount function, they get a score that is way closer to how a user actually interacts with the interface when they are swiping.

Meng: If we look at it from an engineering standpoint, adopting this reformulated metric means we’re setting a higher bar for what constitutes a good carousel layout. It forces us to think about structure and interaction effort simultaneously during the evaluation phase.

Lalam: The final result is that 2DCGref(y) is presented as the new standard because it seems to capture both the category constraints and that mirrored F-pattern behavior they observed in user data <ref:2608.21877#pg1>.

Tom: So, for anyone working on recommender systems for streaming services, this paper suggests ditching the original N2DCG if you want an accurate evaluation of carousel performance. It’s a concrete step toward building better interfaces.

Conclusion: Tom: So, we’ve looked at how they rebuilt N2DCG for carousels and now we need to talk about what that actually means for recommender systems in general.

Jane: The paper is titled "Revisiting N2DCG: An Empirically Grounded Reformulation of Carousel Recommendation Evaluation." It sounds a bit technical, but they are taking an old metric and making it fit how carousels actually work.

Lu: They took the original N2DCG and fixed two big problems. First, it didn't respect the structure of the carousel—like items belonging to different categories being mixed up.

Meng: And second, it didn't model user behavior correctly. The original discount function just doesn't capture that "mirrored F-pattern" where users look in a different way after they swipe horizontally.

Tom: So what’s the payoff? They propose this new metric, 2DCGref(y), which uses empirically grounded discount functions based on eye-tracking data <ref:2608.21877#pg1>. It’s better at actually measuring what matters for these layouts.

Jane: I think it means that instead of using a rough score, we can get a much more accurate idea of how well our AI is designing those scrolling interfaces for users.

Lu: It suggests that just tweaking the original formula isn't enough; you need to ground those adjustments in real user behavior, which they validated using eye-tracking data on the RecGaze dataset.

Meng: For production, this is good because it gives us a metric that actually reflects interaction patterns we see in real traffic rather than just an abstract mathematical concept.

Tom: Exactly. The conclusion is that this reformulated N2DCG is a much better tool for evaluating carousel recommendations because it accounts for the layout structure and the actual way people look at things while scrolling.

Jane: It suggests that future work should focus on taking this metric-based approach and applying it to simulator-based evaluations, which means testing these models in virtual environments instead of just looking at eye-tracking data.

Lu: That makes sense. Moving from observing behavior to simulating behavior is the natural next step for making these metrics even more robust.

University of Amsterdam · Brno University of Technology

cs.IR, cs.HC

Submitted: 2026-08-22

Updated: 2026-08-22

Journal ref: Proceedings of the 35th ACM International Conference on Information and Knowledge Management (CIKM 2026), Rome, Italy, November 7-11, 2026

DOI: 10.1145/3799682.3839894

Code: https://github.com/jkang98/carousel-metric

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 83/100

The gist: Carousel interfaces have been widely used in video and music streaming services, yet it remains unclear how to properly evaluate recommender systems in these two-dimensional layouts The authors…

Key concepts

N2DCG
The original metric used to evaluate how well recommendations are laid out in a two-dimensional carousel interface. It was found to be insufficient because it did not account for the specific way users examine content after swiping horizontally.
Categorical Constraints (Ycat2D)
This concept addresses the limitation where the original metric treated all positions equally, ignoring that carousels have inherent category structures. The authors restricted the ideal layout to respect these categories, creating a more realistic evaluation space for recommender systems.
Mirrored F-Pattern Behavior
This describes a specific user interaction pattern in carousels: after swiping horizontally, users shift their focus to the rightmost position instead of following a standard left-to-right pattern. The new metric incorporates this behavior using specialized discount functions.

Terminology

Summary

Carousel interfaces have been widely used in video and music streaming services, yet it remains unclear how to properly evaluate recommender systems in these two-dimensional layouts The authors propose a reformulation of N2DCG that normalizes appropriately by respecting constraints and uses an empirically grounded discount function

Limitations of the Original Metric

The original N2DCG formulation suffers from two substantial limitations in the context of carousel recommendations First, it ignores categorical constraints inherent to carousel interfaces, treating all positions as a single global pool which leads to invalid layouts when items span different categories Second, the discount function fails to model mirrored F-pattern examination behavior observed in user interaction data.

Proposal for Category-Aware Evaluation

To address the categorical constraint issue, the authors propose a category-aware variant where the ideal layout is restricted to respect category structure, defined as Ycat2D. This leads to a category-aware I2DCGcat which is the maximum 2DCG over this restricted space. p. 1

Where the paper says it

Since each category corresponds to exactly one carousel, Ycat 2𝐷 consists precisely of the layouts obtained by (i) assigning category-specific carousels to display rows and (ii) ranking items within each carousel; items from different categories never share a row. The category-aware I2DCGcat is the maximum 2DCG over this restricted space: I2DCGcat = max 𝑦 ∗ ∈Ycat 2𝐷 2DCG(𝑦 ∗ ), (7) and the category-aware N2DCGcat is defined as 2DCG I2DCGcat .

Modeling Mirrored F-Pattern Behavior

The authors identify a failure of the original N2DCG discount factor to model mirrored F-pattern examination behavior, where attention shifts after horizontal swiping. They propose four novel position-discount formulations, including dmir, which modifies the position discount to account for attention shift after a swipe. p. 2

Where the paper says it

4 Proposal: Mirrored F-Pattern N2DCG To capture both mirrored F-pattern attention and the interaction effort of accessing carousel items, we propose four formulations of the discount function 𝑑 (𝑖, 𝑗) at row 𝑖 and column 𝑗. We first modify the position discount to reflect attention shift after a swipe, and then explore three ways to incorporate user action penalties into the base discount.

Empirical Grounding and Validation

The proposed metric is validated using the RecGaze eye-tracking dataset to estimate parameters of the discount functions. The best-performing formulation is found by combining mirrored F-pattern with the row-page discount, yielding higher accuracy than the original metric in simulated carousel layouts The final reformulated 2DCG is presented as 2DCGref(y). p. 3

Where the paper says it

The best result is achieved by combining the mirrored F-pattern with the row-page discount 𝑑𝑅𝑃𝐷 , which yields the highest Spearman’s 𝜌 and Pearson’s 𝑟 correlations and the lowest MSE after max normalization. Accordingly, the next experiment compares the original and reformulated 2DCG using 𝑑𝑎 and 𝑑𝑅𝑃𝐷 , respectivel

Conclusion

The reformulation of N2DCG addresses both categorical constraints and misalignment with actual browsing behavior, suggesting that the reformulated N2DCG is a better choice to evaluate carousel recommendation. Future work should extend this metric-based evaluation into behavior-aware simulator-based evaluation. p. 4

Where the paper says it

This suggests that our reformulated N2DCG is a better choice to evaluate carousel recommendation. Future work should extend this metric-based evaluation into behavior-aware simulatorbased evaluation, enabling carousel policies to be evaluated under explicit models of user browsing and choice behavior.

--- Page 1 ---

Revisiting N2DCG: An Empirically Grounded Reformulation of Carousel Recommendation Evaluation Jingwei Kang, Santiago de Leon-Martinez, Maarten de Rijke, and Harrie Oosterhuis. 2026. Revisiting N2DCG: An Empirically Grounded Reformulation of Carousel Recommendation Evaluation. p. 0

Where the paper says it

Revisiting N2DCG: An Empirically Grounded Reformulation of Carousel Recommendation Evaluation Jingwei Kang∗ University of Amsterdam Amsterdam, The Netherlands j. nl Santiago de Leon-Martinez∗† Brno University of Technology Brno, Czechia santiago.

In Proceedings of the 35th ACM International Conference on Information and Knowledge Management (CIKM ’26), November 07–11, 2026, Rome, Italy. p. 0
Where the paper says it

Revisiting N2DCG: An Empirically Grounded Reformulation of Carousel Recommendation Evaluation. In Proceedings of the 35th ACM International Conference on Information and Knowledge Management (CIKM ’26), November 07–11, 2026, Rome, Italy.

ACM, New York, NY, USA

Abstract Carousel interfaces have been widely used in video and music streaming services like Netflix [7, 13]. They encourage user exploration by containing multiple lists or rows called carousels, each horizontally swipeable and organized around a single defining theme. p. 0

Where the paper says it

3839894 1 Introduction: New Insights in Carousels Carousel interfaces have become ubiquitous in streaming media services like Netflix [7, 13]. They encourage user exploration by containing multiple lists or rows called carousels, each horizontally swipeable and organized around a single defining theme.

Despite their widespread use, carousels have only recently gained attention in recommender systems research, with work on user click modeling [3, 12] and on evaluation [5, 6]. p. 0
Where the paper says it

They encourage user exploration by containing multiple lists or rows called carousels, each horizontally swipeable and organized around a single defining theme. Despite their widespread use, carousels have only recently gained attention in recommender systems research, with work on user click modeling [3, 12] and on evaluation [5, 6].

However, a recent eye-tracking study [2] and two follow-up analyses [4, 10] reveal a behavioral pattern that prior work does not account for. p. 0
Where the paper says it

Despite their widespread use, carousels have only recently gained attention in recommender systems research, with work on user click modeling [3, 12] and on evaluation [5, 6]. However, a recent eye-tracking study [2] and two follow-up analyses [4, 10] reveal a behavioral pattern that prior work does not account for.

User examination in carousels does not follow a single global F-pattern [2, 4, 10]: the initial page shows a dual-focus F-pattern (with attention peaks at both the top-left and top-right corners), and after swiping this shifts to a mirrored F-pattern, with horizontal decay running right-toleft rather than left-to-right. p. 0
Where the paper says it

However, a recent eye-tracking study [2] and two follow-up analyses [4, 10] reveal a behavioral pattern that prior work does not account for. User examination in carousels does not follow a single global F-pattern [2, 4, 10]: the initial page shows a dual-focus F-pattern (with attention peaks at both the top-left and top-right corners), and after swiping this shifts to a mirrored F-pattern, with horizontal decay running right-toleft rather than left-to-right.

This behavior is attributed to users maintaining their focus on the rightmost position after swiping. p. 0
Where the paper says it

User examination in carousels does not follow a single global F-pattern [2, 4, 10]: the initial page shows a dual-focus F-pattern (with attention peaks at both the top-left and top-right corners), and after swiping this shifts to a mirrored F-pattern, with horizontal decay running right-toleft rather than left-to-right. This behavior is attributed to users maintaining their focus on the rightmost position after swiping.

However, N2DCG [5, 6], the only evaluation metric for carousels, does not account for this behavior. p. 0
Where the paper says it

This behavior is attributed to users maintaining their focus on the rightmost position after swiping. However, N2DCG [5, 6], the only evaluation metric for carousels, does not account for this behavior.

Our work addresses this gap by re-examining and reformulating N2DCG. p. 0
Where the paper says it

However, N2DCG [5, 6], the only evaluation metric for carousels, does not account for this behavior. Our work addresses this gap by re-examining and reformulating N2DCG.

--- Page 2 ---

CIKM ’26, November 07–11, 2026 Jingwei Kang, Santiago de Leon-Martinez, Maarten de Rijke, and Harrie Oosterhuis i-th row and j-th column in a 2D arrangement y: arXiv:2608.21877v1 [cs. p. 0

Where the paper says it

Let 𝑦𝑖,𝑗 denote the item located at the arXiv:2608. 21877v1 [cs.

IR] 22 Aug 2026

--- Page 3 ---

Revisiting N2DCG: An Empirically Grounded Reformulation of Carousel Recommendation Evaluation CIKM ’26, November 07–11, 2026 Jingwei Kang, Santiago de Leon-Martinez, Maarten de Rijke, and Harrie Oosterhuis 1.5 10

--- Page 4 ---

CIKM ’26, November 07–11, 2026 Jingwei Kang, Santiago de Leon-Martinez, Maarten de Rijke, and Harrie Oosterhuis P Ei,j = (N(E i,j = 1) / Ntotal), (13)

--- Page 5 ---

CIKM ’26, November 07–11–2026 Jingwei Kang, Santiago de Leon-Martinez, Maarten de Rijke, and Harrie Oosterhuis Table 2: Accuracy comparison between the original and reformulated 2DCG under different ground-truth differences. Setting Diff. p. 4

Where the paper says it

Validation using empirical examination frequencies and simulated Table 2: Accuracy comparison between the original and reformulated 2DCG under different ground-truth differences. Setting Diff.

p. 4
Where the paper says it

Validation using empirical examination frequencies and simulated Table 2: Accuracy comparison between the original and reformulated 2DCG under different ground-truth differences. Setting Diff.

Δ ≥ Orig. p. 4
Where the paper says it

Setting Diff. Δ ≥ Orig.

p. 4
Where the paper says it

Setting Diff. Δ ≥ Orig.

Acc. Our Acc. Corr. Both Err.

--- Page 6 ---

CIKM ’26, November 07–11–2026 Jingwei Kang, Santiago de Leon-Martinez, Maarten de Rijke, and Harrie Oosterhuis Lessons Learned Experiments show the row-page discount dRPD best captures user attention and preferences in carousel interfaces Thus, our reformulated 2DCG with our best found parameters is: 2DCGref(y) = sum i sum j g(y i,j) log2(4i + 9˜j) · 0.651[j>] · 0. p. 4

Where the paper says it

Thus, our reformulated 2DCG with our best found parameters is: 2𝐷𝐶𝐺ref(𝑦) = ∑︁𝐼 𝑖=1 ∑︁ 𝐽 𝑗=1 𝑔(𝑦𝑖,𝑗) log2 (4𝑖 + 9 ˜𝑗) · 0. 651[𝑗>𝛿 ] · 0.

95i-1 (14)

--- Page 7 ---

CIKM ’26, November 07–11–2026 Conclusion We identified two limitations of N2DCG—the neglect of categorical constraints and the misalignment with actual browsing behavior in carousel interfaces—and proposed a reformulation addressing both Validation using empirical examination frequencies and simulated Table 2: Accuracy comparison between the original and reformulated 2DCG under different ground-truth differences. Setting Diff. Δ ≥ Orig. Acc. Our Acc. Corr. Both Err. pairwise comparisons demonstrates the effectiveness of our reformulation This suggests that our reformulated N2DCG is a better choice to evaluate carousel recommendation. p. 4

Where the paper says it

0% pairwise comparisons demonstrates the effectiveness of our reformulation. This suggests that our reformulated N2DCG is a better choice to evaluate carousel recommendation.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Improved Carousel Recommendation Evaluation Metric

The N2DCG is replaced by a reformulated 2DCG metric that accounts for categorical constraints, ensuring that the ideal layout is constrained to respect the category structure of the carousel interface. This prevents invalid layouts from inflating evaluation scores and ensures comparability across different recommendation scenarios.

  1. Mirrored F-Pattern Attention Modeling

The position discount function is modified to reflect observed user behavior by replacing the original column index with a re-indexed position ˜j, which captures attention shift after a swipe leading to a mirrored F-pattern. This allows the evaluation metric to better predict user preferences between simulated carousel layouts.

  1. User Action Penalty Integration

The metric incorporates strategies to penalize user interaction effort using four novel discount formulations, including the Row-Page Discount (dRPD) and Multiplicative Swipe Penalties (dmul). This directly models the interaction effort of accessing carousel items by applying penalties based on the number of horizontal and vertical swipes.

  1. Category-Aware Layout Optimization

The system can optimize item placement by constraining the ideal layout to respect category boundaries, as defined by Ycat2D = y in Y2D for all i, j, j' where cat(yi,j) = cat(yi,j'). This ensures that the evaluation reflects valid carousel layouts where items from different categories never share a row.

  1. Behavior-Grounded Ranking

The recommendation system can be trained to prioritize item placement according to the empirical examination frequencies derived from RecGaze data (Eq. 13). By fitting discount function parameters by maximizing Spearman’s rank correlation with these frequencies, the metric ensures that positions with higher expected attention correctly receive larger discount.

Abstract

Carousel interfaces have been widely used in video and music streaming services, yet it remains unclear how to properly evaluate recommender systems in these two-dimensional layouts. N2DCG has been proposed to address this gap by adapting NDCG to carousel-based recommendation, but it relies on unverified assumptions borrowed from the single-list web-search setting that do not transfer well to two-dimensional carousel layouts. We identify two substantial limitations of N2DCG: its ideal ranking, used for normalization, violates carousel constraints, and its discount function does not reflect user browsing behavior observed in empirical data. To address both limitations, we propose a reformulation of N2DCG that normalizes appropriately by respecting constraints and uses an empirically grounded discount function. We validate the proposed metric, showing that it better reflects users' empirical behavior on real-world eye-tracking data and better predicts the comparison results of carousel layouts simulated based on empirical examination patterns.

Sources

Related papers