Machine Learning to Predict Digital Frustration from Clickstream Data

arXiv:2512.20438 · cs.LG · Submitted 2025-12-23 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Machine Learning to Predict Digital Frustration from Clickstream Data".

Jane: Here's the gist of the paper. Machine Learning to Predict Digital Frustration from Clickstream Data is a research effort to build a machine learning framework that can identify digital frustration from clickstream data using various models and sequence features The LSTM classifier performs very well, reaching about 91% accuracy and a ROC AUC of 0.97 on the test set Defining Frustration Signals The research defines user frustration by combining several interaction signals such as rage bursts, back-and-forth navigation (U-turns), cart churn, search struggle, and long wandering sessions The rules for labeling a session as frustrated include: 1.

Tom: Rage burst: Repeated clicks on the same element in a very short time window 2.

Jane: U-turns: Shows frustration when the user goes to a page and very quickly comes back 3.

Tom: Cart churn: The user adds an item to the cart, removes it, and never purchases anything in that session 4.

Title and authors: Tom: So we're looking at this paper today, "Machine Learning to Predict Digital Frustration from Clickstream Data." Basically, it uses data from a real e-commerce site to figure out if a user is getting frustrated while they're trying to do something on the website.

Jane: And the authors are Jibin Joseph and his team at the University of Texas at Austin. It sounds like they've looked at how users behave online and tried to build a system to flag those frustrating moments before they cause real problems for businesses.

Lu: The core idea here is taking raw clickstream data, which is just a record of every page a user visits and every action they take, and using that to predict frustration. It’s about turning messy interaction history into something we can actually measure.

Meng: So what does frustration even mean in this context? Is it just when someone clicks the wrong thing? We need to know how they define those signals before we look at the math.

Jane: Well, the paper defines frustration using a few specific signals, like rage bursts where a user clicks on the same thing over and over in a very short time.

Tom: And they also look at back-and-forth navigation, which is what they call U-turns, where someone goes to one page and then comes right back quickly.

Lu: They also track cart churn, which happens when a user adds something to their shopping cart but then takes it out without ever finishing the purchase in that session.

Meng: That sounds like a direct hit on lost sales for any online store. So what else do they look at besides those specific things?

Jane: They also look at search struggle, which is when a user searches multiple times but doesn't find what they need, and long wandering sessions where someone spends too much time looking around without adding anything to their cart.

Tom: It sounds like they’re building a comprehensive set of rules for flagging frustration, which is important because it gives you something concrete to work with when you start training models.

The paper's summary: Tom: Now, let's get into the actual results described in this research on "Machine Learning to Predict Digital Frustration from Clickstream Data." They used five point four million raw clickstream events across three hundred four thousand eight hundred eighty-one sessions for their study <ref:2512.20438#pg1,5.4 million raw clickstream events>.

Jane: So, they didn't just look at a few examples; they applied these frustration rules to a really big amount of data to see how the models actually perform in practice.

Lu: They focused on building a pipeline that takes that raw data, turns it into different kinds of features—behavioral, statistical, and sequential ones—and then trains both standard machine learning models and deep learning sequence models.

Meng: What kind of features are we talking about here? Are they just basic counts of how many times a page was seen? I'm worried about running a model on that if the patterns are really complex.

Tom: They engineer several types of features, including N-gram features, which count how often a specific page appears in the user's sequence or how often someone moves from viewing a page to looking at its details.

Jane: And they also use something called horizontal visibility graph motifs, or HVG motifs. That’s more complex than just counting occurrences; it tries to capture local patterns of how pages are connected in the user's journey.

Lu: Those HVG motifs are designed to capture more intricate local patterns beyond simple frequency counts, which should give the standard classifiers a better view of the interaction structure, and they also use cyclical features for things like hour-of-day and day-of-week information.

Meng: So they're feeding these engineered tabular features into traditional classifiers like Logistic Regression or Random Forest to get a baseline performance number. That’s how you start, right?

Tom: Exactly, and then they compare that baseline to a discriminative Long Short-Term Memory model, which is the deep learning sequence model. And the results show that this LSTM classifier performs best overall.

The paper's improvements: Jane: So what’s actually impressive about their findings on "Machine Learning to Predict Digital Frustration from Clickstream Data"? It’s not just that they built a model; it’s how much better the deep learning approach is compared to the traditional methods.

Lu: The key improvement they highlight is the performance gain you get by using a discriminative LSTM classifier instead of just relying on those engineered tabular features alone.

Tom: And the numbers back up that: XGBoost, which is a very strong standard model, hits about ninety percent accuracy and an ROC AUC of zero point nine five seven nine, but the LSTM classifier manages about ninety-one percent accuracy and an ROC AUC of zero point nine seven zero five on the test set <ref:2512.20438#pg1,ROC AUC of 0.9579>.

Meng: Ninety percent is a solid number for a baseline model, but that difference to ninety-one percent with the LSTM suggests sequence modeling actually adds something substantial here for detecting frustration.

Jane: And they also looked at when you can get reliable predictions early in the user's session, which is a really practical question for anyone building a real-time system.

Tom: The paper found that while accuracy is pretty low for the first five interactions, it starts getting strong around twenty interactions. Specifically, at twenty interactions, the F1 score jumps to about zero point eight seven and the ROC AUC reaches zero point nine four.

Lu: That suggests that by focusing on just those first thirty interactions of a session, you can get results that are almost as good as looking at the entire sequence of events. It points toward a way to make these predictions faster and more immediate in a live session setting.

Jane: So, what does this mean for the practical application? It means we might not need to wait for the user to finish their entire journey before we can start intervening.

Conclusion: Tom: To wrap up this look at "Machine Learning to Predict Digital Frustration from Clickstream Data," the paper shows that combining engineered features with a deep learning sequence model gives you a very solid baseline for detecting frustration.

Jane: It confirms that the LSTM is the better performer, hitting about ninety-one percent accuracy and an ROC AUC of zero point nine seven zero five, which is noticeably higher than XGBoost's result of about ninety percent accuracy and an ROC AUC of zero point nine five seven nine <ref:2512.20438#pg1,ROC AUC of 0.9705>.

Lu: The main implication for me as a researcher is that utilizing the full event sequence with a discriminative LSTM provides that consistent uplift over the tabular feature approach, even though we see strong performance starting much earlier in the session.

Meng: From my side at startup, I’m thinking about deployment; if you can get reliable predictions based on just twenty interactions, that opens up possibilities for proactive help offering during a live session. That's where it gets useful for us in production environments.

Lalam: I'm fairly confident in this direction because the models they built are showing strong predictive power across their testing data, and the early window prediction is a concrete piece of evidence that you can trust when building real-time systems.

Department of Computer Science, The University of Texas at Austin

cs.LG

Submitted: 2025-12-23

Updated: 2026-11-10

Comments: 17 pages, 5 figures. v2: Minor corrections to methodological descriptions, references, and language. Added code availability information. Experimental results and conclusions remain unchanged

Project page: https://www.cloudflare.com/learning/performance/more

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 91/100

The gist: Machine Learning to Predict Digital Frustration from Clickstream Data is a research effort to build a machine learning framework that can identify digital frustration from clickstream data using…

Key concepts

Rage Burst
This occurs when a user repeatedly clicks on the exact same element in a very short period. It is defined as a strong signal of immediate digital frustration, indicating the user is struggling with an element or action on the screen.
U-turns (Back-and-forth Navigation)
A U-turn happens when a user navigates to a specific page and then very quickly returns to the previous page. This rapid switching suggests confusion or difficulty in finding what they are looking for, which is flagged as a frustration signal.
Feature Engineering (N-gram & HVG Motifs)
Researchers created advanced features from click sequences, such as N-grams (tracking how often pages appear sequentially) and Horizontal Visibility Graph Motifs. These help standard models understand complex local patterns in user behavior that simple counts would miss.
Early Window Prediction
The study found that the LSTM model becomes highly reliable for predicting frustration after only 20 to 30 interactions. This means the system can provide accurate predictions very early in a user's session, allowing for proactive support before frustration escalates.

Terminology

Summary

Machine Learning to Predict Digital Frustration from Clickstream Data is a research effort to build a machine learning framework that can identify digital frustration from clickstream data using various models and sequence features The LSTM classifier performs very well, reaching about 91% accuracy and a ROC AUC of 0. p. 197 on the test set

Defining Frustration Signals

The research defines user frustration by combining several interaction signals such as rage bursts, back-and-forth navigation (U-turns), cart churn, search struggle, and long wandering sessions The rules for labeling a session as frustrated include:

  1. Rage burst: Repeated clicks on the same element in a very short time window

  2. U-turns: Shows frustration when the user goes to a page and very quickly comes back

  3. Cart churn: The user adds an item to the cart, removes it, and never purchases anything in that session

  4. Search struggle: If the user does a lot of searches with no success

  5. Long wandering: If the session duration exceeds twenty minutes and the user has visited five or more product detail pages, and there is no add/purchase

Feature Engineering for Models

The paper engineers tabular features from sequential data to feed into standard classifiers The feature engineering methods applied include:

  1. N-gram (k-gram) features: These capture how often a page appears in the user’s sequence and how often the user goes from view → detail or detail → add

  2. Horizontal visibility graph motifs (HVG motifs): These are used to capture more complex local patterns beyond simple frequency counts

  3. Cyclical features: These capture the seasonality of the sessions using sine and cosine transformations for hour-of-day and day-of-week information

Model Performance Comparison

The study compares three standard classifiers (Logistic Regression, Random Forest, XGBoost) with a discriminative LSTM classifier The results show that XGBoost achieves about 90% accuracy and a ROC AUC of 0.9579 The discriminative LSTM classifier performs best, reaching about 91% accuracy and a ROC AUC of 0. p. 09705

Early Window Prediction Insights

The research investigates how early in the user’s session a reliable prediction can be made The performance of the LSTM classifier shows that while accuracy is low for the first 5 events, it becomes strong at 20 interactions, reaching an F1-score of 0.87 and a ROC AUC of 0. p. 094 This indicates that by using the first 30 interactions, the results become almost as good as using the full interaction sequence

Conclusion on Predictive Power

The findings demonstrate that standard models with engineered features provide a very good baseline for frustration detection The LSTM model, by utilizing the full event sequence, provides a small but consistent improvement over the baseline The most useful predictive signal appears within the first 20–30 interactions of a session

Limitations and Future Work

A limitation noted is that the models were trained on a balanced dataset, and no results are reported on the original imbalanced data Furthermore, dedicated early-window models trained only on the first N events could further improve early frustration detection Dedicated early-window models trained only on the first N events, and features could further improve early frustration detection The paper also notes that transformer-based approaches are available but were not used in this research

References

[1] Editorial Staff: E-Commerce Industry Revenue and Growth Statistics (2023). https://boostertheme.com/learn/e-commerce-stats. p. 3 Booster Theme blog, accessed 30 November 2025. p. 0 (2023)

[2] Cramer-Flood, E.: Global Retail Ecommerce Forecast 2023. p. 0 https://www.emarketer.com/content/global-retail-ecommerce-forecast-2023. p. 0 Insider Intelligence / eMarketer report, accessed 30 November 2025. p. 0 (2023)

[3] Bennett, R.: Digital banking trends in 2025. p. 0 https://www.bankrate.com/banking/digital-banking-trends-and-statistics/. p. 0 Bankrate article, accessed 30 November 2025. p. 0 (2025)

[4] Burnett, S.: Online Banking Usage Statistics 2025: Adoption, User Insights, and Security Innovations. p. 0 https://coinlaw.io/online-banking-usage-statistics/. p. 0 CoinLaw article, accessed 30 November 2025. p. 0 (2025)

[5] Suneja, K.: 85.5% Indian households posses at least one smartphone, 99. p. 05% youth use UPI: MoSPI survey. p. 0 https://economictimes.indiatimes.com/news/economy/indicators/85-5-indian-households-posses-at-least-one smartphone-99-5 youth use UPI: MoSPI survey/articleshow/121495764. p. 0cms. The Economic Times, accessed 30 November 2025. p. 1 (2025)

[6] Contentsquare: What is digital experience: understanding the concept. https://contentsquare. p. 1 p. 1com/blog/what-is-a-digital-experience/. p. 0 Contentsquare blog, accessed 30 November 2025. p. 1 (2023)

[7] Neuner, H.: Focus Cart Abandonment: First semi-annual report 2025. p. 0 https://uptain.de/en/blog/e-report-cart-abandonment-first-semi-annual report 2025/. p. 0 uptain blog, accessed 30 November 2025. p. 0 (2025)

[8] Cloudflare: How website performance affects conversion rates. https://www.cloudflare. p. 0com/learning/performance/more/website-performance-conversion-rates/. p. 7 Cloudflare Learning Center, accessed 30 November 2025. p. 0 (2025)

[9] Pujara, A.: Transform customer digital experience with AIDE. p. 1 https://fractal. p. 1ai/blog/transform-customer-digital-experience-with-aide. p. 1 Fractal Analytics blog, accessed 30 November 2025. p. 1 (2023)

[10] Fullstory, Inc.: Rage Clicks, Error Clicks, Dead Clicks, and Thrashed Cursor: Frustration Signals. p. 1 https://help. p. 1fullstory.com/hc/en-us/articles/360020624154-Rage-Clicks-Error-Clicks-Dead-Clicks-and-Thrashed Cursor Frustration Signals. p. 1 Fullstory Help Center article, accessed 30 November 2025. p. 1 (2025)

[11] Contentsquare: Frustration score. https://contentsquare.com/platform/capabilities/frustration-score/. p. 1 Contentsquare platform capability page, accessed 30 November 2025. p. 1 (2023)

[12] Glassbox: Struggle and Error Analysis. https://www.glassbox.com/platform/struggle-error-analysis/. p. 1 Glassbox platform page, accessed 30 November 2025. p. 1 (2025)

[13] Quantum Metric: Experience Alerts. https://www.quantummetric.com/platform/experience-alerts. p. 1 Quantum Metric platform page, accessed 30 November 2025. p. 1 (2025)

[14] Wei, J., Shen, Z., Sundaresan, N., Ma, K.-L.: Visual cluster exploration of web clickstream data. p. 1 In: 2012 IEEE Conference on Visual Analytics Science and Technology (VAST), pp. p. 1 3–12. IEEE, Seattle, WA, USA (2012)

[15] Wang, G., Zhang, X., Tang, S., Wilson, C., Zheng, H., Zhao, B.Y.: Clickstream user behavior models. p. 1 ACM Transactions on the Web 11(4) (2017). p. 1 Article 21;

[16] Requena, B., Cassani, G., Tagliabue, J., Greco, C., Lacasa, L. p. 5 p. 5 p. 5 p. 5: Shopper intent prediction from clickstream e-commerce data with minimal browsing information. p. 1 Scientific Reports 10, 16983 (2020) https://doi. p. 1org/10.1038/s41598-020-73622-y

[17] Tagliabue, J.e.a.: SIGIR 2021 E-Commerce Workshop Data Challenge: Insession Prediction for Purchase Intent and Recommendations. p. 5 arXiv preprint arXiv:2104. p. 009423. Coveo e-commerce dataset (2021)

[18] Bakker, J.: Predicting shopping intention on sequential data using different recurrent neural network algorithms. p. 1 Master’s thesis, Tilburg University, Tilburg, The Netherlands (2020). p. 0 Master’s thesis

[19] Al-Tayeb, Y.: Predicting consumer behavior in online shopping using clickstream data and machine learning algorithms. p. 1 Master’s thesis, Tilburg University, Tilburg, Netherlands (2024). p. 0 Master’s thesis

[20] Yadav, S.: Prediction of online shopper’s buying intention using algorithms of pyspark mllib. p. 1 International Journal of Current Science 13(2), 61–70 (2023)

[21] Baati, K., Mohsil, M.: Real-time prediction of online shoppers’ purchasing intention using random forest. p. 6 In: Artificial Intelligence Applications and Innovations. p. 0 IFIP Advances in Information and Communication Technology, vol. p. 1 583, pp. 43–51. Springer, Cham, Switzerland (2020). p. 1 https://doi.org/10.1007/978-3-030-49161-1

[22] M. Anitha, K.B., Ramya, M.Z.: Predicting online shopping behavior through clickstream analysis. p. 1 International Journal of Advanced Research in Science, Technology and Engineering 14(5) (2024). p. 2

[23] Nghia, L. p. 1T.M.: Real-time prediction of online shoppers’ purchase intent. p. 5 Asian Journal of Management and Commerce (2024). p. 1

[24] Ulitzsch, E. p. 2, Ulitzsch, V. p. 2, He, Q., L¨udtke, O.: A machine learning-based procedure for leveraging clickstream data to investigate early predictability of failure on interactive tasks. p. 2 Behavior Research Methods 55(3), 1392–1412 (2023). p. 2

[25] Wang, Y. p. 1, Zhang, H., Gu, H., Yi, L., Shen, H., Cheng, X.: Cubc: A generalized representation learning method for user behavioral sequence. p. 1 In: NeurIPS 2021 Workshop on Self-Supervised Learning (2021). p. 2 NeurIPS 2021 Workshop paper;

[26] Wang, Z., Wu, Q., Zheng, B., Wang, J., Huang, K., Shi, Y.: Sequence as genes: An user behavior modeling framework for fraud transaction detection in e-commerce. p. 2 In: Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 5194–5203. ACM, New York, NY, USA (2023). p. 1 https://doi.org/10.1145/3580305.3599905

[27] Sanni, B.: Anomaly Detection and User Frustration Prediction Using Machine Learning in Mobile App UX. p. 2 Preprint / article listed on ResearchGate; (2024)

[28] Rahmiati, R., Susanto, P., Hasan, A., Pujani, V.: Understanding use behavior in mobile banking: An extended of UTAUT perspective. p. 2 AFEBI Management and Business Review 7(1) (2022). p. 0 https://doi.org/10.47312/ambr.v7i01.555

[29] Tagliabue, J.: Sharing Is Caring: Our First Dataset Release for the AI Community. p. 2 https://www.coveo.com/blog/dataset-release-intent-prediction-ecommerce/. p. 1 Coveo blog post, accessed 30 November 2025. p. 0 (2020)

[30] Coulthard, S.: Alarming Behavior Tracking: Dead Clicks, Rage Clicks and More. p. 1 https://www.twipla.com/en/blog/alarming-behavior-tracking-dead-clicks-rage-clicks-and-more. p. 1 TWIPLA blog article, accessed 30 November 2025. p. 0 (2025)

[31] Feild, H.A., Allan, J., Jones, R.: Predicting searcher frustration. p. 1 In: Proceedings of the 33rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. p. 5 34–41. Association for Computing Machinery, New York, NY, USA (2010). p. 0 https://doi.org/10.1145/1835449.1835458

[32] Amplitude: What Are Rage Clicks: Detect and Fix User Frustration. https://amplitude.com/explore/analytics/rage-clicks. p. 1 Amplitude article, accessed 30 November 2025. p. 0 (2025)

[33] Luo, J., Yang, Y., Nayak, V., Yang, G.H.: Feature modulation to improve struggle detection in web search: A psychological approach. p. 1 arXiv preprint arXiv:2112. p. 004711 (2021)

[34] Dong, G., Pei, J.: Sequence Data Mining. p. 1 Springer, New York, NY (2007)

[35] Iacovacci, J., Lacasa, L.: Sequential visibility graph motifs. p. 5 Physical Review E 93(4), 042309 (2016)

[36] Iacovacci, J., Lacasa, L.: Sequential motif profile of natural visibility graphs. p. 6 Physical Review E 94(5), 052309 (2016)

[37] Xing, Z., Pei, J., Keogh, E.J.: A brief survey on sequence classification. p. 4 ACM SIGKDD Explorations Newsletter 12(1), 40–48 (2010)

[38] Lacasa, L., Luque, B., Ballesteros, F., Luque, J., Nu˜no, J.C.: From time series to complex networks: The visibility graph. p. 5 Proceedings of the National Academy of Sciences of the USA 105(13), 4972–4975

[39] Luque, B., Lacasa, L., Ballesteros, F., Luque, J.: Horizontal visibility graphs: Exact results for random time series. p. 5 Physical Review E 80(4), 046103

[40] Skforecast developers: Cyclical features in time series forecasting. https://skforecast.org/0.9.0/faq/cyclical-features-time-series Skforecast documentation (version 0. p. 69.0), accessed 30 November 2025. p. 0

[41] Black, W., Manlove, A., Pennington, J., Marchini, A., Ilhan, E., Markeviciute V.: TRACE: Transformer-based user representations from attributed clickstream event sequences. p. 1 arXiv preprint arXiv:2409. p. 012972 (2024) https://doi. p. 1org/10.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Deep learning sequence models can predict frustration reliably using only short sequences, showing that with only the first 20–30 interactions, the LSTM already predicts frustration reliably. This allows for near real-time intervention in live digital sessions.

  2. Feature engineering should include a combination of n-gram features and horizontal visibility graph motifs (HVGm) as they are shown to be very good predictors for the standard classifiers when it comes to digital frustration. These features help standard models capture complex local patterns that simple frequency counts miss.

  3. The research demonstrates that the LSTM classifier outperforms baseline models, reaching about 91% accuracy, F1-score 0.91, and ROC AUC of 0.97 compared to XGBoost's 90% accuracy. This suggests sequence modeling provides a consistent performance uplift for frustration detection.

  4. For early window prediction in real-time systems, the model can reliably detect frustration by using the first 20 events, as With about 20 interactions, the model can reliably tell frustrated vs non-frustrated sessions. This capability enables proactive help offering through automated chatbots or UI updates during a session.

  5. The pipeline should be designed to handle imbalanced data without relying on balanced training sets, as the study notes that The models are not trained, and no results are reported on the original imbalanced data (18.91% frustration and 81.09% non-frustration). This ensures models perform accurately in real-world scenarios where frustration is less common.

Sources

Related papers