From Retrieval to Customer Context: Evaluating Frontier-Model Systems for Voice-of-Customer Analysis

arXiv:2610.09375 · cs.LG · Submitted 2026-10-07 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "From Retrieval to Customer Context".

Jane: The gist The Context Graph Agent achieves an overall score of 0.961 versus 0.710 for the Deep Research Agent and 0.651 for Agentic RAG,

Tom: First, who's behind it and why it matters.

Paper summary: Jane: We’re looking at "From Retrieval to Customer Context: Evaluating Frontier-Model Systems for Voice-of-Customer Analysis" and the core finding is that persistent customer context, not just additional query-time agency, is the loadbearing lever for LLM-based VoC analysis on this corpus and under an analyst-aligned rubric.

Tom: The paper sets up a controlled comparison between three systems: Agentic RAG, a Deep Research Agent, and this new Customer Context Graph-backed Agent, all using the same nine thousand four hundred thirty-two public Cursor feedback records and the same thirty questions. They're looking at how different architectures handle that data.

Lu: The main contribution here is establishing that persistent customer context is the key mechanism for better analysis on this corpus and under an analyst-aligned rubric. They define a customer context graph as a unified model connecting customer objects, operational objects, and analytical or action objects like taxonomy concepts and evidence.

Meng: This means the agent processes feedback into a durable graph that stores typed entities with explicit product taxonomy predictions that last across sessions. The query-time agent then traverses those relationships to group, count, compare, and cite feedback from the stored context. That’s a big architectural shift for deployment if you want this level of structure.

Lalam: The Context Graph Agent achieves an overall score of zero point nine six one versus zero point seven one zero for the Deep Research Agent and zero point six five one for Agentic RAG under their rubric, leading on twenty-seven out of thirty paired questions with its largest advantage being analytical depth at a score of zero point nine six seven versus zero point six four two.

Tom: So if we look at the numbers, the Context Graph Agent scores that overall zero point nine six one, which is significantly higher than the Deep Research Agent's score of zero point seven one zero and Agentic RAG's score of zero point six five one in this specific evaluation setup.

Jane: Exactly what they are saying is that persistent customer context allows the agent to generate more specific, hierarchically developed findings with quantified themes and traceable evidence because it’s built into the model's memory from the start.

Lu: The implication for research is that building this unified model—the customer context graph—is how you get better VoC analysis on this corpus when you are using these frontier models under analyst scrutiny. It’s not about tweaking one component in isolation; it’s about how everything connects persistently, which the paper describes as the complete-system effect.

Meng: From my perspective, that means we need to design the data pipeline to prioritize that graph construction upfront rather than just relying on complex query planning at runtime. If you want deep analytical depth, you need that structure built before you even start querying it constantly.

Lalam: I agree with Lu; from my side, the Context Graph Agent’s strength comes from aggregate drill-down citations, not just higher sampled precision when it comes to evidence quality. It shows that this persistent structure unlocks better analytical depth because the agent can pull together quantified themes and traceable evidence more effectively than just searching for keywords in a database.

Tom: So if you’re building a system for customer feedback analysis, the focus should be on creating that persistent context graph structure during ingestion rather than just making query-time agency smarter. That's what this paper is pointing toward for achieving the highest quality results on this task.

Conclusion: Jane: We’re wrapping up our look at "From Retrieval to Customer Context: Evaluating Frontier-Model Systems for Voice-of-Customer Analysis." The authors compared Agentic RAG, a Deep Research Agent, and the new Customer Context Graph-backed Agent on that set of thirty product incident questions.

Tom: Basically, they concluded that how you structure the context—specifically through this persistent customer context graph—makes a massive difference in the analysis quality when using large language models for VoC.

Lu: That’s right; the paper shows that persistent customer context is not just an add-on feature; it’s actually the loadbearing lever for better VoC analysis on this corpus and under an analyst-aligned rubric. They found this agent scores zero point nine six one overall, which is significantly ahead of the other systems in terms of analytical depth.

Meng: From my side, that means we need to focus our engineering effort on building that durable graph structure during ingestion rather than trying to make every single query-time search smarter or more complex. The foundation needs to be solid before you start querying it constantly.

Lalam: I agree with Lu, and from my perspective, the context graph lets me traverse those relationships to group and count feedback much more effectively than just searching for keywords in a database. This persistent structure is what really unlocks that better analytical depth they measured in the study.

Jane: So for someone listening who doesn't read papers, think of it like this: instead of asking a chatbot a brand new question every time and having it forget everything, you give the AI one big map of all customer issues and how they relate to products, and then you ask follow-up questions about that map.

Tom: Exactly. It’s not about the fancy search tool itself, but how that tool understands what customer 'A' is related to operation 'B'. It’s a system-level decision, and that’s why it beats the others on so many questions.

Lu: The paper's conclusion is that this complete system effect—the persistent context—is what really makes these frontier models capable of high-quality analysis on messy customer data. It’s the whole picture working together.

Meng: It means for us building stuff, we need to design the data pipeline to prioritize that graph construction upfront. If the initial context setup is weak, no amount of clever query planning will save the final output quality.

Lalam: And while it’s a win for analytical depth, they did flag a limitation: if the initial taxonomy isn't specific enough, even this powerful graph approach can inherit that lack of detail in its answer. That’s something to keep an eye on when you deploy these systems.

Tom: Yeah, so the big picture is that we need to move beyond just retrieval and start thinking about creating these rich, durable knowledge models for customer interactions. It shifts where we put our engineering focus next.

Raviraja G, Viraj Bagal, Prabhath Chellingi

cs.LG

Submitted: 2026-10-07

Updated: 2026-10-07

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 91/100

The gist: The gist The Context Graph Agent achieves an overall score of 0.961 versus 0.710 for the Deep Research Agent and 0.651 for Agentic RAG, leading on 27 paired questions with its largest advantage in

Key concepts

Customer Context Graph: A unified model connecting customer objects (like feedback, users), operational objects (like tickets), and analytical objects (like themes). It stores typed entities with explicit taxonomies that remain durable across multiple sessions, enabling agents to traverse relationships for deep analysis.
Agentic RAG: An analyst built with an agent SDK that performs transient tasks like semantic search and record inspection. Unlike the graph agent, its generated groups and comparisons are temporary; they do not persist across different questions or queries.
Analytical Depth: A scoring dimension measuring how specific, hierarchically developed, and quantified the findings are. The Context Graph Agent excelled here by producing more detailed results with traceable evidence and quantified themes compared to other systems.

Summary

The gist The Context Graph Agent achieves an overall score of 0.961 versus 0.710 for the Deep Research Agent and 0.651 for Agentic RAG, leading on 27 paired questions with its largest advantage in analytical depth>

How it works

The core contribution of the study is establishing that persistent customer context, not additional query-time agency, is the loadbearing lever for LLM-based VoC analysis on this corpus and under an analyst-aligned rubric (Page 2). This is achieved by defining a customer context graph as a unified model connecting customer objects, operational objects, and analytical or action objects (Page 1). The Context Graph Agent achieves this by processing feedback into a persistent graph that stores typed customer-feedback entities (records, users, sources, themes, subthemes, sentiment, ratings) with an explicit product taxonomy and record-level taxonomy predictions that are durable across sessions (Page 2). This ingestion-time processing allows the query-time agent to traverse those relationships to group, count, compare, and cite feedback (Page 5).

Comparison of Systems

The paper compares three concrete implementations: Agentic RAG, a Deep Research Agent, and a Customer Context Graph-backed Agent (Page 3). The agents receive the same query text and evidence from the same frozen corpus but differ in their preparation and execution architecture (Page 3).

Agentic RAG is described as one customer-feedback analyst built with an agent SDK: bounded tool use, semantic search, record inspection, exact metadata aggregation, segment comparison, and cited synthesis (Page 3). It runs its own comparisons transiently (Any semantic groups it creates belong to that run and do not persist across questions) (Page 3).
The Deep Research Agent employs a decomposed workflow: a manager delegates to qualitative, quantitative, and incident specialists that search, inspect records, aggregate exactly (Page 3). It constructs matched rolling windows for specific queries (On q013 it constructs matched rolling 14-day windows (3,720 versus 3,717 records) so all three sources are present in both) (Page 5).
The Context Graph Agent receives normalized records after they are processed into a persistent customer context graph and then traverses those relationships to group, count, compare, and cite feedback (Page 5).

Evaluation Protocol

Responses are scored using a comparative rubric calibrated on 28 of the 30 questions (Page 1). The scoring involves three dimensions: answer quality (coverage and organization), analytical depth (specificity and decomposition), and evidence quality (citation support and traceability) (Page 1). The overall score is calculated as Overallq = (Qq + Dq + Eq)/3, where Qq, Dq, and Eq are the query-level scores for each dimension under a paired evaluator protocol (Page 5).

Key Findings

The results show that the Context Graph Agent leads on every dimension in Table 4, achieving an overall score of 0.961 (Page 6). Its largest advantage is analytical depth (0.967 versus 0.642), reflecting more specific, hierarchically developed findings with quantified themes and traceable evidence (Page 1). The Deep Research Agent improves over Agentic RAG by 0.059, driven primarily by evidence quality (0.806 vs 0.686) (Page 1). The Context Graph Agent leads the Deep Research Agent on 27 of 30 overall queries and is strictly best on 26 of 30 individual queries (Page 5). However, the paper notes that the largest gap is analytical depth (Page 6), and that when a pre-computed taxonomy lacks a sufficiently specific node, the graph-backed answer inherits that coarseness (Page 10).

Limitations

The study acknowledges limitations regarding validity; The rubric was aligned with analyst preferences after inspecting answers from q001–q027, and one later narrowing was prompted by q029 (Page 5). The results are scoped to the frozen configuration and do not prove universal architectural superiority (Page 3). Furthermore, the decisive test—new queries authored after the rubric was frozen, scored without further rubric change—is not performed here and remains the primary outstanding validation (Page 6). It is also noted that The Context Graph Agent emits 913 citations of which 127 are member-audited and 786 are aggregate citations with valid drill-down metadata but without exported members (Page 6).

The findings suggest that ingestion-time processing makes stored summaries, taxonomy predictions, metadata, and relationships durable: at query time the agent traverses those relationships to bounded themes with aggregate-citation quantification rather than reconstructing product concepts each question (Page 11). This is not universal dominance; the Deep Research Agent wins on bespoke matched comparisons when they matter more than concept reuse (Page 11). The context-graph citation advantage comes from aggregate drill-down citations, not from higher sampled precision (Page 6). The Context Graph Agent's performance shows that the complete-system effect of persistent customer context on this corpus, question set, and calibrated rubric; not the isolated causal effect of any component in isolation (Page 1). This is the main conclusion>

--- Page 1 ---

From Retrieval to Customer Context:

Evaluating Frontier-Model Systems for Voice-of-Customer Analysis

Raviraja G, Viraj Bagal, Prabhath Chellingi<ref:2610.09375#pg2>Enterpret<ref:2610.09375#pg2>Abstract<ref:2610.09375#pg2>Organizations increasingly use frontier language models to analyze customer feedback, but answer quality also depends on how that feedback is organized and made available<ref:2610.09375#pg2>We define a customer context graph as a unified model of customer and business context<ref:2610.09375#pg2>Typed relationships connect customer objects (feedback, conversations, users, and accounts), operational objects (tickets, support agents, opportunities, and competitors), and analytical or action objects (taxonomy concepts, evidence, insights, work items, and outcomes)<ref:2610.09375#pg2>This lets an agent investigate not only what customers say but why who is affected what action followed who owns it and whether it was resolved<ref:2610.09375#pg2>For this experiment the graph is populated from public Cursor feedback<ref:2610.09375#pg2>the same architecture can support any type of feedback source<ref:2610.09375#pg2>We compare Agentic RAG a Deep Research Agent and a Customer Context Graph-backed Agent on the same 9,432 public Cursor feedback records using 30 realistic product incident comparison and metadata questions<ref:2610.09375#pg2>Without exhaustive ground truth we jointly score responses on answer quality coverage and organization analytical depth specificity and decomposition evidence quality citation support and traceability using a comparative rubric calibrated on 28 of the 30 questions<ref:2610.09375#pg2>We sample cited records against their claims and weight the three dimensions equally<ref:2610.09375#pg2>Under this aligned rubric the Customer Context Graph-backed Agent scores 0.961 overall versus 0.710 for the Deep Research Agent and 0.651 for Agentic RAG<ref:2610.09375#pg2>and leads the Deep Research Agent on 27 of 30 paired questions (sign-test p Its largest advantage is analytical depth (0.967 versus 0.642), reflecting more specific, hierarchically developed findings with quantified themes and traceable evidence<ref:2610.09375#pg2>Because the rubric was calibrated on these outputs we report the result as exploratory: the ranking is internally consistent and survives the two questions withheld from calibration, but a confirmatory pass on freshly authored questions scored under the frozen rubric is outstanding and is the validation we consider decisive<ref:2610.09375#pg2>Within this frozen configuration frontier-model agents grounded in a persistent customer context graph produce higher-quality customer-feedback analysis than agents analyzing the same feedback without that structured context<ref:2610.09375#pg2>Keywords customer feedback analysis; retrieval-augmented generation; language-model agents; voice of customer; evidence evaluation<ref:2610.09375#pg2>Introduction<ref:2610.09375#pg3>Voice-of-customer (VoC) analysis turns qualitative feedback into a structured account of customer needs their relationships and their priorities (Griffin and Hauser 1993)<ref:2610.

Improvements for AI systems

Bold header: Persistent Customer Context Graph for Durable Knowledge

This system stores typed customer-feedback entities (records, users, sources, themes, subthemes, sentiment, ratings) with an explicit product taxonomy and record-level taxonomy predictions that are durable across sessions. This allows the agent to traverse those relationships to group, count, compare, and cite feedback, moving beyond transient context.

Bold header: Decomposed Evidence Audit for Deep Research

This system implements a decomposed workflow where a manager delegates to qualitative, quantitative, and incident specialists that search, inspect records and uses an evidence ledger to audit claims before finalization. This improves evidence quality by ensuring the verifier audits claims, counts, coverage, and citations.

Bold header: Quantified Context-Aware Analysis

The improved system can generate analysis that reports theme counts at conversation grain with per-theme drill-down rather than just exclusive classifications across the full record set. This allows for quantification architecture, where the agent develop[s] their principal drivers and links quantified claims to evidence sets.

Bold header: Dynamic Contextual Refusal Mechanism

The system can employ epistemic restraint by performing analyses like those seen in the Context Graph Agent, which declines a trend claim because the preceding calendar window is 24 Trustpilot-only conversations while the recent period is 98.4% Reddit. This prevents period-over-period trend claims when comparable baseline data is absent.

Bold header: Targeted Query-Time Computation

For specific comparison needs, the system can leverage query time to reconstruct product concepts as seen in Agentic RAG, which is useful when a bespoke comparison matters more than concept reuse.