VizCopilot: Fostering Appropriate Reliance on Enterprise Chatbots with Context Visualization

summary

The gist

Enterprise AI chatbots show promise in supporting knowledge workers in information synthesis tasks by retrieving context from large, heterogeneous databases before generating answers However…

In short

Enterprise AI chatbots struggle when retrieved context doesn't match user needs, leading to wrong answers. VizCopilot introduces a treemap visualization tool to help users manage and control this context. By organizing data visually, it allows users to sense what information they need, verify responses easily, and make group-level changes to steer the AI effectively.

Key concepts

Context Misalignment
This occurs when the data retrieved by an AI chatbot does not align with what the user actually intended. It can mean the context is entirely irrelevant or that the chatbot misinterprets it due to lacking background knowledge, resulting in an 'irrelevantly right' answer.
Sensemaking
This is a crucial skill where users develop enough understanding of the data context to guide the AI correctly. The paper suggests using visualizations to scaffold this process by organizing complex information into topics or keywords, helping users navigate and understand the data incrementally.
Progressive Disclosure
This design technique organizes information in layers, starting with a high-level overview and allowing users to drill down into more detailed views only when necessary. This reduces cognitive load by preventing users from being overwhelmed by too much raw data at once.
Group-Level Control
This feature enables users to modify context data for the entire group simultaneously, rather than having to edit files individually. This allows for substantial changes that are effective in steering the chatbot's behavior across multiple documents efficiently.

Terminology used across episodes

This episode discusses

The paper

VizCopilot: Fostering Appropriate Reliance on Enterprise Chatbots with Context Visualization · Read on arXiv

University of California, Davis University of California, Davis Department of Computer Science Microsoft Microsoft

Enterprise AI chatbots show promise in supporting knowledge workers in information synthesis tasks by retrieving context from large, heterogeneous databases before generating answers. However, when the retrieved context misaligns with user intentions, these chatbots often produce ``irrelevantly right'' responses that raise the risk of overreliance. To foster appropriate reliance, we need a different approach than asking users to verify chatbot outputs. We posit that human oversight and modification of the context retrieval process can improve the quality of chatbots' outputs and users' awareness, resulting in more appropriate reliance and higher accountability. In this work, we explore this approach through a Research-through-Design study using VizCopilot, a chatbot extended with corpus visualization techniques. Through a within-subject study comparing VizCopilot and a simplified pure-text chatbot, we evaluate the role of visualization in enterprise chatbots and surface future design opportunities for fostering appropriate reliance through context engineering. Our findings show that visualization not only helps users detect and correct misaligned context, but also enhances user agency and system transparency. At the same time, the study reveals limitations regarding verification support and trust in AI summaries. Based on these findings, we discuss how visualization can foster appropriate reliance, the associated design trade-offs, and our vision towards transparent and sustainable human-AI interaction.

Transcript

Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "VizCopilot: Fostering Appropriate Reliance on Enterprise Chatbots with Context Visualization".

Jane: Here's the gist of the paper. Enterprise AI chatbots show promise in supporting knowledge workers in information synthesis tasks by retrieving context from large, heterogeneous databases before generating answers However, when the retrieved context misaligns with user intentions, these chatbots often produce “irrelevantly right” responses that raise the risk of overreliance This work explores the value of giving users control over context engineering through data visualization to foster appropriate reliance on AI The Problem with Current Context Engineering Enterprise AI chatbots retrieve context from large databases, but autonomous retrieval approaches are susceptible to context misalignment This misalignment manifests in two key issues: first, the retrieved context may be irrelevant to the user’s prompt, and second, the chatbot may synthesize the retrieved context inappropriately due to a lack of background knowledge or incorrect assumptions Current retrieval methods struggle with messy formats and out-of-context terminologies in enterprise databases Design Requirements for Context Control The paper identifies three design requirements to address the challenges of large-scale context.

Tom: These requirements are: DR1 Sensemaking: The need is that users need to develop enough knowledge of context to be able to steer the chatbot.

Jane: Corpus visualizations can scaffold this process by organizing data with topics or keywords, allowing users to navigate information incrementally through progressive disclosure DR2 Verification Support: The system should support users in verification by both raising awareness of the need to verify responses and by reducing the cognitive load of verifying LLM outputs against context data.

Tom: Corpus visualizations can highlight areas requiring verification to minimize the amount of information users must process during verification DR3 Control at Group-Level: Users need the ability to examine and modify context data at a group level rather than file-by-file, ensuring changes are substantial enough to steer chatbots effectively VizCopilot Prototype Design VizCopilot was developed as a research prototype extending an AI chatbot with treemap-based context visualization.

Title and authors: Tom: So the title itself tells us the main goal is fostering appropriate reliance, which is a big deal because we want people to use these tools without blindly trusting them when they might be wrong. It's not about making the AI smarter, it's about making sure humans stay in charge of the process.

Jane: Exactly. They’re looking at how visualization can give users control over context engineering so they can build a better relationship with the AI rather than just being passively reliant on its potentially flawed synthesis.

Lu: The authors explore a specific approach that supports human oversight and intervention through this data visualization, which is their main contribution to the field. They evaluate if this method actually helps foster that sense of control and confidence in users.

Meng: So, it’s not just a nice UI addition; they are testing if this visualization actually changes how people interact with these systems when they're dealing with enterprise data retrieval. That sounds like a necessary test.

Lalam: They report findings from a research-through-design study using the VizCopilot prototype, which showed that incorporating this visualization did lead to improved user confidence and a stronger sense of control over the context provided to the chatbot.

The paper's summary: Tom: What they found is that users who used VizCopilot interacted with it in a more structured way, actually consulting the data context panel or visualization when cross-referencing what the AI gave them with what was actually in the data.

Jane: They also found that users could identify two main types of errors when they looked at the visualization—missing or redundant context, and misinterpretation of what was being presented. These were things they could spot by just glancing at the file view within that visualization.

Lu: This suggests that visualizing the data allows users to develop a better sense of what's relevant and helps them detect when the AI is pulling in garbage context. It scaffolds their knowledge incrementally, which is key for understanding complex enterprise information.

Meng: So, it’s not just about seeing the data; it’s about using that structure to actively check the AI’s assumptions against reality before they make a big decision based on that synthesized answer. That seems like a pragmatic step for deployment.

Lalam: The system organizes information across three levels: first you get a topical overview, then there are AI summaries for those subtopics, and finally you have a file view for raw data inspection if you need to see the details. This progressive disclosure helps manage that cognitive load we talked about earlier.

The paper's improvements: Tom: They suggest three big design requirements to make this work better: first, sensemaking—users need enough knowledge of the context to steer the chatbot, and visualizations can help scaffold that by organizing data by topic or keyword.

Jane: Then there’s verification support. The system should help users know when they actually need to check something and reduce how much mental energy it takes to verify those LLM outputs against the source data.

Lu: And third, they emphasize control at a group level, meaning users shouldn't have to go through every single file one by one; they should be able to examine and modify context data in bulk rather than doing it file-by-file.

Meng: The idea of modifying context at a group level is interesting because if you’re working on a big project, you don't want to manually edit hundreds of files just to subtly change the retrieval behavior for one AI interaction. That sounds efficient at scale.

Lalam: They propose that the visualization can surface misalignment between what the AI retrieved and what the user intended by highlighting expected context areas against the user’s query, which helps users detect and correct that misaligned context before it leads to a bad answer.

Conclusion: Tom: So, to wrap up, they argue that visualization fosters appropriate reliance because it suggests to the user rather than just telling them the answer is right or wrong, forcing that active sensemaking we talked about. It makes the retrieval process visible without creating a big concern about how faithful the AI is being.

Jane: They suggest that incorporating these visualizations into AI systems can actually enhance human cognition by taking those critical but demanding tasks off our plates, leading to a better way for us to collaborate with AI.

Lu: The limitation they point out is that they used a synthetic dataset for their study, and there’s also the need to improve how computationally scalable this visualization method is when dealing with much larger real-world enterprise datasets.

Meng: From my side, I worry about the scaling issue. If this visualization has to process petabytes of data, we're going to run into serious computational bottlenecks unless they solve that efficiency problem first.

Lalam: The paper on VizCopilot is about giving users control over context engineering through data visualization, and it shows that when you give people tools to see the structure and verify the context, their trust in the AI actually improves because they are actively involved.

More episodes

← Home