Interactive Visual Gym: Evaluating Visual Interaction Models with Simulated Users in Interactive Worlds
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "Interactive Visual Gym".
Jane: Here's the gist of the paper. Interactive Visual Gym is a framework designed for closed-loop evaluation and training of visual interaction models through joint simulation of executable task environments and human users, addressing the limitations of existing offline replay benchmarks.
Tom: The framework comprises three main components: first, simulators that recreate everyday tasks across cooking, household manipulation, and mobile interface domains in executable visual environments; second, a harness that transforms a state-of-the-art Visual Language Model (VLM) into simulated users capable of planning and acting within these environments while interacting with the assistant and revising their actions based on guidance; and third, the simulation provides execution traces necessary for comprehensive evaluation covering task completion, error detection, and interaction quality.
Jane: As an initial instantiation, IVG-Bench is constructed as a benchmark featuring 315 interaction scenarios: 150 cooking, 75 household manipulation, and 90 mobile-interface interaction scenarios.
Tom: Evaluation of state-of-the-art VLMs reveals substantial limitations; for instance, Gemini 4 Argon achieved the highest task success but only completed about 53.4% of tasks within the time limit across all environments, exhibiting weaknesses in factual grounding and actionable guidance, and being too slow to prevent errors when response latency is considered.
Title and authors: Tom: So we’re looking at this paper, "Interactive Visual Gym: Evaluating Visual Interaction Models with Simulated Users in Interactive Worlds," and the authors are from Salesforce AI Research. Jane, what’s the main idea behind that title?
Jane: Well, the core idea is moving away from just looking at a model's performance on a static test set to actually training it through closed-loop evaluation, where you simulate both an executable task environment and a real human user together.
Lu: It’s about capturing how an assistant's responses directly change what the user does next and how that affects the final outcome of the task, which is something offline replay benchmarks just can't do.
Meng: So you’re saying instead of just seeing if a model gets the right answer once, you let it interact in a loop where it has to adapt based on feedback? That sounds like a lot of setup work for deployment.
Tom: Exactly. They build this framework to test visual interaction models using three different domains: cooking, household manipulation, and mobile interface tasks. It’s not just one thing; they’re covering everyday stuff you actually do every day across these different settings.
Jane: That covers the simulators—CookSim for cooking and household manipulation, and ScreenSim for mobile interfaces—which gives us a really diverse testing ground compared to just one type of environment.
Lu: And they tie it all together with a harness that takes a state-of-the-art visual language model, or VLM, and turns it into a simulated user that can plan actions and even revise those actions based on the assistant's guidance.
Meng: What kind of control do these simulated users have? Are they just following the instructions exactly, or can they mess up in ways that actually test the assistant's correction abilities?
Jane: They incorporate controlled execution errors and other behaviors into their interactions so they can test how the assistant adapts to different user profiles, like a novice versus an expert.
Tom: That’s key. And they also introduce scheduled events, which are things that happen on a timer or at specific points in the task, like serving a wrong ingredient or forgetting a step entirely.
Lu: Those scheduled events show how the system handles things that happen over time, like needing proactive assistance to prevent someone from omitting an important step while cooking.
Title and authors: Meng: From an engineering standpoint, I wonder how much computational overhead this closed-loop simulation adds compared to just running the VLM offline? Is it feasible for high throughput?
Tom: They show that they can account for assistant delay by converting measured latency into simulation ticks, which means they’re explicitly modeling the time lag between the assistant responding and the user acting.
Jane: It also looks at how things like guidance timing affect task success; for instance, if the assistant gives advice only after the user has already moved on in their independent execution.
Lu: The results show that this framework provides useful learning signals, because experiments, like a GRPO experiment on the Qwen3 point 5-4B model in CookSim, actually improved both task success and interaction quality.
Meng: So you’re saying this iterative training approach can actually make the assistant better through these simulations? That’s interesting because offline methods usually just give you a static score.
Tom: It does, but the results are important because they show that models like Qwen3 point 5-4B improved in-time task success by about eight point nine percentage points after twelve updates when trained in this environment against a Gemini three point seven Flash user.
Jane: And it’s not just success; they measure interaction quality across five dimensions: factual grounding, situational relevance, actionable guidance, user intent uptake, and guidance conciseness.
Lu: What's really compelling is that their human studies validate the interaction paradigm; in seventy-eight point five percent of participant-assistant-pair comparisons, the evaluation metric aligned with what humans preferred.
Meng: So if humans prefer a certain style of guidance, this framework is actually measuring something meaningful beyond just whether the final dish was correct?
Jane: It does. They found that simulated users reproduce most human behaviors in terms of interaction patterns, even though they might speak to assistants more often than actual people do.
Tom: And response delays are shown to substantially reduce in-time task success for frontier API models, which is a direct result of modeling those latencies into the simulation ticks.
Lu: They specifically look at how persona conditioning shapes requests and responses, showing that a novice user behaves differently than an expert user when asking for help.
Meng: That’s good to see because in production, we often have to deal with users who are either very confused or overly demanding, and knowing how the model responds under those conditions matters a lot.
Title and authors: Jane: They also look at how different events change things: like when a user makes a wrong object selection in household tasks that triggers guidance correction from the assistant.
Tom: The recordings show specific scenarios illustrating this, like raw-serving errors in CookSim or identifying and repairing multiple incorrect settings in ScreenSim for mobile interface tasks.
Lu: This shows the framework captures complex temporal dynamics, such as temporal questions during cooking or proactive assistance to stop a user from omitting a step altogether.
Meng: So it’s not just about the final step; it’s about keeping the user on track through these small, timely interventions throughout the entire process.
Jane: And they show that when you give users an option to accept or reject advice, which is how we infer user intent uptake, human studies confirm that those quality metrics align with participant preferences.
Tom: So this whole thing points toward a system where training isn't just about accuracy on a final answer, but about building robust interaction skills under realistic constraints like time and different user types.
Lu: This framework is significant because it validates the idea that these simulated worlds provide concrete learning signals that offline benchmarks simply miss when it comes to how an assistant actually guides a user through a physical task.
Meng: For deployment, this means we can start testing models on these specific, controllable scenarios instead of just hoping they generalize well in the wild.
Jane: It opens up a way to systematically measure those interaction quality dimensions and see if the AI is actually being helpful or just technically correct.
Tom: So, if we look at the full scope of this work, we see that Interactive Visual Gym is a framework for closed-loop evaluation and training of visual interaction models through joint simulation of executable task environments and human users.
Lu: The improvements they suggest are significant because they introduce three simulators—CookSim for cooking, VHSim for household manipulation, and ScreenSim for mobile interface use—providing diverse domains to test these visual interaction models.
Meng: And the persona-conditioned user harness is crucial because it lets the simulated user incorporate controlled execution errors and out-of-plan behaviors while interacting with the assistant.
Title and authors: Jane: Plus, they provide comprehensive evaluation metrics covering in-time success, error detection recall, and those five quality dimensions we just talked about: factual grounding, situational relevance, actionable guidance, user intent uptake, and guidance conciseness.
Tom: The preliminary reinforcement learning experiments are a big deal because they demonstrate that the environment itself can improve assistant performance; for example, training Qwen3 point 5-4B with GRPO on CookSim improved in-time task success by eight point nine percentage points after twelve updates with a Gemini three point seven Flash user.
Lu: They also detail how assistant response latency can be explicitly modeled by converting measured latency into simulation ticks to assess its impact on interaction quality and error prevention, showing that measured latency substantially reduces in-time task success for frontier API models.
Meng: And they have these mechanisms for temporal question answering and proactive requests, scheduling events based on task progress conditions so the assistant can provide timely assistance when required, like notifying a user when an ingredient is ready within a specific window.
Jane: And this leads to the idea that by providing users with options like accepting or rejecting advice, and having human studies confirm that interaction quality metrics align with participant preferences in seventy-eight point five percent of comparisons, we get a solid way to measure intent uptake.
Tom: So what this means for us is that training isn't just about accuracy on a final answer; it’s about building robust interaction skills under realistic constraints like time and different user types in these simulated worlds.
Tom: So we’re looking at the Interactive Visual Gym framework, which is designed for closed-loop evaluation and training of visual interaction models through joint simulation of executable task environments and human users.
Lu: To wrap up, it’s important to remember that the three simulators—CookSim, VHSim, and ScreenSim—provide diverse domains across cooking, household tasks, and mobile interfaces.
Meng: And the persona-conditioned harness lets us control user behavior by simulating different skill levels and trust in AI while incorporating execution errors into those interactions.
Jane: Plus, the framework gives us those comprehensive evaluation metrics focusing on how well the model grounds its facts, stays relevant to the situation, guides effectively, and handles user intent uptake.
Tom: And ultimately, because they’ve shown that closed-loop training improves performance by using these simulation ticks to model assistant delay and schedule events, it gives us a practical path forward for training models that can actually interact reliably in the real world.
The paper's summary: Tom: So we’ve been talking about this framework called Interactive Visual Gym, and now we're going to look at what they actually found in their summary.
Jane: They built this whole system to test visual interaction models—the kind that understand cooking or using a mobile app—by putting them into these simulated worlds with real users who can mess up.
Tom: Exactly. The main point of the paper is that you can’t just look at a model once on a static test set anymore; you have to train it through this closed-loop cycle where the assistant gives guidance, and the user acts on it, and then the system sees what happens next.
Jane: They use three different simulators for this—CookSim for cooking, VHSim for household stuff like cleaning up, and ScreenSim for mobile interfaces. That gives them a really broad set of environments to test in.
Tom: And they show that this iterative training actually works; they ran an experiment with a Qwen3 point five-4B model, and after twelve updates using this feedback loop against a Gemini three point seven Flash user, the in-time task success improved by nearly nine percentage points. That’s a solid number you can actually hold onto.
Jane: It’s not just about the final success rate; they look at interaction quality across five dimensions—how factually grounded the assistant is, how relevant its advice is to what's happening, and how well it picks up what the user actually wants to do.
Tom: And they explicitly model things like latency—the time it takes for the AI to respond—by turning that delay into simulation ticks so they can see exactly how that slow response kills the chance of completing a task on time.
Jane: They also found that their simulated users, even though they are programmed to be imperfect, actually mimic a lot of real human behaviors when interacting with an assistant, which is important for seeing if the AI is actually helpful.
Tom: It’s interesting because they show how things like user persona matter; treating the simulated user as a novice versus an expert completely changes how much guidance they ask for and how they react to advice.
Jane: The implication here is that we can finally move past just testing whether a model gets the right answer and start training it to be a better, more reliable partner in complex, real-world visual tasks.
Tom: It’s about building models that can actually adapt their behavior based on the specific situation and the user they're talking to.
Jane: And this whole process gives us a much richer set of data than just an offline score ever could provide, which is what we need to see if these models are ready for real deployment.
Tom: So, if you’re thinking about how we can actually build these kinds of training loops for production systems, that's what this paper shows us.
The paper's improvements: Tom: So we're going to talk about what the authors suggest as improvements for this Interactive Visual Gym framework, because they aren't just stopping there at the simulation itself.
Jane: They’re suggesting ways to make the training even better, focusing on how we handle those delays and those different user types we talked about before.
Tom: One big thing is modeling assistant delay more explicitly; instead of just guessing or ignoring it, they propose converting measured latency—that time it takes for the AI to reply—directly into simulation ticks.
Jane: That way, you can see precisely how that response lag affects the user's actions and whether the guidance still feels relevant at that exact moment.
Tom: And they show this makes a real difference; measured latency substantially reduces in-time task success for those frontier API models because they can finally account for it mathematically.
Jane: Then there’s the idea of scheduling events, which lets the system be proactive instead of just reactive; imagine the assistant knows an ingredient is ready and tells you before you even have to ask.
Tom: That’s a temporal question answering thing, like if you’re cooking and it schedules a notification for when your sauce will be perfectly reduced. It makes the interaction feel much more natural.
Jane: Plus, they emphasize that giving users an option to accept or reject advice is the best way to figure out what their intent really is; human studies showed that quality metrics line up with what people actually prefer doing.
Tom: And they point back to the persona conditioning idea—making sure the simulation really captures how a novice user asks for help versus an expert user who just wants a quick fix.
Jane: It seems like the future work they suggest is to see if we can use this framework not just for evaluation, but as a direct training signal that actively improves the assistant’s skill level in these complex visual tasks.
Tom: If it works, it means we stop relying only on static test sets and start training models using dynamic, interactive feedback from simulated users.
Conclusion: Tom: So we’re wrapping up our look at Interactive Visual Gym, which is this framework for testing visual interaction models through joint simulation of task environments and human users.
Jane: Basically, they proved that you can train these models in a way that actually captures how they interact with people in the real world instead of just testing them once on a static test set.
Tom: They showed how closed-loop training—where the assistant guides and the user acts—improves performance by using those simulation ticks to model things like response delay and scheduling events.
Jane: It’s about getting models that are not only technically correct but also actually helpful when someone is trying to complete a physical task or use an app.
Tom: The implications are huge because it gives us a systematic way to measure interaction quality across five dimensions, like how good the guidance is and how well the assistant understands what the user wants.
Jane: It changes things for people who only listen to this show by showing that we can test models in these realistic scenarios before we ever send them out into the wild.
Lu: I think it’s important because it moves us past just looking at accuracy and toward training models that have actual interaction skills, which is what representation learning is really about.
Meng: For deployment, my main question is how much computation this simulation adds; we need to know if running this closed-loop evaluation framework is actually feasible when we’re trying to ship a model at scale.
Lalam: Based on the literature I’m seeing, I’m very confident that this methodology provides a more robust learning signal than what we see in most offline benchmarks.
Tom: So, Interactive Visual Gym gives us this solid foundation for training models that can actually adapt their behavior under real-world constraints like time and different user types.
Jane: It opens up a way to systematically measure those interaction quality dimensions and see if the AI is actually being helpful in a practical sense.
Lu: It’s ambitious, but if the evidence holds up from these experiments, it means we can start building assistants that are smarter about how they talk to us during tasks.
Meng: I still worry about the scaling of this simulation; we need to ensure that whatever runs this harness doesn't become too expensive or slow for production use.
Lalam: I’m sure the paper is right; it’s a very concrete framework for getting better interaction performance in these visual domains.
University of Washington Salesforce AI Research
cs.CV
Submitted: 2026-11-05
Updated: 2026-11-12
Comments: 39 pages, including references and appendices
Code: https://github.com/SalesforceAIResearch/Interactive-Visual-Gym
Project page: https://interactive-visual-gym.salesforceairesearch.com
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 100/100
The gist: Interactive Visual Gym is a framework designed for closed-loop evaluation and training of visual interaction models through joint simulation of executable task environments and human users…
Key concepts
- Simulators
- These are specialized environments like CookSim for cooking or ScreenSim for mobile interfaces that recreate real-world tasks in a visual setting. They allow the framework to test models in diverse, executable scenarios beyond simple offline benchmarks.
- User Harness
- This component transforms a Visual Language Model (VLM) into an interactive user capable of planning actions and revising them based on guidance from the assistant. It allows for controlled interaction where simulated users can make mistakes and respond to feedback.
- Persona Conditioning
- This involves simulating different user profiles, such as novices or experts, which changes how the simulated user requests help and reacts to advice. This tests whether an assistant's guidance adapts effectively based on the user's perceived skill or trust.
- Interaction Quality Dimensions
- These are five specific metrics used to evaluate assistant performance: factual grounding, situational relevance, actionable guidance, user intent uptake, and guidance conciseness. These dimensions provide a detailed view of how helpful and accurate an AI interaction truly is.
Terminology
Summary
Interactive Visual Gym is a framework designed for closed-loop evaluation and training of visual interaction models through joint simulation of executable task environments and human users, addressing the limitations of existing offline replay benchmarks. p. 0 Existing
benchmarks largely rely on offline replay and therefore cannot capture these effects. We introduce
Interactive Visual Gym, a framework for closed-loop evaluation and training of visual interaction
models through joint simulation of executable task environments and human users. We introduce
Interactive Visual Gym, a framework for closed-loop evaluation and training of visual interaction
models through joint simulation of executable task environments and human users. We contribute
three simulators that recreate everyday tasks and human-aligned action spaces across cooking, household manipulation, and mobile-interface domains, together with a persona-conditioned user simulation
harness that collaborates with the assistant while incorporating controlled execution errors and other
out-of-plan behaviors.Where the paper says it
Where the paper says it
As an initial instantiation, IVG-Bench is constructed as a benchmark featuring 315 interaction scenarios: 150 cooking, 75 household manipulation, and 90 mobile-interface interaction scenarios. p. 1 This captures how assistance affects task execution
and user experience, which offline replay cannot assess. As an initial instantiation of our framework, we construct IVG-Bench, a benchmark with 150 cooking,
75 household, and 90 mobile-interface interaction scenarios to evaluate state-of-the-art VLMs as visual
interaction models. We instantiate the framework as IVG-Bench, a benchmark with 315 interaction
scenarios. Evaluating state-of-the-art VLMs as visual interaction models reveals substantial limitations:
the best-performing model, Gemini 4 Argon, completes only 53. Evaluating state-of-the-art VLMs as visual interaction models reveals substantial limitations:
the best-performing model, Gemini 4 Argon, completes only 53. 4% of tasks within the time limit, exhibits significant weaknesses in factual grounding and actionable guidance, and is often too slow to
prevent errors when response latency is considered.Where the paper says it
Where the paper says it
Where the paper says it
The framework demonstrates that these environments provide useful learning signals; for example, a proof-of-concept GRPO experiment showed that closed-loop training improved both task success and interaction quality for the Qwen3. p. 0 5% agreement between our interaction-quality metric and participant preferences. We further show that the environments provide useful learning signals: in a proof-of-concept GRPO
experiment, closed-loop training improves both task success and interaction quality for Qwen3. We further show that the environments provide useful learning signals: in a proof-of-concept GRPO
experiment, closed-loop training improves both task success and interaction quality for Qwen3. 5-4B
model in our cooking environment. 1), and validate our evaluation paradigm through a human study (Sec. We further present a preliminary reinforcement learning experiment showing that our engine can support
training visual interaction models and improve their interaction performance (Sec. The metric agrees with human preferences in
78. 5% of participant–assistant-pair comparisons, corresponding to a pooled Kendall’s τ of +0. We also found a clear deviation
between the communication hebavior between human agent and real human: although their utterance
lengths are similar, simulated users speak to assistants much more often than human participants. One
possible reason is that the simulated user is invoked very frequently to reconsider whether to speak,
repeatedly making conversation salient. We use the same tasks, personas,
and scheduled events in both settings, each averaged across three independent runs. Response latency substantially reduces in-time task success for frontier API models. We define five dimensions of evaluation informed by
dialogue evaluation (Mehri & Eskenazi, 2020) and HCI guidance on context-sensitive, user-controllable
assistance (Amershi et al. , 2019): factual grounding, situational relevance, actionable guidance, user
intent uptake, and guidance conciseness.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
The framework details mechanisms for accounting for assistant delay by converting measured latency into simulation ticks. p. 4 2 seconds to walk in the real world, we assign each tick a duration of
∆ = 1. We convert the measured assistant inference latency ℓt into a delay of dt = ⌈ℓt/∆⌉
simulation ticks. We introduce
Interactive Visual Gym, a framework for closed-loop evaluation and training of visual interaction
models through joint simulation of executable task environments and human users. We contribute
three simulators that recreate everyday tasks and human-aligned action spaces across cooking, household manipulation, and mobile-interface domains, together with a persona-conditioned user simulation
harness that collaborates with the assistant while incorporating controlled execution errors and other
out-of-plan behaviors.Where the paper says it
Where the paper says it
-
Guidance Timing: In VHSim, guidance can arrive after the user has advanced in their independent execution, requiring the assistant to correct actions based on the current state.
-
Persona Conditioning: CookSim episodes show how different user profiles—such as a novice (low skill/high trust) versus an expert (high skill/low trust)—shape requests for guidance and responses to assistant input, leading to varied completion times and interaction styles.
-
Scheduled Events: The harness introduces scheduled events such as raw-serving errors in CookSim, forgotten-step events that are prevented, wrong-object events that trigger guidance corrections (as seen in VHSim), temporal questions during cooking, proactive assistance to prevent omitted steps (CookSim), and the identification and repair of multiple incorrect settings in ScreenSim.
These recorded interactions illustrate how different factors—timing, persona, and scheduled events—shape task execution across these diverse simulated worlds. p. 1 As an initial instantiation of our framework, we construct IVG-Bench, a benchmark with 150 cooking,
75 household, and 90 mobile-interface interaction scenarios to evaluate state-of-the-art VLMs as visual
interaction models. In each episode, the assistant is tasked to support a simulated user with a sampled
persona through task execution, and needs to deal with a set of scheduled out-of-plan events.Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Closed-loop evaluation framework for visual interaction models allows for training through joint simulation of executable task environments and human users, capturing how assistant responses change subsequent user behavior and task outcomes which existing offline replay benchmarks cannot capture.
-
The Interactive Visual Gym framework includes three simulators: CookSim for cooking, VHSim for household manipulation, and ScreenSim for mobile interface use, providing diverse domains for testing visual interaction models.
-
A persona-conditioned user simulation harness allows the simulated user to incorporate controlled execution errors and out-of-plan behaviors while interacting with the assistant, enabling evaluation of how assistants adapt to different users through attributes like skill level and trust in AI.
-
The framework provides comprehensive evaluation metrics including in-time success, error detection recall, and interaction quality across five dimensions: factual grounding, situational relevance, actionable guidance, user intent uptake, and guidance conciseness.
-
A preliminary reinforcement learning experiment demonstrates that the environment can improve assistant performance; for example, training Qwen3.5-4B with GRPO on CookSim improved in-time task success by 8.9 percentage points with a Gemini 3.7 Flash user after 12 updates.
-
Assistant response latency can be explicitly modeled to assess its impact on interaction quality and error prevention; measured latency was shown to substantially reduce in-time task success for frontier API models, often causing a decline in situational relevance and actionable guidance.
-
The system enables temporal question answering and proactive requests by scheduling events based on task progress conditions, allowing the assistant to provide timely assistance when required, such as notifying a user when an ingredient is ready within a specific window.
-
The system allows for the inference of user intent uptake by providing users with options like accepting or rejecting advice, where human study confirmed that interaction quality metrics align with participant preferences in 78.5% of comparisons.
Abstract
Visual interaction models that support users during ongoing physical tasks require evaluation in closed loop, where assistant responses can change subsequent user behavior and task outcomes. Existing benchmarks largely rely on offline replay and therefore cannot capture these effects. We introduce Interactive Visual Gym, a framework for closed-loop evaluation and training of visual interaction models through joint simulation of executable task environments and human users. We contribute three simulators that recreate everyday tasks and human-aligned action spaces across cooking, household manipulation, and mobile-interface domains, together with a persona-conditioned user simulation harness that collaborates with the assistant while incorporating controlled execution errors and other out-of-plan behaviors. We instantiate the framework as IVG-Bench, a benchmark with 315 interaction scenarios. Evaluating state-of-the-art VLMs as visual interaction models reveals substantial limitations: the best-performing model, Gemini 4 Argon, completes only 53.4% of tasks within the time limit, exhibits significant weaknesses in factual grounding and actionable guidance, and is often too slow to prevent errors when response latency is considered. We verify our evaluation protocol with a human study which finds 78.5% agreement between our interaction-quality metric and participant preferences. We further show that the environments provide useful learning signals: in a proof-of-concept GRPO experiment, closed-loop training improves both task success and interaction quality for Qwen3.5-4B model in our cooking environment.
Sources
- $\tau^2$-Bench: Evaluating Conversational Agents in a Dual-Control Environment
- Streaming Interventions: Can Video Large Language Models Correct Mistakes as They Occur?
- Don't Pause! Every prediction matters in a streaming video
- Molmo2: Open Weights and Data for Vision-Language Models with Video Understanding and Grounding
- MiniCPM-o 4.5: Towards Real-Time Full-Duplex Omni-Modal Interaction
- Kimi-VL Technical Report
- Plan, Watch, Recover: A Benchmark and Architectures for Proactive Procedural Assistance
- GuideMe: Multi-Domain Task Guidance and Intervention in Streaming Video
- Gym-V: A Unified Vision Environment System for Agentic Vision Research
- Multimodal Duplex Interaction Agent
- EgoPro-Bench: Benchmarking Personalized Proactive Interaction in Egocentric Video Streams
- $\tau$-Voice: Benchmarking Full-Duplex Voice Agents on Real-World Domains
- Training Proactive and Personalized LLM Agents
- VSAS-Bench: Real-Time Evaluation of Visual Streaming Assistant Models
- VisGym: Diverse, Customizable, Scalable Environments for Multimodal Agents
- Generative Tutorial: Towards Live Contextualized Visual Instructions for Physical Tasks
- Qwen3-Omni Technical Report
- StreamingVLM: Real-Time Understanding for Infinite Video Streams
- Proact-VL: A Proactive VideoLLM for Real-Time AI Companions
- InternVideo3: Agentify Foundation Models with Multimodal Contextual Reasoning
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs