VTO: Visual Tool Orchestration for Video Anomaly Detection
Rui Wang, Yeteng Wu, Xianling Zhang, Mengshi Qi
cs.CV, cs.AI
Submitted: 2026-08-08
Updated: 2026-08-11
Comments: Accepted by ACM MM 2026
Code: https://github.com/MICLAB-BUPT/VTO
License: http://creativecommons.org/licenses/by/4.0/
The gist: Video anomaly detection (VAD) is a critical yet challenging task due to the complex and diverse nature of real-world scenarios.
Terminology
Abstract
Video anomaly detection (VAD) is a critical yet challenging task due to the complex and diverse nature of real-world scenarios. Traditional deep learning approaches are fundamentally limited by poor generalization across diverse scenarios. While multimodal agents offer a promising tool-learning paradigm for VAD, current systems relying on supervised fine-tuning struggle with complex orchestration, and standard reinforcement learning often causes premature termination due to coarse-grained outcome rewards. To address these challenges, we propose VTO, a process-supervised reinforcement learning framework. Moving beyond static tool usage, VTO enables the agent to dynamically explore and interact with the environment. Specifically, we introduce a foundation model-driven cognitive evaluator to provide context-aware semantic feedback, which is seamlessly integrated into a Process-Supervised Cognitive Alignment that delivers fine-grained, step-wise supervision. By explicitly penalizing logical truncation and rewarding complete causal chains, the agent optimizes its multi-step reasoning policy for interrelated tool orchestration. To support our proposed framework, we meticulously crafted VAD-Tool, a hierarchical visual tool set comprising 12 specialized vision tools spanning from entity tracking to high-stakes hazard detection, and established the corresponding benchmark for rigorous multi-step reasoning evaluation. Extensive experiments on VAD-Tool demonstrate that VTO significantly outperforms baselines, achieving up to a 10.2% absolute accuracy improvement in tool scheduling. Code and data are available at https://github.com/MICLAB-BUPT/VTO.
Sources
- Qwen3-VL Technical Report
- The Evolution of Video Anomaly Detection: A Unified Framework from DNN to MLLM
- HyCoVAD: A Hybrid SSL-LLM Model for Complex Video Anomaly Detection
- SEW: Self-Evolving Agentic Workflows for Automated Code Generation
- Video Anomaly Detection and Explanation via Large Language Models
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- Llama 2: Open Foundation and Fine-Tuned Chat Models
- RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning
- VTC-Bench: Evaluating Agentic Multimodal Models via Compositional Visual Tool Chaining
Related papers
- Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
- AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
- Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
- MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
- TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
- A Survey on Efficient Vision-Language-Action Models