OmniGUI: Benchmarking GUI Agents in Omni-Modal Smartphone Environments
OmniGUI introduces a new benchmark for GUI agents by providing continuous, interleaved multimodal inputs—static images, audio, and video—at every action step. This tests agents in real-world smartphone environments where actions depend on dynamic temporal and auditory cues. Findings show that while models handle static tasks well, performance drops significantly when processing synchronized audio and video.