CANVAS: Captioning Art with Narrative Visual-Audio AI Systems
cs.HC, cs.AI, cs.CL
Submitted: 2026-04-30
Updated: 2026-10-16
Comments: Springer Nature: Universal Access in the Information Society (UAIS)
License: http://creativecommons.org/licenses/by-sa/4.0/
The gist: For a person who is blind or has low vision (BLV), a painting is usually represented by a few words of alt-text, if anything at all.
Terminology
Abstract
For a person who is blind or has low vision (BLV), a painting is usually represented by a few words of alt-text, if anything at all. That text rarely conveys the colour, texture, mood or arrangement that a sighted visitor takes in, and rich audio descriptions are too slow and costly to produce by hand for most collections. This study asks how far off-the-shelf generative AI can close that gap. We built CANVAS, a no-code Zapier workflow that sends an artwork image to a multimodal large language model, checks the returned description, narrates it with a fixed ElevenLabs voice and logs every run. With prompt, pipeline and voice held constant, Gemini 2.5 Flash, GPT-5 Mini and Claude 3.5 Haiku each described the same 50 artworks, and we compared the 150 narrated descriptions artwork by artwork. Gemini and GPT-5 Mini wrote text of similar Flesch Reading Ease (49.95 and 49.47); Claude's was harder to read (37.35, p < 0.001), largely because it used longer words. Lexical diversity did not differ. The findings show both promise and a clear design requirement. All 150 narrated descriptions cost US5.10, so a first draft of audio description for an entire collection is affordable even for small museums and educators. Yet while every description conveyed colour, only 16-18% used explicit layout terms such as left, right, foreground or background, and at most 10% signalled uncertainty. Spatial structure must therefore be requested explicitly and checked by people; accuracy and usefulness to BLV listeners remain to be tested.
Related papers
- Calibrating LLM Explanations to Human Decisions with Reinforcement Learning
- Rethinking Opinion Measurement Interfaces for Human-in-the-Loop Optimization
- EduGage: A Multimodal Dataset and Benchmark for Sensor-Based Momentary Assessment of Engagement in Self-Guided Video Learning
- EvoDesign: Agentic Editable Diagram Creation via Design Expertise Evolution
- HAGI++: Head-Assisted Gaze Imputation and Generation
- Explanation User Interfaces: A Systematic Literature Review