CANVAS: Captioning Art with Narrative Visual-Audio AI Systems

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "CANVAS: Captioning Art with Narrative Visual-Audio AI Systems".

Jane: Here's the gist of the paper. The gist The system presents an automated workflow that generates multi-sensory art descriptions and synchronized audio narration using large language models and text-to-speech services, enabling rapid, scalable production of accessible media.

Tom: Project Introduction & Brief Literature Review Visual art remains largely inaccessible to blind and low-vision (BLV) audiences due to brief or absent alt‐text, which rarely conveys the sensory, spatial, or emotional qualities of an artwork <ref:2606.09846#pg2> In practice, alt‐text is typically limited to a one‐sentence caption of roughly ten words, an economy of language that cannot capture context, style, or emotional tone <ref:2606.09846#pg4> Accessible art description guidelines (e.g.

Jane: the Art Beyond Sight framework) emphasize structured language, consistent terminology, and spatial ordering to build robust mental models for BLV audiences <ref:2606.09846#pg2> The art experience for a BLV person requires not just naming visible elements but conveying style, emotion, and context – for example, the play of light and shadow in a painting or the historical and symbolic meaning behind its imagery <ref:2606.09846#pg2> In response to these gaps, advances in AI‐driven image captioning promise new possibilities <ref:2606.09846#pg2> Early automated captioning systems used fixed templates or simple object recognition (“Image may contain“), which often produced stilted or irrelevant text <ref:2606.09846#pg2> In the deep learning era, encoder–decoder models using convolutional neural networks (CNNs) to process the image and recurrent networks (e.g.

Tom: LSTMs) to generate text became standard <ref:2606.09846#pg2> Attention mechanisms (and later scene‐graph models) improved the ability to localize salient image regions, while Transformer architectures and vision‐language pretraining have yielded even richer, more fluent captions <ref:2606.09846#pg2> Generic captioning systems tend to emphasize object labels and literal scene description, missing the narrative, sensory, and contextual layers that make art meaningful <ref:2606.09846#pg2> To appreciate an artwork nonvisually, BLV audiences require rich, narrative descriptions that go well beyond object lists <ref:2606.09846#pg2> They benefit from multi‐sensory language that evokes texture, sound, emotion, and story <ref:2606.09846#pg2> Joselia Neves emphasizes that effective descriptions should include subjective interpretation and sensory metaphor so that BLV people can “experience visual art at a level comparable to sighted individuals” <ref:2606.09846#pg2> In short, BLV users often prefer a storytelling voice that situates the artwork in space and emotion, rather than a terse clinical list of contents <ref:2606.09846#pg2> The model operates through an end-to-end automation that generates short, narrative audio descriptions of artworks, focusing on system design decisions, controls, and execution flow <ref:2606.09846#pg2> The pipeline is implemented in Zapier to ensure repeatable processing for every image and to isolate the experimental variable (the choice of large language model) while holding all other components constant <ref:2606.09846#pg2> The workflow is triggered the moment a new image is saved to a designated Dropbox folder, aptly titled “CANVAS Research” <ref:2606.09846#pg2> Immediately after the trigger, a brief delay is inserted to ensure the file is fully synced and available for subsequent steps <ref:2606.09846#pg2> After this controlled pause, the image reference and metadata are passed to the “AI by Zapier” action, which communicates with the selected language model to produce a compact two- to three-sentence script describing the artwork in rich, listener-friendly prose <ref:2606.09846#pg2> The language model step is the only element that varies across experimental conditions <ref:2606.09846#pg2> The prompt requests a concise narrative oriented to nonvisual consumption, with instructions to describe spatial layout, salient visual elements, and mood without introducing unsupported speculation <ref:2606.09846#pg2> Once the model responds, a filter validates that the return type is a plain text string and that it is nonempty <ref:2606.09846#pg2> Routing logic then forks: valid strings continue to text-to-speech, while any invalid or missing output triggers an alert <ref:2606.09846#pg2> Valid model text proceeds to a constant text-to-speech stage implemented with ElevenLabs through their API <ref:2606.09846#pg2> Keeping this component fixed across all runs ensures that prosody, voice timbre, and synthesis parameters do not become hidden variables in the comparison between models <ref:2606.09846#pg2> The audio file path, model name, timestamp, and a hash of the input image filename are logged to a Google Sheets ledger via Zapier <ref:2606.09846#pg2> This logging step provides a tamper-evident run record and a simple way to cross-reference generated outputs with later evaluations <ref:2606.09846#pg2> Standardization is central to the methodology, as all environmental details, including Zap configuration, delay duration, voice selection, output format, and logging fields are identical between automation variants <ref:2606.09846#pg2> Only the model invoked by the “AI by Zapier” step changes across experimental runs <ref:2606.09846#pg2> Image Sourcing for Input Dataset The input dataset consists of 50 artworks grouped into five thematic categories (Renaissance/Baroque, Impressionism, Modern/Abstract, Photography/Mixed Media, and Narrative Scenes/Landscapes), as outlined above <ref:2606.09846#pg2> These images are fixed and provided identically to each model, ensuring that any differences in output are due to the model rather than variation in input <ref:2606.09846#pg2> The dataset’s composition is therefore designed for stress-testing the captioning models across diverse scenarios <ref:2606.09846#pg2> Category 1 (Renaissance/ Baroque) includes highly detailed, realistic human figures and symbolic narratives; Category 2 (Impressionism) showcases changes in light and visible brushwork <ref:2606.09846#pg2> Category 3 (Modern/Abstract) contains distorted forms and emotional motifs <ref:2606.09846#pg2> Category 4 (Photography/Mixed) adds real-world scenes and conceptual compositions <ref:2606.09846#pg2> Category 5 (Narrative Scenes/Landscapes) offers broad vistas and action-filled genre scenes <ref:2606.09846#pg2> By exposing models to such a wide range, we test not only their object recognition but also their ability to interpret style, context, and metaphor <ref:2606.09846#pg2> This multicategory dataset effectively simulates a multi-domain benchmark <ref:2606.09846#pg2> Results & Data Collection For each of the 50 images, the automation produced both text and audio in under 20 seconds and at a total cost under 5 per image <ref:2606.09846#pg2> This timing reflects the end-to-end pipeline, not just generation latency <ref:2606.09846#pg2> Every output was captured in a single Google Sheets workbook with three tabs, one per model: “Google Gemini 2.5 Flash,” “OpenAI GPT-5 Mini,” and “Anthropic Claude 3.5 Haiku” <ref:2606.09846#pg2> We assessed readability with the Flesch–Kincaid Reading Ease (FKRE) metric, which maps sentence length and word syllabification to a 0–100 scale <ref:2606.09846#pg2> Averaging across the 50 images per model, the FKRE scores are: Google Gemini 2.5 Flash: 49.95, OpenAI GPT-5 Mini: 49.47, and Anthropic Claude 3.5 Haiku: 37.35 <ref:2606.09846#pg2> These values show that Gemini and GPT-5 produced captions clustered around the upper-high-school range, while Claude’s captions were substantially more complex, aligning with college-level difficulty under FKRE’s conventional interpretation <ref:2606.09846#pg2> The mean difference between Claude and the other two models is notable, suggesting a systematic tendency toward more complex phrasing from Claude under the common prompt and fixed TTS settings <ref:2606.

Title and authors: Tom: So we’re looking at the CANVAS: Captioning Art with Narrative Visual-Audio AI Systems paper today. It was written by Vignesh Nagarajan, who is from BASIS Phoenix High School in Arizona.

Jane: That title tells you right away they aren't just making basic captions; they are aiming for a whole system that generates descriptions and audio together.

Lu: What’s interesting is that they are using this automated workflow, orchestrated through Zapier, to convert an image into a rich narrative caption without needing a human writer first.

Meng: I see the focus on the automated workflow being orchestrated through Zapier as smart because it sounds like they’ve built this system to be repeatable for every single image without constant manual setup.

Lalam: The authors mention that this system converts uploaded images into rich narrative captions without human intervention, which is a big part of how it aims to solve the problem of fragmented information for blind and low-vision audiences.

Tom: It seems like the core idea is using AI to generate these multi-sensory descriptions and synchronized audio narration, taking an image and turning it into something that can be heard as well as read.

Jane: And they are testing this by showing that their system produces text plus audio outputs in under twenty seconds per image at a cost below.05 <ref:2606.09846#pg2,seconds per image at a cost below $0.05>. That’s a very tight turnaround for what they’re delivering, which is important for accessibility tools.

Lu: The paper sets up the problem by pointing out that most visual art remains inaccessible to blind or low-vision audiences because of brief or absent alt-text that rarely captures the sensory or emotional qualities of an artwork <ref:2606.09846#pg2>.

Meng: They are essentially tackling the gap between simple object labels and what a person who is blind needs to understand about a scene.

Lalam: The system they describe is designed specifically to produce these multi-sensory art descriptions and synchronized audio narration using large language models and text-to-speech services <ref:2606.09846#pg2>.

Tom: So, the main mechanics they are showing us involve taking an image as input and generating something that can be both heard and read out loud.

Jane: And before we get into what they found, let’s quickly look at the setup: how did they build this pipeline to make sure it actually works reliably?

The paper's summary: Tom: The core of the CANVAS paper is presenting an automated workflow that generates multi-sensory art descriptions and synchronized audio narration using large language models and text-to-speech services <ref:2606.09846#pg2>.

Jane: They found that when they tested this system across fifty artworks, it produces descriptions with significantly higher lexical diversity, adjective density, and narrative detail compared to the baseline captions they used as a starting point <ref:2606.09846#pg2>.

Lu: The statistical tests they ran—t-tests and ANOVA—confirmed that there are meaningful differences in richness and length between the AI-generated descriptions and the simpler ones <ref:2606.09846#pg2>.

Meng: I’m looking at those results because I want to know if this richness is just noise, or if it actually helps a nonvisual user build a mental model of the artwork better <ref:2606.09846#pg2>.

Lalam: The findings demonstrate that AI-generated descriptions contain significantly higher lexical diversity, adjective density, and narrative detail than baseline captions, which shows the AI is adding actual substance to the description <ref:2606.09846#pg2>.

Tom: So they’re confirming that this approach isn't just making things longer; it’s making them more detailed in terms of vocabulary and descriptive language <ref:2606.09846#pg2>.

Jane: They are showing that this detailed output still manages to maintain comparable readability levels, which is a really important finding for practical application <ref:2606.09846#pg2>.

Lu: This suggests that the system is successfully injecting narrative context and sensory depth into the description without sacrificing clarity for the user <ref:2606.09846#pg2>.

Meng: If it’s keeping readability comparable, then this automated approach might be feasible for real-world deployment because it doesn't just produce verbose text that nobody can process <ref:2606.09846#pg2>.

Lalam: The paper shows that this end-to-end pipeline works reliably across fifty different artworks to generate output with significantly higher lexical diversity and narrative detail than simpler baseline methods <ref:2606.09846#pg2>.

Tom: So, to put it plainly, CANVAS is about using AI to bridge the gap between a visual artwork and a rich, accessible audio experience for blind or low-vision audiences <ref:2606.09846#pg2>.

Jane: It’s a practical application of these large language models that shows how they can be used to create something genuinely meaningful for people who experience the world through sound and narrative <ref:2606.09846#pg2>.

The paper's improvements: Jane: Moving on to what they suggest as improvements, the authors point out that their system, while good at generating rich text, still needs refinement in how that text is structured for maximum accessibility <ref:2606.09846#pg2>.

Lu: They propose using a controllable prompt system to enforce structural requirements, nudging the outputs toward shorter sentences and simpler vocabulary when targeting general audiences <ref:2606.09846#pg2>.

Tom: That makes sense. If the goal is rapid production for a broad audience, you don't want super complex prose that might overwhelm someone who needs a clear path through the description <ref:2606.09846#pg2>.

Meng: I wonder if forcing structural requirements would impact the model’s ability to capture subtle nuances of style or emotion that are important in art, even if it makes it easier to read at scale <ref:2606.09846#pg2>.

Lalam: The paper suggests this control mechanism is also tied into an evaluation process where they look at readability metrics like the Flesch-Kincaid Reading Ease score, which maps sentence length and word syllabification to a zero to one hundred scale <ref:2606.09846#pg2>.

Jane: And that’s where I think it gets interesting, because they are trying to move beyond just measuring sentence length and complexity as a proxy for quality <ref:2606.09846#pg2>.

Lu: They also mention that effective descriptions should include subjective interpretation and sensory metaphor so that BLV people can experience visual art at a level comparable to sighted individuals <ref:2606.09846#pg2>.

Tom: So they’re moving the goalposts from just making it long to making sure the language actually hits the right emotional and spatial notes for a viewer <ref:2606.09846#pg2>.

Meng: From a deployment view, they need this control mechanism if they want to deploy it widely because you can't afford to send out descriptions that are too dense or too vague <ref:2606.09846#pg2>.

Lalam: The paper highlights that the system needs to be able to adapt its language based on the specific audience it is trying to serve, which is a key step toward making art truly accessible <ref:2606.09846#pg2>.

Conclusion: Tom: So, wrapping things up on CANVAS, the main implication is that we can rapidly scale the production of accessible media by automating the process of generating rich, multi-sensory descriptions and synchronized audio narration from an image <ref:2606.09846#pg2>.

Jane: What this means for us is that we can potentially deliver artwork to blind or low-vision audiences much faster than before, providing them with a level of detail that goes way beyond simple object labels <ref:2606.09846#pg2>.

Lu: The authors are clearly focused on making sure the output is not just long, but actually useful by implementing those structural controls in the prompt system they discussed earlier <ref:2606.09846#pg2>.

Meng: From a deployment view, the key thing here is that it’s automated and cost-effective, producing text plus audio outputs in under twenty seconds per image at a cost below.05 <ref:2606.09846#pg2> <ref:2606.09846#pg2,seconds per image at a cost below $0.05>.

Tom: So, in short, CANVAS is about using AI to bridge the gap between a visual artwork and a rich, accessible audio experience for BLV audiences <ref:2606.09846#pg2>.

Jane: It’s a practical application of these large language models that shows how they can be used to create something genuinely meaningful for people who experience the world through sound and narrative <ref:2606.09846#pg2>.