Appear2Meaning: A Cross-Cultural Benchmark for Structured Cultural Metadata Inference from Images
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "Appear2Meaning: A Cross-Cultural Benchmark for Structured Cultural Metadata Inference from Images".
Jane: Here's the gist of the paper. Recent advances in vision-language models (VLMs) have improved image captioning for cultural heritage; however, inferring structured cultural metadata from visual input remains underexplored.
Tom: This work introduces a multi-category, cross-cultural benchmark for evaluating structured metadata inference from image-only input.
Jane: The results show that models capture fragmented signals and exhibit substantial performance variation across cultures and metadata types.
Tom: These findings highlight the limitations of current VLMs in structured cultural metadata inference beyond visual perception.
Paper summary: Tom: So, wrapping up "Appear2Meaning: A Cross-Cultural Benchmark for Structured Cultural Metadata Inference from Images," the authors are showing us that these vision-language models can definitely pick up some cultural signals from images, but getting the exact metadata right—culture, period, and origin—is still really tough.
Jane: The paper uses this benchmark to prove that performance isn't uniform; it varies a lot depending on whether you're looking at East Asia or Europe or the Americas. This variation points to the fact that these models are relying on shortcuts instead of deep reasoning about cultural context.
Lu: The authors conclude that while current vision-language models can capture partial cultural signals, exact metadata inference remains challenging, especially for culture, period, and origin. They suggest that many predictions rely on stylistic shortcuts and memorized creator associations rather than robust multi-attribute reasoning.
Meng: If you're deploying a model like this in production right now, you need to be aware of those limitations because the performance is inconsistent across different cultural contexts. It’s not a one-size-fits-all accuracy number.
Lalam: The findings suggest that we should treat the outputs as hypothesis-like descriptions rather than authoritative judgments about what a piece of heritage actually is, especially when it comes to deep cultural attribution.
Tom: So, in short, the challenge isn't just seeing the visual features; it’s aligning those features with provenance-constrained metadata without over-relying on common stylistic associations.
Jane: This work provides a crucial cross-cultural benchmark for evaluating how well these models perform when they are asked to infer complex cultural knowledge from images alone.
Lu: Future work is going toward extending this benchmark to finergrained cultural distinctions and exploring ways to integrate external knowledge sources into the models to improve that coherent multi-attribute metadata inference.
Meng: For deployment, we need more than just a good score on this benchmark; we need a system that can handle that performance variation robustly across all the different cultures it might encounter.
Lalam: So, while they show what's possible now, the authors clearly indicate that for truly reliable cultural metadata inference, we still have some significant work ahead before these models become fully trustworthy.
Conclusion: Tom: So, this paper, "Appear2Meaning," is introducing a way to test how well vision language models can figure out structured cultural stuff from just pictures.
Jane: It's testing if these models can predict things like the culture, the time period, and where something came from, based only on what they see in an image.
Tom: The authors are using a multi-category benchmark to see how different cultures and different types of cultural labels affect model performance.
Lu: They’re formalizing it as a structured prediction problem where the model has to infer specific metadata attributes like culture, period, origin, and creator from just the visual input.
Meng: The setup is interesting because they are using publicly available data from places like the Getty and Metropolitan Museum of Art to build this dataset.
Lalam: They curated it through rule-based filtering and two stages of human verification to get a balanced cross-cultural set, focusing on things that aren't always obvious just by looking at the picture.
Tom: The results show that models capture fragmented signals and there's a lot of variation across different cultures and metadata types.
Jane: That means the models aren't performing equally well everywhere; they struggle in some places more than others.
Lu: They also found that the evaluation uses an LLM as a judge framework to check for semantic alignment between what the model predicts and the actual reference annotations, which is better than just checking if words match up.
Meng: The error analysis is where it gets really useful because they categorize mistakes into four types, like cross-cultural misattribution or temporal inference errors.
Tom: Apparently, a common mistake is when models map visual style to a more familiar Eurocentric category even when the object isn't from there.
Jane: That’s a big deal for us because it shows these models are relying on learned associations instead of actual deep understanding of cultural context.
Lu: They also pointed out that creator attribution can be successful sometimes, but only if the model has memorized that specific creator rather than reasoning about the surrounding cultural context.
Meng: It’s a lot of work to get those kinds of nuanced inferences when you're trying to ship a model to production, especially when you don't have perfect knowledge of what the data implies.
Tom: The authors conclude that while current models can grab some partial cultural signals, getting exact metadata right for culture and period is still really hard.
Jane: They stress that these outputs should be seen as hypothesis-like descriptions rather than authoritative facts about the object's history.
Lu: Future work involves creating a finer benchmark for those cultural distinctions and thinking about how to bring in external knowledge sources to make the multi-attribute inference more coherent.
Meng: For us, that means we need better ways to test if these models are actually robust enough for real-world use cases where they have to handle all this variability.
Tom: So, the main point here is that we've got these tools that can see images and guess some cultural metadata, but the guessing part is where the real technical challenge lies.
Jane: And the next thing we need to figure out is how to make sure those guesses are reliable enough for people who actually want to trust them with cultural heritage information.
University of Manchester · School of Artificial Intelligence, Wuhan University · Getty Conservation Institute
cs.CV, cs.CL, cs.MM
Submitted: 2026-04-08
Updated: 2026-11-15
Code: https://github.com/QwenLM/Qwen-VL
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 75/100
The gist: Recent advances in vision-language models (VLMs) have improved image captioning for cultural heritage; however, inferring structured cultural metadata from visual input remains underexplored.
Key concepts
- Structured Metadata Inference
- This is the task of forcing an AI to predict specific, organized pieces of information about an object—such as its culture or creator—using only a picture. It treats heritage understanding as a structured prediction problem where the model must output predefined labels from museum schemas.
- LLM-as-a-Judge Framework
- Instead of using traditional metrics, this method uses a powerful language model (like GPT) to evaluate the AI's predictions. The judge assesses semantic alignment between what the model predicted and the correct reference metadata, focusing on meaning rather than just matching words.
- Cross-Cultural Misattribution
- This is a common error where models incorrectly assign cultural labels based on visual style. For example, a model might map an object's artistic style to a more familiar European category instead of its true origin, showing that visual cues often trigger biased associations.
- Fragmented Signals
- Models often capture only small parts of the necessary information from an image. This means they can recognize some objects but fail to connect the visual evidence robustly enough to infer complex, multi-attribute cultural details like precise historical periods or origins.
Terminology
Summary
Recent advances in vision-language models (VLMs) have improved image captioning for cultural heritage; however, inferring structured cultural metadata from visual input remains underexplored. p. 1 Abstract
Recent advances in vision-language models (VLMs) have improved
image captioning for cultural heritage. However, inferring structured
cultural metadata (e. This gap indicates that models capture
individual attributes but struggle to produce coherent multi-attribute
predictions across cultural contexts. 5 Conclusion and Future Work
We introduced a cross-cultural benchmark for evaluating visionlanguage models on structured cultural metadata inference from
image-only input. To assess cultural reasoning, we report exact-match,
partial-match, and attribute-level accuracy across cultural regions. Results show that models capture fragmented signals and exhibit
substantial performance variation across cultures and metadata types,
leading to inconsistent and weakly grounded predictions. Results show that models capture fragmented signals and exhibit
substantial performance variation across cultures and metadata types,
leading to inconsistent and weakly grounded predictions. These findings highlight the limitations of current VLMs in structured cultural
metadata inference beyond visual perception.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Pruning redundant reasoning tokens
The paper formalizes heritage understanding as a structured prediction problem where models are prompted to infer structured metadata attributes from visual input only. The goal is to predict structured metadata M =
mculture,mperiod,morigin,mcreator where each attribute corresponds to a well-defined label grounded in museum metadata schemas. The evaluation objective is M ≈ M ˆ ∗ using an LLM-as-a-Judge framework that measures semantic alignment with reference annotations. p. 1 , creator, origin, period) from visual input remains underexplored. We introduce a multi-category, cross-cultural
benchmark for this task and evaluate VLMs using an LLM-as-aJudge framework that measures semantic alignment with reference
annotations.Where the paper says it
How the training signal is built
The dataset is curated from publicly available records of the Getty Art Collections and the Metropolitan Museum of Art Open Access collection. The curation process involves rule-based filtering over metadata fields followed by two-stage human verification to build a balanced cross-cultural dataset. p. 3 ACM MM ’26, November 10–14, 2026, Rio de Janeiro, Brazil Jiang et al. Figure 2: Data curation pipeline combining rule-based filtering and
human verification to build a balanced cross-cultural dataset. As
shown in Figure 1, models predict structured metadata from visual input and are evaluated using an LLM-as-Judge against normalized and
raw museum annotations. The benchmark targets non-observable,
culturally grounded attributes and enables fine-grained analysis
of performance and bias across cultural contexts.Where the paper says it
Where the paper says it
Where the gains come from
The evaluation uses an LLM-as-a-Judge framework to assess attribute-level correctness by prioritizing semantic alignment between predicted and reference metadata over lexical similarity. Performance varies across cultural regions, with higher accuracy in East Asia and lower accuracy in Europe and the Americas. p. 1 At the attribute level, models perform
better on title and creator, while culture, period, and origin remain
more difficult. Performance also varies across cultural regions, with
higher accuracy in East Asia and lower accuracy in Europe and the
Americas. 609) and Qwen-VL-Max
(0. Notably, open-weight Qwen models demonstrate competitive or superior performance compared to closed-source models in
terms of partial correctness.Where the paper says it
Where the paper says it
Error Analysis
The error analysis categorizes model errors into four types corresponding to patterns observed across attributes and cultural contexts. The most common failure is cross-cultural misattribution, often by mapping visual style to a more familiar Eurocentric category. p. 5 Cross-cultural misattribution. The most common failure is assigning an artifact to the wrong cultural contexts, often by mapping
visual style to a more familiar Eurocentric category. A
second failure mode is that models recognize coarse visual form but
miss the historically specific object type. This leads to superficially
plausible but semantically wrong titles. Period compression and stylistic over-anchoring. A third pattern is incorrect temporal inference driven by broad stylistic priors. Creator memorization versus holistic reasoning. A fourth pattern is partial success on creator attribution without correspondingWhere the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Case Studies and Error Analysis
Case study A illustrates a recurring pattern of cross-cultural misattribution across multiple models, where predictions are grounded in plausible stylistic associations rather than strictly incorrect reasoning. Case study B highlights a pattern of style-driven cross-cultural confusion where visual resemblance is associated with predictions shifting the object’s cultural attribution toward European contexts. p. 9 1 Case Study A: Systematic Cross-Cultural Misattribution
Object ID: 1055_Butter Pat
Figure 3: Example images for Object ID 1055_Butter Pat. Ground Truth:
• Title: Butter Pat
• Culture: American
• Period: 1885
• Creator: Union Porcelain Works
Representative Predictions:
• Claude Haiku 4. MLF [32] and CArt15K [55]
focus on ceramics with factor-driven or metaphor-enriched annotations, yet remain restricted to specific object types and cultural
contexts. Painting-centric datasets such as GalleryGPT [9], ArtCap [36], DEArt [42], and ARTSEEK [18] emphasize visual analysis and stylistic recognition without modeling structured cultural
metadata. This pattern is
especially frequent for objects in the Americas. This suggests that
models rely on visual similarity and training data priors rather than
accurate inference of cultural metadata.Where the paper says it
Where the paper says it
Where the paper says it
Conclusion and Future Work
The work concludes that while current models can capture partial cultural signals, exact metadata inference remains challenging, especially for culture, period, and origin. Error analysis reveals that many predictions rely on stylistic shortcuts and memorized creator associations rather than robust multi-attribute reasoning. p. 6 Our results show that,
while current models can capture partial cultural signals, exact metadata inference remains challenging, especially for culture, period,
and origin. Error analysis further reveals that many predictions rely on stylistic shortcuts, culturally dominant priors, and memorised creator
associations, rather than robust multi-attribute reasoning. Future work will therefore extend the benchmark toward finergrained cultural distinctions, broader object categories, and larger
and more balanced sample sizes, enabling more targeted and representative evaluation across diverse cultural contexts. We will also
explore integrating external knowledge sources, including retrievalaugmented and ontology-grounded approaches, as well as structured
mechanisms that connect visual predictions to museum knowledge
bases, to improve coherent multi-attribute metadata inference. To mitigate these risks,
we encourage incorporating domain expertise, expanding culturally diverse datasets, and transparently reporting model limitations. Responsible use should prioritise human oversight, especially in culturally sensitive contexts, and avoid treating automated predictions
as definitive cultural interpretations.Where the paper says it
Where the paper says it
Where the paper says it
Ethical Considerations
This work uses publicly available cultural heritage data from museum collections under open-access policies and releases code and annotations under the MIT License. Cultural heritage data reflect historical, institutional, and curatorial biases; consequently, models trained or evaluated on such data may inherit and amplify these biases. p. 6 , CC0 or CC BY), and releases code
and annotations under the MIT License. Cultural heritage data reflect
historical, institutional, and curatorial biases; consequently, models
trained or evaluated on such data may inherit and amplify these biases, as evidenced by performance disparities across cultural regions. Inferring
non-observable cultural attributes (e. , title, culture, period, origin,
creator) from images introduces epistemic uncertainty and may produce incorrect or overconfident outputs, which should not be treated
as authoritative judgments or used without expert validation in realworld applications, particularly in museum, educational, or heritage
contexts. , title, culture, period, origin,
creator) from images introduces epistemic uncertainty and may produce incorrect or overconfident outputs, which should not be treated
as authoritative judgments or used without expert validation in realworld applications, particularly in museum, educational, or heritage
contexts. The use of LLM-as-a-Judge introduces evaluation bias, as
judgments depend on the knowledge and assumptions encoded in the
evaluator model, potentially reinforcing dominant cultural narratives
or overlooking alternative interpretations.Where the paper says it
Where the paper says it
Where the paper says it
References
[1] Manar Abu Talib, Iman Ibrahim, and Manar Anwer Abusirdaneh. 2026. Reusability and Benchmarking Potential of Architectural Cultural Heritage Datasets for Generative AI: An Analytical Study. p. 6 References
[1] Manar Abu Talib, Iman Ibrahim, and Manar Anwer Abusirdaneh. Reusability and Benchmarking Potential of Architectural Cultural Heritage Datasets for
Generative AI: An Analytical Study. Reusability and Benchmarking Potential of Architectural Cultural Heritage Datasets for
Generative AI: An Analytical Study. Expert Systems With Applications (2026). Expert Systems With Applications (2026). Published online 16 January 2026.Where the paper says it
Where the paper says it
Where the paper says it
[5] Hannah Andrews and Aurora Hawcroft. p. 6 In Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition (CVPR). [5] Hannah Andrews and Aurora Hawcroft. [5] Hannah Andrews and Aurora Hawcroft. Articulating arts-led AI: artists
and technological development in cultural policy. Articulating arts-led AI: artists
and technological development in cultural policy. European Journal of Cultural
Management and Policy (2024).Where the paper says it
Where the paper says it
Where the paper says it
[7] Murtha Baca (Ed. p. 6 https://www. p. 3 1
under an image-only setting. For each input image, models generate
structured metadata predictions, which are subsequently evaluated
1
https://www. 1
under an image-only setting. For each input image, models generate
structured metadata predictions, which are subsequently evaluated
1
https://www. CCS Concepts
• Computing methodologies → Scene understanding; Information extraction; • Applied computing → Arts and humanities. Keywords
cultural heritage, vision-language models, multimodal understanding, metadata inference, benchmark dataset, LLM-as-Judge
1 Introduction
“Man is an animal suspended in webs of significance he himself has
spun. Introduction to Metadata (3 ed. Getty Research
Institute, Los Angeles. https://www. edu/publications/intrometadata/
[8] Longju Bai, Angana Borah, Oana Ignat, and Rada Mihalcea.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
[9] Yi Bin, Wenhao Shi, Yujuan Ding, Zhiqiang Hu, Zheng Wang, Yang Yang, SeeKiong Ng, and Heng Tao Shen. p. 6 Association for Computational Linguistics. [9] Yi Bin, Wenhao Shi, Yujuan Ding, Zhiqiang Hu, Zheng Wang, Yang Yang, SeeKiong Ng, and Heng Tao Shen. [9] Yi Bin, Wenhao Shi, Yujuan Ding, Zhiqiang Hu, Zheng Wang, Yang Yang, SeeKiong Ng, and Heng Tao Shen. GalleryGPT: Analyzing Paintings with Large
Multimodal Models. GalleryGPT: Analyzing Paintings with Large
Multimodal Models. In Proceedings of the 32nd ACM International Conference
on Multimedia (MM ’24). In Proceedings of the 32nd ACM International Conference
on Multimedia (MM ’24). ACM, Melbourne, Australia.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
[16] Martin Doerr. p. 7 , Red Hook,
NY, USA, Article 2142, 18 pages. [16] Martin Doerr. [16] Martin Doerr. The CIDOC Conceptual Reference Model: An Ontological
Approach to Semantic Interoperability of Metadata. The CIDOC Conceptual Reference Model: An Ontological
Approach to Semantic Interoperability of Metadata. International Journal of
Human-Computer Studies 43, 5 (2003), 75–92.Where the paper says it
Where the paper says it
Where the paper says it
[28] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. p. 7 org, Article 814, 13 pages. [28] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. [28] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding
and Generation. BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding
and Generation. In Proceedings of the 39th International Conference on Machine
Learning (Proceedings of Machine Learning Research, Vol. BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding
and Generation. In Proceedings of the 39th International Conference on Machine
Learning (Proceedings of Machine Learning Research, Vol. In Proceedings of the 39th International Conference on Machine
Learning (Proceedings of Machine Learning Research, Vol. 162), Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato
(Eds. In Proceedings of the 39th International Conference on Machine
Learning (Proceedings of Machine Learning Research, Vol. 162), Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato
(Eds.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
[39] OpenAI. p. 4 Appear2Meaning: A Cross-Cultural Benchmark for Structured Cultural Metadata Inference from ImagesACM MM ’26, November 10–14, 2026, Rio de Janeiro, Brazil
Model Organization Release Time
Open-Weight Models
Qwen-VL-Max [22] Alibaba 2024
Qwen3-VL-Plus [23] Alibaba 2025
Qwen3-VL-Flash [23] Alibaba 2025
Qwen3-VL-8B-Instruct [23] Alibaba 2025
Qwen3-VL-32B-Instruct [23] Alibaba 2025
Pixtral-12B [2] Mistral AI 2024
Close-Source Models
GPT-4. 1-mini [39] OpenAI 2025
GPT-5. [39] OpenAI. https://openai.Where the paper says it
Where the paper says it
[41] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. p. 7 https://openai. [41] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh,
Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark,
Gretchen Krueger, and Ilya Sutskever. [41] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh,
Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark,
Gretchen Krueger, and Ilya Sutskever. Learning Transferable Visual Models
From Natural Language Supervision. Learning Transferable Visual Models
From Natural Language Supervision. In Proceedings of the 38th International
Conference on Machine Learning (Proceedings of Machine Learning Research,
Vol. In Proceedings of the 38th International
Conference on Machine Learning (Proceedings of Machine Learning Research,
Vol. 139), Marina Meila and Tong Zhang (Eds.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
[52] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. p. 7 European
Journal of Cultural Management and Policy (2025). [52] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan
Salakhudinov, Rich Zemel, and Yoshua Bengio. [52] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan
Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, Attend and Tell:
Neural Image Caption Generation with Visual Attention. Show, Attend and Tell:
Neural Image Caption Generation with Visual Attention. In Proceedings of the
32nd International Conference on Machine Learning (Proceedings of Machine
Learning Research, Vol. In Proceedings of the
32nd International Conference on Machine Learning (Proceedings of Machine
Learning Research, Vol. 37), Francis Bach and David Blei (Eds.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
[6] Anthropic. p. 4 4-mini [40] OpenAI 2026
Claude Haiku 4. 5 [6] Anthropic 2025
Table 2: Overview of evaluated vision-language models, including openweight and close-source models. [6] Anthropic. Claude 4 Model Family.Where the paper says it
Where the paper says it
[38] National Cultural Heritage Administration of China. p. 7 AI: A Museum Planning Toolkit. p. 7 [37] Oonagh Murphy and Elena Villaespesa. AI: A Museum Planning Toolkit. [38] National Cultural Heritage Administration of China. Classification and
Codes for Cultural Relics. Classification and
Codes for Cultural Relics. Cultural Relics Press, Beijing.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
[43] Matteo Stefanini, Marcella Cornia, Lorenzo Baraldi, Massimiliano Corsini, and Rita Cucchiara. p. 7 Association for Computational Linguistics, 42–50. [43] Matteo Stefanini, Marcella Cornia, Lorenzo Baraldi, Massimiliano Corsini, and
Rita Cucchiara. [43] Matteo Stefanini, Marcella Cornia, Lorenzo Baraldi, Massimiliano Corsini, and
Rita Cucchiara. Artpedia: A New Visual-Semantic Dataset with Visual and
Contextual Sentences in the Artistic Domain. Artpedia: A New Visual-Semantic Dataset with Visual and
Contextual Sentences in the Artistic Domain. In Proceedings of the International
Conference on Image Analysis and Processing (ICIAP 2019).Where the paper says it
Where the paper says it
Where the paper says it
[45] Elena Villaespesa and Seth Crider. p. 7 org/abs/2510. p. 7 CV] https://arxiv. org/abs/2510. 11178
[45] Elena Villaespesa and Seth Crider. Computer Vision Tagging the Metropolitan Museum of Art’s Collection: A Comparison of Three Systems. Computer Vision Tagging the Metropolitan Museum of Art’s Collection: A Comparison of Three Systems. Journal on
Computing and Cultural Heritage (2021).Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
[47] Elena Villaespesa and Oonagh Murphy. p. 7 IEEE Transactions on Computational
Social Systems (2024). [37] Oonagh Murphy and Elena Villaespesa. p. 7 IEEE Transactions on Computational
Social Systems (2024). [37] Oonagh Murphy and Elena Villaespesa. [47] Elena Villaespesa and Oonagh Murphy. This is not an apple! This is not an apple! Benefits
and challenges of applying computer vision to museum collections. Benefits
and challenges of applying computer vision to museum collections. Museum
Management and Curatorship (2021).Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
[53] Haorui Yu, Xuehang Wen, Fengrui Zhang, and Qiufeng Yi. p. 7 press/v37/xuc15. html
[53] Haorui Yu, Xuehang Wen, Fengrui Zhang, and Qiufeng Yi. html
[53] Haorui Yu, Xuehang Wen, Fengrui Zhang, and Qiufeng Yi. A Multicultural Vision-Language Benchmark for Evaluating Cultural Understanding. A Multicultural Vision-Language Benchmark for Evaluating Cultural Understanding. arXiv
preprint arXiv:2601. arXiv
preprint arXiv:2601. 07986 (2026).Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
[54] Cheng Zhang, Hongxia Xie, Bin Wen, Songhan Zuo, Ruoxuan Zhang, and WenHuang Cheng. p. 7 org/html/2601. 07986v3
[54] Cheng Zhang, Hongxia Xie, Bin Wen, Songhan Zuo, Ruoxuan Zhang, and WenHuang Cheng. 07986v3
[54] Cheng Zhang, Hongxia Xie, Bin Wen, Songhan Zuo, Ruoxuan Zhang, and WenHuang Cheng. EmoArt: A Multidimensional Dataset for Emotion-Aware
Artistic Generation. EmoArt: A Multidimensional Dataset for Emotion-Aware
Artistic Generation. In Proceedings of the 33rd ACM International Conference Appear2Meaning: A Cross-Cultural Benchmark for Structured Cultural Metadata Inference from ImagesACM MM ’26, November 10–14, 2026, Rio de Janeiro, Brazil
on Multimedia (MM ’25). ACM, Dublin, Ireland, 12644–12650.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
[18] Nicola Fanelli, Gennaro Vessio, and Giovanna Castellano. p. 7 European Journal of Cultural Management and
Policy (2024). [18] Nicola Fanelli, Gennaro Vessio, and Giovanna Castellano. [18] Nicola Fanelli, Gennaro Vessio, and Giovanna Castellano. ARTSEEK:
Deep Artwork Understanding via Multimodal In-Context Reasoning and Late
Interaction Retrieval. ARTSEEK:
Deep Artwork Understanding via Multimodal In-Context Reasoning and Late
Interaction Retrieval. arXiv preprint arXiv:2507. arXiv preprint arXiv:2507. 21917 (July 2025).Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
[19] Marco Fiorucci et al. p. 7 21917 (July 2025). [19] Marco Fiorucci et al. [19] Marco Fiorucci et al. Machine Learning for Cultural Heritage: A Survey. Machine Learning for Cultural Heritage: A Survey. Pattern Recognition Letters (2020). Pattern Recognition Letters (2020). https://doi.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
[44] Bryan Chen Zhengyu Tan, Zheng Weihua, Zhengyuan Liu, Nancy F. p. 7 In Proceedings of the International
Conference on Image Analysis and Processing (ICIAP 2019). [44] Bryan Chen Zhengyu Tan, Zheng Weihua, Zhengyuan Liu, Nancy F. [44] Bryan Chen Zhengyu Tan, Zheng Weihua, Zhengyuan Liu, Nancy F. Chen,
Hwaran Lee, Kenny Tsu Wei Choo, and Roy Ka-Wei Lee. Chen,
Hwaran Lee, Kenny Tsu Wei Choo, and Roy Ka-Wei Lee. BLEnD-Vis:
Benchmarking Multimodal Cultural Understanding in Vision Language Models. BLEnD-Vis:
Benchmarking Multimodal Cultural Understanding in Vision Language Models. arXiv:2510. arXiv:2501. p. 7 CultureVLM: Characterizing
and Improving Cultural Understanding of Vision-Language Models for over 100
Countries. arXiv:2501. arXiv:2501. AI] https://arxiv.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
[36] Yue Lu, Chao Guo, Xingyuan Dai, and Fei-Yue Wang. p. 7 02265 (2019). [36] Yue Lu, Chao Guo, Xingyuan Dai, and Fei-Yue Wang. [36] Yue Lu, Chao Guo, Xingyuan Dai, and Fei-Yue Wang. ArtCap: A Dataset
for Image Captioning of Fine Art Paintings. ArtCap: A Dataset
for Image Captioning of Fine Art Paintings. IEEE Transactions on Computational
Social Systems (2024).Where the paper says it
Where the paper says it
Where the paper says it
[51] Paula Westenberger and Despoina Farmaki. p. 7 press/v162/wang22al. html
[51] Paula Westenberger and Despoina Farmaki. html
[51] Paula Westenberger and Despoina Farmaki. Artificial intelligence for cultural heritage research: the challenges in UK copyright law and policy. Cultural work, wellbeing, and AI. European Journal of Cultural Management and Policy (2025).Where the paper says it
Where the paper says it
Where the paper says it
[34] Shudong Liu, Yiqiao Jin, Cheng Li, Derek F. p. 7 , Red Hook, NY, USA, Article 1516, 25 pages. [34] Shudong Liu, Yiqiao Jin, Cheng Li, Derek F. [34] Shudong Liu, Yiqiao Jin, Cheng Li, Derek F. Wong, Qingsong Wen, Lichao Sun,
Haipeng Chen, Xing Xie, and Jindong Wang. Wong, Qingsong Wen, Lichao Sun,
Haipeng Chen, Xing Xie, and Jindong Wang. CultureVLM: Characterizing
and Improving Cultural Understanding of Vision-Language Models for over 100
Countries. AI] https://arxiv. org/abs/2501.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
[37] Oonagh Murphy and Elena Villaespesa. 2020. AI: A Museum Planning Toolkit.
[48] Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. p. 7 Museum
Management and Curatorship (2021). [48] Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. [48] Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show
and Tell: A Neural Image Caption Generator. Bottom-Up and Top-Down Attention for
Image Captioning and Visual Question Answering. In Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition (CVPR).Where the paper says it
Where the paper says it
Where the paper says it
[50] Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. p. 7 org/abs/2409. 12191
[50] Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma,
Chang Zhou, Jingren Zhou, and Hongxia Yang. 12191
[50] Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma,
Chang Zhou, Jingren Zhou, and Hongxia Yang. OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning
Framework.Where the paper says it
Where the paper says it
[21] Clifford Geertz. p. 1 Keywords
cultural heritage, vision-language models, multimodal understanding, metadata inference, benchmark dataset, LLM-as-Judge
1 Introduction
“Man is an animal suspended in webs of significance he himself has
spun. ”
— Clifford Geertz, The Interpretation of Cultures (1973) [21]
What about vision-language models? [21] Clifford Geertz. The Interpretation of Cultures: Selected Essays. The Interpretation of Cultures: Selected Essays. Basic
Books, New York.Where the paper says it
Where the paper says it
Where the paper says it
Summary of Error Case Studies
Taken together, these cases suggest that the observed error patterns are shaped not only by model capability but also by interactions among training priors, dataset composition, visual signal quality, and evaluation constraints. First, regional performance differences are unlikely to reflect a single factor. p. 1 Artpedia [43] provides visual and
contextual descriptions for paintings but is limited to a single category and lacks cross-cultural variation. MLF [32] and CArt15K [55]
focus on ceramics with factor-driven or metaphor-enriched annotations, yet remain restricted to specific object types and cultural
contexts. First, an annotator selects 50 artifacts per culture–type
combination based on images and metadata. Second, a different
annotator verifies cultural region and object type assignments. These findings suggest that current VLMs rely heavily on surfacelevel visual cues and memorized patterns, with limited ability to
generalize cultural knowledge across regions. 3 Error Analysis
We categorise model errors into four types, corresponding to the
patterns observed across attributes and cultural contexts. Culture is not strictly bounded by
geography, and this approximation may obscure intra-cultural diversity and reinforce reductive or essentialist interpretations. Inferring
non-observable cultural attributes (e.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Final Synthesis
The central challenge is not only in detecting relevant visual features but also in aligning them with provenance-constrained metadata without over-reliance on prevalent stylistic associations. Outputs should be interpreted as hypothesis-like descriptions rather than authoritative attributions. p. 6 Inferring
non-observable cultural attributes (e. , title, culture, period, origin,
creator) from images introduces epistemic uncertainty and may produce incorrect or overconfident outputs, which should not be treated
as authoritative judgments or used without expert validation in realworld applications, particularly in museum, educational, or heritage
contexts.Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Improve structured metadata inference by moving beyond simple caption generation to a formal structured prediction task using an LLM-as-a-Judge framework that measures semantic alignment with reference annotations across four cultural regions.
-
Enhance model performance by addressing cross-cultural misattribution and style transfer confusion through targeted training or fine-tuning that reduces reliance on
learned stylistic priors
when distinguishing between visual similarity and historical provenance. -
Develop systems capable of identifying object-type recognition without functional understanding by incorporating knowledge bases that link coarse visual forms to historically specific object labels, preventing the model from producing
superficially plausible but semantically wrong titles.
-
Increase robustness against period compression by training models to infer temporal context from culturally consistent metadata rather than anchoring predictions to familiar stylistic patterns, thereby preventing artifacts like shifting American ceramics into European production timelines.
-
Improve creator attribution accuracy by incorporating mechanisms that integrate memorized maker associations with surrounding cultural context in a way that ensures
coherent multi-attribute predictions,
rather than allowing models to succeed on creator attribution without corresponding correct culture, period, and origin fields. -
Refine evaluation protocols to distinguish between
semantic plausibility and exact metadata matching
when assessing outputs like Case Study F, ensuring that coherent but non-canonical predictions are not incorrectly penalized under strict field-by-field comparison rules.
Abstract
Museum records express how people understand heritage objects through cultural and historical relations. Recovering these human attributions from images is a technical challenge for vision-language models (VLMs). We introduce Appear2Meaning, a cross-cultural benchmark of 750 museum objects for image-only recovery of title, culture, period, origin, and creator. Its evaluation framework distinguishes field-level from complete-record compatibility and uses human validation for semantic judgement. Across nine VLMs, mean field accuracy is 36.16--49.82%; 7.20--17.02% of records have all four contextual fields compatible, and 2.13--7.87% have all five, including title. For the two tested models, image-only input yields 12.6--32.2 percentage points higher reference compatibility for culture, period, and origin than a museum-and-title-only control.
Sources
- ArtSeek: Deep artwork understanding via multimodal in-context reasoning and late interaction retrieval
- DenseAnnotate: Enabling Scalable Dense Caption Collection for Images and 3D Scenes via Spoken Descriptions
- CultureVLM: Characterizing and Improving Cultural Understanding of Vision-Language Models for over 100 Countries
- ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks
- BLEnD-Vis: Benchmarking Multimodal Cultural Understanding in Vision Language Models
- Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution
- VULCA-Bench: A Multicultural Vision-Language Benchmark for Evaluating Cultural Understanding
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs