Appear2Meaning: A Cross-Cultural Benchmark for Structured Cultural Metadata Inference from Images

arXiv:2604.07338 · cs.CV, cs.CL, cs.MM · Submitted 2026-04-08 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Appear2Meaning: A Cross-Cultural Benchmark for Structured Cultural Metadata Inference from Images".

Jane: Here's the gist of the paper. Recent advances in vision-language models (VLMs) have improved image captioning for cultural heritage; however, inferring structured cultural metadata from visual input remains underexplored.

Tom: This work introduces a multi-category, cross-cultural benchmark for evaluating structured metadata inference from image-only input.

Jane: The results show that models capture fragmented signals and exhibit substantial performance variation across cultures and metadata types.

Tom: These findings highlight the limitations of current VLMs in structured cultural metadata inference beyond visual perception.

Paper summary: Tom: So, wrapping up "Appear2Meaning: A Cross-Cultural Benchmark for Structured Cultural Metadata Inference from Images," the authors are showing us that these vision-language models can definitely pick up some cultural signals from images, but getting the exact metadata right—culture, period, and origin—is still really tough.

Jane: The paper uses this benchmark to prove that performance isn't uniform; it varies a lot depending on whether you're looking at East Asia or Europe or the Americas. This variation points to the fact that these models are relying on shortcuts instead of deep reasoning about cultural context.

Lu: The authors conclude that while current vision-language models can capture partial cultural signals, exact metadata inference remains challenging, especially for culture, period, and origin. They suggest that many predictions rely on stylistic shortcuts and memorized creator associations rather than robust multi-attribute reasoning.

Meng: If you're deploying a model like this in production right now, you need to be aware of those limitations because the performance is inconsistent across different cultural contexts. It’s not a one-size-fits-all accuracy number.

Lalam: The findings suggest that we should treat the outputs as hypothesis-like descriptions rather than authoritative judgments about what a piece of heritage actually is, especially when it comes to deep cultural attribution.

Tom: So, in short, the challenge isn't just seeing the visual features; it’s aligning those features with provenance-constrained metadata without over-relying on common stylistic associations.

Jane: This work provides a crucial cross-cultural benchmark for evaluating how well these models perform when they are asked to infer complex cultural knowledge from images alone.

Lu: Future work is going toward extending this benchmark to finergrained cultural distinctions and exploring ways to integrate external knowledge sources into the models to improve that coherent multi-attribute metadata inference.

Meng: For deployment, we need more than just a good score on this benchmark; we need a system that can handle that performance variation robustly across all the different cultures it might encounter.

Lalam: So, while they show what's possible now, the authors clearly indicate that for truly reliable cultural metadata inference, we still have some significant work ahead before these models become fully trustworthy.

Conclusion: Tom: So, this paper, "Appear2Meaning," is introducing a way to test how well vision language models can figure out structured cultural stuff from just pictures.

Jane: It's testing if these models can predict things like the culture, the time period, and where something came from, based only on what they see in an image.

Tom: The authors are using a multi-category benchmark to see how different cultures and different types of cultural labels affect model performance.

Lu: They’re formalizing it as a structured prediction problem where the model has to infer specific metadata attributes like culture, period, origin, and creator from just the visual input.

Meng: The setup is interesting because they are using publicly available data from places like the Getty and Metropolitan Museum of Art to build this dataset.

Lalam: They curated it through rule-based filtering and two stages of human verification to get a balanced cross-cultural set, focusing on things that aren't always obvious just by looking at the picture.

Tom: The results show that models capture fragmented signals and there's a lot of variation across different cultures and metadata types.

Jane: That means the models aren't performing equally well everywhere; they struggle in some places more than others.

Lu: They also found that the evaluation uses an LLM as a judge framework to check for semantic alignment between what the model predicts and the actual reference annotations, which is better than just checking if words match up.

Meng: The error analysis is where it gets really useful because they categorize mistakes into four types, like cross-cultural misattribution or temporal inference errors.

Tom: Apparently, a common mistake is when models map visual style to a more familiar Eurocentric category even when the object isn't from there.

Jane: That’s a big deal for us because it shows these models are relying on learned associations instead of actual deep understanding of cultural context.

Lu: They also pointed out that creator attribution can be successful sometimes, but only if the model has memorized that specific creator rather than reasoning about the surrounding cultural context.

Meng: It’s a lot of work to get those kinds of nuanced inferences when you're trying to ship a model to production, especially when you don't have perfect knowledge of what the data implies.

Tom: The authors conclude that while current models can grab some partial cultural signals, getting exact metadata right for culture and period is still really hard.

Jane: They stress that these outputs should be seen as hypothesis-like descriptions rather than authoritative facts about the object's history.

Lu: Future work involves creating a finer benchmark for those cultural distinctions and thinking about how to bring in external knowledge sources to make the multi-attribute inference more coherent.

Meng: For us, that means we need better ways to test if these models are actually robust enough for real-world use cases where they have to handle all this variability.

Tom: So, the main point here is that we've got these tools that can see images and guess some cultural metadata, but the guessing part is where the real technical challenge lies.

Jane: And the next thing we need to figure out is how to make sure those guesses are reliable enough for people who actually want to trust them with cultural heritage information.

University of Manchester · School of Artificial Intelligence, Wuhan University · Getty Conservation Institute

cs.CV, cs.CL, cs.MM

Submitted: 2026-04-08

Updated: 2026-11-15

Code: https://github.com/QwenLM/Qwen-VL

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 75/100

The gist: Recent advances in vision-language models (VLMs) have improved image captioning for cultural heritage; however, inferring structured cultural metadata from visual input remains underexplored.

Key concepts

Structured Metadata Inference
This is the task of forcing an AI to predict specific, organized pieces of information about an object—such as its culture or creator—using only a picture. It treats heritage understanding as a structured prediction problem where the model must output predefined labels from museum schemas.
LLM-as-a-Judge Framework
Instead of using traditional metrics, this method uses a powerful language model (like GPT) to evaluate the AI's predictions. The judge assesses semantic alignment between what the model predicted and the correct reference metadata, focusing on meaning rather than just matching words.
Cross-Cultural Misattribution
This is a common error where models incorrectly assign cultural labels based on visual style. For example, a model might map an object's artistic style to a more familiar European category instead of its true origin, showing that visual cues often trigger biased associations.
Fragmented Signals
Models often capture only small parts of the necessary information from an image. This means they can recognize some objects but fail to connect the visual evidence robustly enough to infer complex, multi-attribute cultural details like precise historical periods or origins.

Terminology

Summary

Recent advances in vision-language models (VLMs) have improved image captioning for cultural heritage; however, inferring structured cultural metadata from visual input remains underexplored. p. 1

Where the paper says it

Abstract Recent advances in vision-language models (VLMs) have improved image captioning for cultural heritage. However, inferring structured cultural metadata (e.

This work introduces a multi-category, cross-cultural benchmark for evaluating structured metadata inference from image-only input. p. 6
Where the paper says it

This gap indicates that models capture individual attributes but struggle to produce coherent multi-attribute predictions across cultural contexts. 5 Conclusion and Future Work We introduced a cross-cultural benchmark for evaluating visionlanguage models on structured cultural metadata inference from image-only input.

The results show that models capture fragmented signals and exhibit substantial performance variation across cultures and metadata types. p. 1
Where the paper says it

To assess cultural reasoning, we report exact-match, partial-match, and attribute-level accuracy across cultural regions. Results show that models capture fragmented signals and exhibit substantial performance variation across cultures and metadata types, leading to inconsistent and weakly grounded predictions.

These findings highlight the limitations of current VLMs in structured cultural metadata inference beyond visual perception. p. 1
Where the paper says it

Results show that models capture fragmented signals and exhibit substantial performance variation across cultures and metadata types, leading to inconsistent and weakly grounded predictions. These findings highlight the limitations of current VLMs in structured cultural metadata inference beyond visual perception.

Pruning redundant reasoning tokens

The paper formalizes heritage understanding as a structured prediction problem where models are prompted to infer structured metadata attributes from visual input only. The goal is to predict structured metadata M =

mculture,mperiod,morigin,mcreator where each attribute corresponds to a well-defined label grounded in museum metadata schemas. The evaluation objective is M ≈ M ˆ ∗ using an LLM-as-a-Judge framework that measures semantic alignment with reference annotations. p. 1

Where the paper says it

, creator, origin, period) from visual input remains underexplored. We introduce a multi-category, cross-cultural benchmark for this task and evaluate VLMs using an LLM-as-aJudge framework that measures semantic alignment with reference annotations.

How the training signal is built

The dataset is curated from publicly available records of the Getty Art Collections and the Metropolitan Museum of Art Open Access collection. The curation process involves rule-based filtering over metadata fields followed by two-stage human verification to build a balanced cross-cultural dataset. p. 3

Where the paper says it

ACM MM ’26, November 10–14, 2026, Rio de Janeiro, Brazil Jiang et al. Figure 2: Data curation pipeline combining rule-based filtering and human verification to build a balanced cross-cultural dataset.

The benchmark targets non-observable, culturally grounded attributes and enables fine-grained analysis of performance and bias across cultural contexts. p. 2
Where the paper says it

As shown in Figure 1, models predict structured metadata from visual input and are evaluated using an LLM-as-Judge against normalized and raw museum annotations. The benchmark targets non-observable, culturally grounded attributes and enables fine-grained analysis of performance and bias across cultural contexts.

Where the gains come from

The evaluation uses an LLM-as-a-Judge framework to assess attribute-level correctness by prioritizing semantic alignment between predicted and reference metadata over lexical similarity. Performance varies across cultural regions, with higher accuracy in East Asia and lower accuracy in Europe and the Americas. p. 1

Where the paper says it

At the attribute level, models perform better on title and creator, while culture, period, and origin remain more difficult. Performance also varies across cultural regions, with higher accuracy in East Asia and lower accuracy in Europe and the Americas.

Open-weight Qwen models demonstrate competitive or superior performance compared to closed-source models in terms of partial correctness. p. 4
Where the paper says it

609) and Qwen-VL-Max (0. Notably, open-weight Qwen models demonstrate competitive or superior performance compared to closed-source models in terms of partial correctness.

Error Analysis

The error analysis categorizes model errors into four types corresponding to patterns observed across attributes and cultural contexts. The most common failure is cross-cultural misattribution, often by mapping visual style to a more familiar Eurocentric category. p. 5

Where the paper says it

Cross-cultural misattribution. The most common failure is assigning an artifact to the wrong cultural contexts, often by mapping visual style to a more familiar Eurocentric category.

Second failure mode is object-type recognition without functional understanding, leading to superficially plausible but semantically wrong titles. p. 5
Where the paper says it

A second failure mode is that models recognize coarse visual form but miss the historically specific object type. This leads to superficially plausible but semantically wrong titles.

Third pattern is incorrect temporal inference driven by broad stylistic priors, where artifacts are relocated into entirely different historical and cultural periods. p. 5
Where the paper says it

Period compression and stylistic over-anchoring. A third pattern is incorrect temporal inference driven by broad stylistic priors.

Finally, creator memorization versus holistic reasoning shows partial success on creator attribution without corresponding success on the surrounding cultural context. p. 5
Where the paper says it

Creator memorization versus holistic reasoning. A fourth pattern is partial success on creator attribution without corresponding

Case Studies and Error Analysis

Case study A illustrates a recurring pattern of cross-cultural misattribution across multiple models, where predictions are grounded in plausible stylistic associations rather than strictly incorrect reasoning. Case study B highlights a pattern of style-driven cross-cultural confusion where visual resemblance is associated with predictions shifting the object’s cultural attribution toward European contexts. p. 9

Where the paper says it

1 Case Study A: Systematic Cross-Cultural Misattribution Object ID: 1055_Butter Pat Figure 3: Example images for Object ID 1055_Butter Pat. Ground Truth: • Title: Butter Pat • Culture: American • Period: 1885 • Creator: Union Porcelain Works Representative Predictions: • Claude Haiku 4.

Case study C demonstrates partial object recognition without accurate cultural attribution, where functional category is recoverable but cultural metadata is systematically shifted toward European contexts. p. 1
Where the paper says it

MLF [32] and CArt15K [55] focus on ceramics with factor-driven or metaphor-enriched annotations, yet remain restricted to specific object types and cultural contexts. Painting-centric datasets such as GalleryGPT [9], ArtCap [36], DEArt [42], and ARTSEEK [18] emphasize visual analysis and stylistic recognition without modeling structured cultural metadata.

The analysis suggests that errors are shaped by the interaction between dataset composition, visual ambiguity, and model priors rather than as definitive misinterpretations of cultural identity. p. 5
Where the paper says it

This pattern is especially frequent for objects in the Americas. This suggests that models rely on visual similarity and training data priors rather than accurate inference of cultural metadata.

Conclusion and Future Work

The work concludes that while current models can capture partial cultural signals, exact metadata inference remains challenging, especially for culture, period, and origin. Error analysis reveals that many predictions rely on stylistic shortcuts and memorized creator associations rather than robust multi-attribute reasoning. p. 6

Where the paper says it

Our results show that, while current models can capture partial cultural signals, exact metadata inference remains challenging, especially for culture, period, and origin. Error analysis further reveals that many predictions rely on stylistic shortcuts, culturally dominant priors, and memorised creator associations, rather than robust multi-attribute reasoning.

Future work will extend the benchmark toward finergrained cultural distinctions and explore integrating external knowledge sources to improve coherent multi-attribute metadata inference. p. 6
Where the paper says it

Future work will therefore extend the benchmark toward finergrained cultural distinctions, broader object categories, and larger and more balanced sample sizes, enabling more targeted and representative evaluation across diverse cultural contexts. We will also explore integrating external knowledge sources, including retrievalaugmented and ontology-grounded approaches, as well as structured mechanisms that connect visual predictions to museum knowledge bases, to improve coherent multi-attribute metadata inference.

Responsible use should prioritize human oversight, especially in culturally sensitive contexts. p. 6
Where the paper says it

To mitigate these risks, we encourage incorporating domain expertise, expanding culturally diverse datasets, and transparently reporting model limitations. Responsible use should prioritise human oversight, especially in culturally sensitive contexts, and avoid treating automated predictions as definitive cultural interpretations.

Ethical Considerations

This work uses publicly available cultural heritage data from museum collections under open-access policies and releases code and annotations under the MIT License. Cultural heritage data reflect historical, institutional, and curatorial biases; consequently, models trained or evaluated on such data may inherit and amplify these biases. p. 6

Where the paper says it

, CC0 or CC BY), and releases code and annotations under the MIT License. Cultural heritage data reflect historical, institutional, and curatorial biases; consequently, models trained or evaluated on such data may inherit and amplify these biases, as evidenced by performance disparities across cultural regions.

Inferring non-observable cultural attributes introduces epistemic uncertainty and may produce incorrect or overconfident outputs which should not be treated as authoritative judgments. p. 6
Where the paper says it

Inferring non-observable cultural attributes (e. , title, culture, period, origin, creator) from images introduces epistemic uncertainty and may produce incorrect or overconfident outputs, which should not be treated as authoritative judgments or used without expert validation in realworld applications, particularly in museum, educational, or heritage contexts.

The use of LLM-as-a-Judge introduces evaluation bias, as judgments depend on the knowledge and assumptions encoded in the evaluator model. p. 6
Where the paper says it

, title, culture, period, origin, creator) from images introduces epistemic uncertainty and may produce incorrect or overconfident outputs, which should not be treated as authoritative judgments or used without expert validation in realworld applications, particularly in museum, educational, or heritage contexts. The use of LLM-as-a-Judge introduces evaluation bias, as judgments depend on the knowledge and assumptions encoded in the evaluator model, potentially reinforcing dominant cultural narratives or overlooking alternative interpretations.

References

[1] Manar Abu Talib, Iman Ibrahim, and Manar Anwer Abusirdaneh. 2026. Reusability and Benchmarking Potential of Architectural Cultural Heritage Datasets for Generative AI: An Analytical Study. p. 6

Where the paper says it

References [1] Manar Abu Talib, Iman Ibrahim, and Manar Anwer Abusirdaneh. Reusability and Benchmarking Potential of Architectural Cultural Heritage Datasets for Generative AI: An Analytical Study.

Expert Systems With Applications (2026). p. 6
Where the paper says it

Reusability and Benchmarking Potential of Architectural Cultural Heritage Datasets for Generative AI: An Analytical Study. Expert Systems With Applications (2026).

Published online 16 January 2026. p. 6
Where the paper says it

Expert Systems With Applications (2026). Published online 16 January 2026.

[5] Hannah Andrews and Aurora Hawcroft. p. 6

Where the paper says it

In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). [5] Hannah Andrews and Aurora Hawcroft.

2024. Articulating arts-led AI: artists and technological development in cultural policy. p. 6
Where the paper says it

[5] Hannah Andrews and Aurora Hawcroft. Articulating arts-led AI: artists and technological development in cultural policy.

European Journal of Cultural Management and Policy (2024). p. 6
Where the paper says it

Articulating arts-led AI: artists and technological development in cultural policy. European Journal of Cultural Management and Policy (2024).

[7] Murtha Baca (Ed. p. 6

Where the paper says it

https://www. p. 3

Where the paper says it

1 under an image-only setting. For each input image, models generate structured metadata predictions, which are subsequently evaluated 1 https://www.

p. 3
Where the paper says it

1 under an image-only setting. For each input image, models generate structured metadata predictions, which are subsequently evaluated 1 https://www.

[7] Murtha Baca (Ed.

). 2016. Introduction to Metadata (3 ed. p. 1
Where the paper says it

CCS Concepts • Computing methodologies → Scene understanding; Information extraction; • Applied computing → Arts and humanities. Keywords cultural heritage, vision-language models, multimodal understanding, metadata inference, benchmark dataset, LLM-as-Judge 1 Introduction “Man is an animal suspended in webs of significance he himself has spun.

). Getty Research Institute, Los Angeles. p. 6
Where the paper says it

Introduction to Metadata (3 ed. Getty Research Institute, Los Angeles.

https://www.getty.edu/publications/intrometadata/. p. 6
Where the paper says it

https://www. edu/publications/intrometadata/ [8] Longju Bai, Angana Borah, Oana Ignat, and Rada Mihalcea.

[9] Yi Bin, Wenhao Shi, Yujuan Ding, Zhiqiang Hu, Zheng Wang, Yang Yang, SeeKiong Ng, and Heng Tao Shen. p. 6

Where the paper says it

Association for Computational Linguistics. [9] Yi Bin, Wenhao Shi, Yujuan Ding, Zhiqiang Hu, Zheng Wang, Yang Yang, SeeKiong Ng, and Heng Tao Shen.

2024. GalleryGPT: Analyzing Paintings with Large Multimodal Models. p. 6
Where the paper says it

[9] Yi Bin, Wenhao Shi, Yujuan Ding, Zhiqiang Hu, Zheng Wang, Yang Yang, SeeKiong Ng, and Heng Tao Shen. GalleryGPT: Analyzing Paintings with Large Multimodal Models.

In Proceedings of the 32nd ACM International Conference on Multimedia (MM ’24). p. 6
Where the paper says it

GalleryGPT: Analyzing Paintings with Large Multimodal Models. In Proceedings of the 32nd ACM International Conference on Multimedia (MM ’24).

ACM, Melbourne, Australia. p. 6
Where the paper says it

In Proceedings of the 32nd ACM International Conference on Multimedia (MM ’24). ACM, Melbourne, Australia.

[16] Martin Doerr. p. 7

Where the paper says it

, Red Hook, NY, USA, Article 2142, 18 pages. [16] Martin Doerr.

2003. The CIDOC Conceptual Reference Model: An Ontological Approach to Semantic Interoperability of Metadata. p. 7
Where the paper says it

[16] Martin Doerr. The CIDOC Conceptual Reference Model: An Ontological Approach to Semantic Interoperability of Metadata.

International Journal of Human-Computer Studies 43, 5 (2003), 75–92. p. 7
Where the paper says it

The CIDOC Conceptual Reference Model: An Ontological Approach to Semantic Interoperability of Metadata. International Journal of Human-Computer Studies 43, 5 (2003), 75–92.

doi:10.1016/j.ijhcs.2003.10.003.

[28] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. p. 7

Where the paper says it

org, Article 814, 13 pages. [28] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi.

2022. BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation. p. 7
Where the paper says it

[28] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation.

In Proceedings of the 39th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. p. 7
Where the paper says it

BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation. In Proceedings of the 39th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol.

p. 7
Where the paper says it

BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation. In Proceedings of the 39th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol.

162), Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (Eds. p. 7
Where the paper says it

In Proceedings of the 39th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 162), Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (Eds.

p. 7
Where the paper says it

In Proceedings of the 39th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 162), Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (Eds.

). PMLR.

[39] OpenAI. p. 4

Where the paper says it

Appear2Meaning: A Cross-Cultural Benchmark for Structured Cultural Metadata Inference from ImagesACM MM ’26, November 10–14, 2026, Rio de Janeiro, Brazil Model Organization Release Time Open-Weight Models Qwen-VL-Max [22] Alibaba 2024 Qwen3-VL-Plus [23] Alibaba 2025 Qwen3-VL-Flash [23] Alibaba 2025 Qwen3-VL-8B-Instruct [23] Alibaba 2025 Qwen3-VL-32B-Instruct [23] Alibaba 2025 Pixtral-12B [2] Mistral AI 2024 Close-Source Models GPT-4. 1-mini [39] OpenAI 2025 GPT-5.

2025. GPT-4.1 Mini. https://openai. p. 7
Where the paper says it

[39] OpenAI. https://openai.

com.

[41] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. p. 7

Where the paper says it

https://openai. [41] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.

2021. Learning Transferable Visual Models From Natural Language Supervision. p. 7
Where the paper says it

[41] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning Transferable Visual Models From Natural Language Supervision.

In Proceedings of the 38th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. p. 7
Where the paper says it

Learning Transferable Visual Models From Natural Language Supervision. In Proceedings of the 38th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol.

139), Marina Meila and Tong Zhang (Eds. p. 7
Where the paper says it

In Proceedings of the 38th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 139), Marina Meila and Tong Zhang (Eds.

). PMLR.

[52] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. p. 7

Where the paper says it

European Journal of Cultural Management and Policy (2025). [52] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio.

2015. Show Attend and Tell: Neural Image Caption Generation with Visual Attention. p. 7
Where the paper says it

[52] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, Attend and Tell: Neural Image Caption Generation with Visual Attention.

In Proceedings of the 32nd International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. p. 7
Where the paper says it

Show, Attend and Tell: Neural Image Caption Generation with Visual Attention. In Proceedings of the 32nd International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol.

37), Francis Bach and David Blei (Eds. p. 7
Where the paper says it

In Proceedings of the 32nd International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 37), Francis Bach and David Blei (Eds.

). PMLR.

[6] Anthropic. p. 4

Where the paper says it

4-mini [40] OpenAI 2026 Claude Haiku 4. 5 [6] Anthropic 2025 Table 2: Overview of evaluated vision-language models, including openweight and close-source models.

2025. Claude 4 Model Family. p. 6
Where the paper says it

[6] Anthropic. Claude 4 Model Family.

https://www.anthropic.com.

[38] National Cultural Heritage Administration of China. p. 7

Where the paper says it

AI: A Museum Planning Toolkit. p. 7

Where the paper says it

[37] Oonagh Murphy and Elena Villaespesa. AI: A Museum Planning Toolkit.

[38] National Cultural Heritage Administration of China.

2008. Classification and Codes for Cultural Relics. p. 7
Where the paper says it

[38] National Cultural Heritage Administration of China. Classification and Codes for Cultural Relics.

Cultural Relics Press, Beijing. p. 7
Where the paper says it

Classification and Codes for Cultural Relics. Cultural Relics Press, Beijing.

[43] Matteo Stefanini, Marcella Cornia, Lorenzo Baraldi, Massimiliano Corsini, and Rita Cucchiara. p. 7

Where the paper says it

Association for Computational Linguistics, 42–50. [43] Matteo Stefanini, Marcella Cornia, Lorenzo Baraldi, Massimiliano Corsini, and Rita Cucchiara.

2019. Artpedia: A New Visual-Semantic Dataset with Visual and Contextual Sentences in the Artistic Domain. p. 7
Where the paper says it

[43] Matteo Stefanini, Marcella Cornia, Lorenzo Baraldi, Massimiliano Corsini, and Rita Cucchiara. Artpedia: A New Visual-Semantic Dataset with Visual and Contextual Sentences in the Artistic Domain.

In Proceedings of the International Conference on Image Analysis and Processing (ICIAP 2019). p. 7
Where the paper says it

Artpedia: A New Visual-Semantic Dataset with Visual and Contextual Sentences in the Artistic Domain. In Proceedings of the International Conference on Image Analysis and Processing (ICIAP 2019).

Springer.

[45] Elena Villaespesa and Seth Crider. p. 7

Where the paper says it

org/abs/2510. p. 7

Where the paper says it

CV] https://arxiv. org/abs/2510.

11178 [45] Elena Villaespesa and Seth Crider.

2021. Computer Vision Tagging the Metropolitan Museum of Art’s Collection: A Comparison of Three Systems. p. 7
Where the paper says it

11178 [45] Elena Villaespesa and Seth Crider. Computer Vision Tagging the Metropolitan Museum of Art’s Collection: A Comparison of Three Systems.

Journal on Computing and Cultural Heritage (2021). p. 7
Where the paper says it

Computer Vision Tagging the Metropolitan Museum of Art’s Collection: A Comparison of Three Systems. Journal on Computing and Cultural Heritage (2021).

[47] Elena Villaespesa and Oonagh Murphy. p. 7

Where the paper says it

IEEE Transactions on Computational Social Systems (2024). [37] Oonagh Murphy and Elena Villaespesa. p. 7

Where the paper says it

IEEE Transactions on Computational Social Systems (2024). [37] Oonagh Murphy and Elena Villaespesa.

2021. This is not an apple! p. 7
Where the paper says it

[47] Elena Villaespesa and Oonagh Murphy. This is not an apple!

Benefits and challenges of applying computer vision to museum collections. p. 7
Where the paper says it

This is not an apple! Benefits and challenges of applying computer vision to museum collections.

Museum Management and Curatorship (2021). p. 7
Where the paper says it

Benefits and challenges of applying computer vision to museum collections. Museum Management and Curatorship (2021).

[53] Haorui Yu, Xuehang Wen, Fengrui Zhang, and Qiufeng Yi. p. 7

Where the paper says it

press/v37/xuc15. html [53] Haorui Yu, Xuehang Wen, Fengrui Zhang, and Qiufeng Yi.

2026. A Multicultural Vision-Language Benchmark for Evaluating Cultural Understanding. p. 7
Where the paper says it

html [53] Haorui Yu, Xuehang Wen, Fengrui Zhang, and Qiufeng Yi. A Multicultural Vision-Language Benchmark for Evaluating Cultural Understanding.

arXiv preprint arXiv:2601. p. 7
Where the paper says it

A Multicultural Vision-Language Benchmark for Evaluating Cultural Understanding. arXiv preprint arXiv:2601.

07986 (2026). p. 7
Where the paper says it

arXiv preprint arXiv:2601. 07986 (2026).

[54] Cheng Zhang, Hongxia Xie, Bin Wen, Songhan Zuo, Ruoxuan Zhang, and WenHuang Cheng. p. 7

Where the paper says it

org/html/2601. 07986v3 [54] Cheng Zhang, Hongxia Xie, Bin Wen, Songhan Zuo, Ruoxuan Zhang, and WenHuang Cheng.

2025. EmoArt: A Multidimensional Dataset for Emotion-Aware Artistic Generation. p. 7
Where the paper says it

07986v3 [54] Cheng Zhang, Hongxia Xie, Bin Wen, Songhan Zuo, Ruoxuan Zhang, and WenHuang Cheng. EmoArt: A Multidimensional Dataset for Emotion-Aware Artistic Generation.

In Proceedings of the 33rd ACM International Conference on Multimedia (MM ’25). p. 7
Where the paper says it

EmoArt: A Multidimensional Dataset for Emotion-Aware Artistic Generation. In Proceedings of the 33rd ACM International Conference

ACM, Dublin, Ireland. p. 8
Where the paper says it

Appear2Meaning: A Cross-Cultural Benchmark for Structured Cultural Metadata Inference from ImagesACM MM ’26, November 10–14, 2026, Rio de Janeiro, Brazil on Multimedia (MM ’25). ACM, Dublin, Ireland, 12644–12650.

[18] Nicola Fanelli, Gennaro Vessio, and Giovanna Castellano. p. 7

Where the paper says it

European Journal of Cultural Management and Policy (2024). [18] Nicola Fanelli, Gennaro Vessio, and Giovanna Castellano.

2025. ARTSEEK: Deep Artwork Understanding via Multimodal In-Context Reasoning and Late Interaction Retrieval. p. 7
Where the paper says it

[18] Nicola Fanelli, Gennaro Vessio, and Giovanna Castellano. ARTSEEK: Deep Artwork Understanding via Multimodal In-Context Reasoning and Late Interaction Retrieval.

arXiv preprint arXiv:2507. p. 7
Where the paper says it

ARTSEEK: Deep Artwork Understanding via Multimodal In-Context Reasoning and Late Interaction Retrieval. arXiv preprint arXiv:2507.

21917 (July 2025). p. 7
Where the paper says it

arXiv preprint arXiv:2507. 21917 (July 2025).

[19] Marco Fiorucci et al. p. 7

Where the paper says it

21917 (July 2025). [19] Marco Fiorucci et al.

2020. Machine Learning for Cultural Heritage: A Survey. p. 7
Where the paper says it

[19] Marco Fiorucci et al. Machine Learning for Cultural Heritage: A Survey.

Pattern Recognition Letters (2020). p. 7
Where the paper says it

Machine Learning for Cultural Heritage: A Survey. Pattern Recognition Letters (2020).

https://doi. p. 7
Where the paper says it

Pattern Recognition Letters (2020). https://doi.

org/10.1016/j.patrec.2020.02.017.

[44] Bryan Chen Zhengyu Tan, Zheng Weihua, Zhengyuan Liu, Nancy F. p. 7

Where the paper says it

In Proceedings of the International Conference on Image Analysis and Processing (ICIAP 2019). [44] Bryan Chen Zhengyu Tan, Zheng Weihua, Zhengyuan Liu, Nancy F.

Chen, Hwaran Lee, Kenny Tsu Wei Choo, and Roy Ka-Wei Lee. p. 7
Where the paper says it

[44] Bryan Chen Zhengyu Tan, Zheng Weihua, Zhengyuan Liu, Nancy F. Chen, Hwaran Lee, Kenny Tsu Wei Choo, and Roy Ka-Wei Lee.

2026. BLEnD-Vis: Benchmarking Multimodal Cultural Understanding in Vision Language Models. p. 7
Where the paper says it

Chen, Hwaran Lee, Kenny Tsu Wei Choo, and Roy Ka-Wei Lee. BLEnD-Vis: Benchmarking Multimodal Cultural Understanding in Vision Language Models.

arXiv:2510. p. 7
Where the paper says it

BLEnD-Vis: Benchmarking Multimodal Cultural Understanding in Vision Language Models. arXiv:2510.

11178 [cs.CV] https://arxiv. p. 7
Where the paper says it

arXiv:2501. p. 7

Where the paper says it

CultureVLM: Characterizing and Improving Cultural Understanding of Vision-Language Models for over 100 Countries. arXiv:2501.

AI] https://arxiv. p. 7
Where the paper says it

arXiv:2501. AI] https://arxiv.

org/abs/2510.11178.

[36] Yue Lu, Chao Guo, Xingyuan Dai, and Fei-Yue Wang. p. 7

Where the paper says it

02265 (2019). [36] Yue Lu, Chao Guo, Xingyuan Dai, and Fei-Yue Wang.

2024. ArtCap: A Dataset for Image Captioning of Fine Art Paintings. p. 7
Where the paper says it

[36] Yue Lu, Chao Guo, Xingyuan Dai, and Fei-Yue Wang. ArtCap: A Dataset for Image Captioning of Fine Art Paintings.

IEEE Transactions on Computational Social Systems (2024). p. 7
Where the paper says it

ArtCap: A Dataset for Image Captioning of Fine Art Paintings. IEEE Transactions on Computational Social Systems (2024).

[51] Paula Westenberger and Despoina Farmaki. p. 7

Where the paper says it

press/v162/wang22al. html [51] Paula Westenberger and Despoina Farmaki.

2025. Artificial intelligence for cultural heritage research: the challenges in UK copyright law and policy. p. 7
Where the paper says it

html [51] Paula Westenberger and Despoina Farmaki. Artificial intelligence for cultural heritage research: the challenges in UK copyright law and policy.

European Journal of Cultural Management and Policy (2025). p. 7
Where the paper says it

Cultural work, wellbeing, and AI. European Journal of Cultural Management and Policy (2025).

[34] Shudong Liu, Yiqiao Jin, Cheng Li, Derek F. p. 7

Where the paper says it

, Red Hook, NY, USA, Article 1516, 25 pages. [34] Shudong Liu, Yiqiao Jin, Cheng Li, Derek F.

Wong, Qingsong Wen, Lichao Sun, Haipeng Chen, Xing Xie, and Jindong Wang. p. 7
Where the paper says it

[34] Shudong Liu, Yiqiao Jin, Cheng Li, Derek F. Wong, Qingsong Wen, Lichao Sun, Haipeng Chen, Xing Xie, and Jindong Wang.

2025. CultureVLM: Characterizing and Improving Cultural Understanding of Vision-Language Models for over 100 Countries. p. 7
Where the paper says it

Wong, Qingsong Wen, Lichao Sun, Haipeng Chen, Xing Xie, and Jindong Wang. CultureVLM: Characterizing and Improving Cultural Understanding of Vision-Language Models for over 100 Countries.

arXiv:2501.01282 [cs.AI] https://arxiv.org/abs/2501. p. 7
Where the paper says it

AI] https://arxiv. org/abs/2501.

01282.

[37] Oonagh Murphy and Elena Villaespesa. 2020. AI: A Museum Planning Toolkit.

[48] Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. p. 7

Where the paper says it

Museum Management and Curatorship (2021). [48] Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan.

2015. Show and Tell: A Neural Image Caption Generator. p. 7
Where the paper says it

[48] Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and Tell: A Neural Image Caption Generator.

In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). p. 6
Where the paper says it

Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[50] Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. p. 7

Where the paper says it

org/abs/2409. 12191 [50] Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang.

2022. OFA: Unifying Architectures Through a Simple Sequence-to-Sequence Learning Framework. p. 7
Where the paper says it

12191 [50] Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework.

In Proceedings of the 39th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 162), Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (Eds.). PMLR.

[21] Clifford Geertz. p. 1

Where the paper says it

Keywords cultural heritage, vision-language models, multimodal understanding, metadata inference, benchmark dataset, LLM-as-Judge 1 Introduction “Man is an animal suspended in webs of significance he himself has spun. ” — Clifford Geertz, The Interpretation of Cultures (1973) [21] What about vision-language models?

1973. The Interpretation of Cultures: Selected Essays. p. 7
Where the paper says it

[21] Clifford Geertz. The Interpretation of Cultures: Selected Essays.

Basic Books, New York. p. 7
Where the paper says it

The Interpretation of Cultures: Selected Essays. Basic Books, New York.

Summary of Error Case Studies

Taken together, these cases suggest that the observed error patterns are shaped not only by model capability but also by interactions among training priors, dataset composition, visual signal quality, and evaluation constraints. First, regional performance differences are unlikely to reflect a single factor. p. 1

Where the paper says it

Artpedia [43] provides visual and contextual descriptions for paintings but is limited to a single category and lacks cross-cultural variation. MLF [32] and CArt15K [55] focus on ceramics with factor-driven or metaphor-enriched annotations, yet remain restricted to specific object types and cultural contexts.

Second, the dataset itself likely contributes to these differences as regions differ substantially in object-type composition. p. 3
Where the paper says it

First, an annotator selects 50 artifacts per culture–type combination based on images and metadata. Second, a different annotator verifies cultural region and object type assignments.

Third, visual signal strength varies across regions and object types. p. 5
Where the paper says it

These findings suggest that current VLMs rely heavily on surfacelevel visual cues and memorized patterns, with limited ability to generalize cultural knowledge across regions. 3 Error Analysis We categorise model errors into four types, corresponding to the patterns observed across attributes and cultural contexts.

Finally, the case studies reinforce that several target fields are only partially observable from images. p. 6
Where the paper says it

Culture is not strictly bounded by geography, and this approximation may obscure intra-cultural diversity and reinforce reductive or essentialist interpretations. Inferring non-observable cultural attributes (e.

Final Synthesis

The central challenge is not only in detecting relevant visual features but also in aligning them with provenance-constrained metadata without over-reliance on prevalent stylistic associations. Outputs should be interpreted as hypothesis-like descriptions rather than authoritative attributions. p. 6

Where the paper says it

Inferring non-observable cultural attributes (e. , title, culture, period, origin, creator) from images introduces epistemic uncertainty and may produce incorrect or overconfident outputs, which should not be treated as authoritative judgments or used without expert validation in realworld applications, particularly in museum, educational, or heritage contexts.

This suggests that performance reflects a combination of visual recognition, prior-driven association, and contextual inference.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Improve structured metadata inference by moving beyond simple caption generation to a formal structured prediction task using an LLM-as-a-Judge framework that measures semantic alignment with reference annotations across four cultural regions.

  2. Enhance model performance by addressing cross-cultural misattribution and style transfer confusion through targeted training or fine-tuning that reduces reliance on learned stylistic priors when distinguishing between visual similarity and historical provenance.

  3. Develop systems capable of identifying object-type recognition without functional understanding by incorporating knowledge bases that link coarse visual forms to historically specific object labels, preventing the model from producing superficially plausible but semantically wrong titles.

  4. Increase robustness against period compression by training models to infer temporal context from culturally consistent metadata rather than anchoring predictions to familiar stylistic patterns, thereby preventing artifacts like shifting American ceramics into European production timelines.

  5. Improve creator attribution accuracy by incorporating mechanisms that integrate memorized maker associations with surrounding cultural context in a way that ensures coherent multi-attribute predictions, rather than allowing models to succeed on creator attribution without corresponding correct culture, period, and origin fields.

  6. Refine evaluation protocols to distinguish between semantic plausibility and exact metadata matching when assessing outputs like Case Study F, ensuring that coherent but non-canonical predictions are not incorrectly penalized under strict field-by-field comparison rules.

Abstract

Museum records express how people understand heritage objects through cultural and historical relations. Recovering these human attributions from images is a technical challenge for vision-language models (VLMs). We introduce Appear2Meaning, a cross-cultural benchmark of 750 museum objects for image-only recovery of title, culture, period, origin, and creator. Its evaluation framework distinguishes field-level from complete-record compatibility and uses human validation for semantic judgement. Across nine VLMs, mean field accuracy is 36.16--49.82%; 7.20--17.02% of records have all four contextual fields compatible, and 2.13--7.87% have all five, including title. For the two tested models, image-only input yields 12.6--32.2 percentage points higher reference compatibility for culture, period, and origin than a museum-and-title-only control.

Sources

Related papers