Abstract
As Vision and Language models (VLMs) are reaching users across the globe, assessing their cultural understanding has become a critical challenge. In this paper, we introduce CROPE, a visual question answering benchmark designed to probe the knowledge of culture-specific concepts and evaluate the capacity for cultural adaptation through contextual information. This allows us to distinguish between parametric knowledge acquired during training and contextual knowledge provided during inference via visual and textual descriptions. Our evaluation of several state-of-the-art open VLMs shows large performance disparities between culture-specific and common concepts in the parametric setting. Moreover, experiments with contextual knowledge indicate that models struggle to effectively utilize multimodal information and bind culture-specific concepts to their depictions. Our findings reveal limitations in the cultural understanding and adaptability of current VLMs that need to be addressed toward more culturally inclusive models.
| Original language | English |
|---|---|
| Title of host publication | Proceedings of the 2025 Annual Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics |
| Subtitle of host publication | Human Language Technologies |
| Editors | Luis Chiruzzo, Alan Ritter, Lu Wang |
| Publisher | Association for Computational Linguistics |
| Pages | 7917-7936 |
| Number of pages | 20 |
| Volume | 1 |
| ISBN (Electronic) | 9798891761896 |
| DOIs | |
| Publication status | Published - Apr 2025 |
| Event | 2025 Annual Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies, - Hybrid, Albuquerque, United States Duration: 29 Apr 2025 → 4 May 2025 |
Conference
| Conference | 2025 Annual Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics |
|---|---|
| Abbreviated title | NAACL-HLT 2025 |
| Country/Territory | United States |
| City | Hybrid, Albuquerque |
| Period | 29/04/25 → 4/05/25 |
ASJC Scopus subject areas
- Computer Networks and Communications
- Hardware and Architecture
- Information Systems
- Software
Fingerprint
Dive into the research topics of 'CROPE: Evaluating In-Context Adaptation of Vision and Language Models to Culture-Specific Concepts'. Together they form a unique fingerprint.Cite this
- APA
- Author
- BIBTEX
- Harvard
- Standard
- RIS
- Vancouver