Skip to main navigation Skip to search Skip to main content

CROPE: Evaluating In-Context Adaptation of Vision and Language Models to Culture-Specific Concepts

  • Malvina Nikandrou
  • , Georgios Pantazopoulos
  • , Nikolas Vitsakis
  • , Ioannis Konstas
  • , Alessandro Suglia

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

2 Downloads (Pure)

Abstract

As Vision and Language models (VLMs) are reaching users across the globe, assessing their cultural understanding has become a critical challenge. In this paper, we introduce CROPE, a visual question answering benchmark designed to probe the knowledge of culture-specific concepts and evaluate the capacity for cultural adaptation through contextual information. This allows us to distinguish between parametric knowledge acquired during training and contextual knowledge provided during inference via visual and textual descriptions. Our evaluation of several state-of-the-art open VLMs shows large performance disparities between culture-specific and common concepts in the parametric setting. Moreover, experiments with contextual knowledge indicate that models struggle to effectively utilize multimodal information and bind culture-specific concepts to their depictions. Our findings reveal limitations in the cultural understanding and adaptability of current VLMs that need to be addressed toward more culturally inclusive models.

Original languageEnglish
Title of host publicationProceedings of the 2025 Annual Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics
Subtitle of host publicationHuman Language Technologies
EditorsLuis Chiruzzo, Alan Ritter, Lu Wang
PublisherAssociation for Computational Linguistics
Pages7917-7936
Number of pages20
Volume1
ISBN (Electronic)9798891761896
DOIs
Publication statusPublished - Apr 2025
Event2025 Annual Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies, - Hybrid, Albuquerque, United States
Duration: 29 Apr 2025 → 4 May 2025

Conference

Conference2025 Annual Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics
Abbreviated titleNAACL-HLT 2025
Country/TerritoryUnited States
CityHybrid, Albuquerque
Period29/04/25 → 4/05/25

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Hardware and Architecture
  • Information Systems
  • Software

Fingerprint

Dive into the research topics of 'CROPE: Evaluating In-Context Adaptation of Vision and Language Models to Culture-Specific Concepts'. Together they form a unique fingerprint.

Cite this