SHADES: Towards a Multilingual Assessment of Stereotypes in Large Language Models

Margaret Mitchell; Hamdan Al-Ali; Giuseppe Attanasio; Ioana Baldini; Miruna Clinciu; Jordan Clive; Pieter Delobelle; Manan Dey; Kaustubh Dhole; Timm Dill; Amirbek Djanibekov; Tair Djanibekov; Jad Doughman; Ritam Dutt; Jessica Zosa Forde; Jay Gala; Avijit Ghosh; Sil Hamilton; Carolin Holtermann; Jerry Huang; Lucie Aimée Kaffee; Janavi Kasera; Tanmay Laud; Anne Lauscher; Roberto Luis López; Jonibek Mansurov; Maraim Masoud; Sagnik Mukherjee; Nurdaulet Mukhituly; Nikita Nangia; Shangrui Nie; Anaelia Ovalle; Giada Pistilli; Esther Ploeger; Jeremy Qin; Dragomir Radev; Vipul Raheja; Beatrice Savoldi; Shanya Sharma; Xudong Shen; Karolina Stańczak; Arjun Subramonian; Kaiser Sun; Eliza Szczechla; Tiago Timponi Torrent; Deepak Tunuguntla; Emilio Villa-Cueva; Marcelo Viridiano; Oskar van der Wal; Adina Yakefu; Kayo Yin; Mike Zhang; Sydney Zink; Aurélie Névéol; Zeerak Talat

doi:10.18653/v1/2025.naacl-long.600

SHADES: Towards a Multilingual Assessment of Stereotypes in Large Language Models

Margaret Mitchell
, Hamdan Al-Ali
, Giuseppe Attanasio
, Ioana Baldini
, Miruna Clinciu
, Jordan Clive
, Pieter Delobelle
, Manan Dey
, Kaustubh Dhole
, Timm Dill
, Amirbek Djanibekov
, Tair Djanibekov
, Jad Doughman
, Ritam Dutt
, Jessica Zosa Forde
, Jay Gala
, Avijit Ghosh
, Sil Hamilton
, Carolin Holtermann
, Jerry Huang

Lucie Aimée Kaffee, Janavi Kasera, Tanmay Laud, Anne Lauscher, Roberto Luis López, Jonibek Mansurov, Maraim Masoud, Sagnik Mukherjee, Nurdaulet Mukhituly, Nikita Nangia, Shangrui Nie, Anaelia Ovalle, Giada Pistilli, Esther Ploeger, Jeremy Qin, Dragomir Radev, Vipul Raheja, Beatrice Savoldi, Shanya Sharma, Xudong Shen, Karolina Stańczak, Arjun Subramonian, Kaiser Sun, Eliza Szczechla, Tiago Timponi Torrent, Deepak Tunuguntla, Emilio Villa-Cueva, Marcelo Viridiano, Oskar van der Wal, Adina Yakefu, Kayo Yin, Mike Zhang, Sydney Zink, Aurélie Névéol, Zeerak Talat

School of Engineering & Physical Sciences

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

5 Citations (Scopus)

Abstract

Large Language Models (LLMs), the bedrock of many “artificial intelligence” (AI) applications, are known to reproduce social biases present in their training data. Yet resources to measure and control this issue are limited. Research identifying and mitigating stereotype biases have primarily been concentrated around English, lagging the rapid advancement of LLMs in multilingual settings. To help further advance the ability to address stereotype bias in AI systems, we introduce a new multilingual dataset: SHADES. Designed for examining culturally-specific stereotypes that may be learned by LLMs, SHADES includes over 300 stereotypes from 37 regions, translated across 16 languages and annotated with multiple features to aid multilingual stereotype analysis. All statements in all languages are paired with templates, to serve as a resource for unlimited generation of new evaluation data. We demonstrate the utility of the dataset in a series of exploratory evaluations that reveal significant differences in how stereotypes are recognized and reflected across models and languages.

Original language	English
Title of host publication	Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics
Subtitle of host publication	Human Language Technologies
Editors	Luis Chiruzzo, Alan Ritter, Lu Wang
Publisher	Association for Computational Linguistics
Pages	11995-12041
Number of pages	47
Volume	1
ISBN (Electronic)	9798891761896
DOIs	https://doi.org/10.18653/v1/2025.naacl-long.600
Publication status	Published - Apr 2025
Event	2025 Annual Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies, - Hybrid, Albuquerque, United States Duration: 29 Apr 2025 → 4 May 2025

Conference

Conference	2025 Annual Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics
Abbreviated title	NAACL-HLT 2025
Country/Territory	United States
City	Hybrid, Albuquerque
Period	29/04/25 → 4/05/25

ASJC Scopus subject areas

Computer Networks and Communications
Hardware and Architecture
Information Systems
Software

Access to Document

10.18653/v1/2025.naacl-long.600Licence: CC BY

Download pdfFinal published version, 3.54 MBLicence: CC BY

Cite this

Mitchell, M., Al-Ali, H., Attanasio, G., Baldini, I., Clinciu, M., Clive, J., Delobelle, P., Dey, M., Dhole, K., Dill, T., Djanibekov, A., Djanibekov, T., Doughman, J., Dutt, R., Forde, J. Z., Gala, J., Ghosh, A., Hamilton, S., Holtermann, C., ... Talat, Z. (2025). SHADES: Towards a Multilingual Assessment of Stereotypes in Large Language Models. In L. Chiruzzo, A. Ritter, & L. Wang (Eds.), Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Vol. 1, pp. 11995-12041). Association for Computational Linguistics. https://doi.org/10.18653/v1/2025.naacl-long.600

Mitchell, Margaret ; Al-Ali, Hamdan ; Attanasio, Giuseppe et al. / SHADES : Towards a Multilingual Assessment of Stereotypes in Large Language Models. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies. editor / Luis Chiruzzo ; Alan Ritter ; Lu Wang. Vol. 1 Association for Computational Linguistics, 2025. pp. 11995-12041

@inproceedings{e001ee06ab5d451098fbbe210d5d937b,

title = "SHADES: Towards a Multilingual Assessment of Stereotypes in Large Language Models",

abstract = "Large Language Models (LLMs), the bedrock of many “artificial intelligence” (AI) applications, are known to reproduce social biases present in their training data. Yet resources to measure and control this issue are limited. Research identifying and mitigating stereotype biases have primarily been concentrated around English, lagging the rapid advancement of LLMs in multilingual settings. To help further advance the ability to address stereotype bias in AI systems, we introduce a new multilingual dataset: SHADES. Designed for examining culturally-specific stereotypes that may be learned by LLMs, SHADES includes over 300 stereotypes from 37 regions, translated across 16 languages and annotated with multiple features to aid multilingual stereotype analysis. All statements in all languages are paired with templates, to serve as a resource for unlimited generation of new evaluation data. We demonstrate the utility of the dataset in a series of exploratory evaluations that reveal significant differences in how stereotypes are recognized and reflected across models and languages.",

author = "Margaret Mitchell and Hamdan Al-Ali and Giuseppe Attanasio and Ioana Baldini and Miruna Clinciu and Jordan Clive and Pieter Delobelle and Manan Dey and Kaustubh Dhole and Timm Dill and Amirbek Djanibekov and Tair Djanibekov and Jad Doughman and Ritam Dutt and Forde, \{Jessica Zosa\} and Jay Gala and Avijit Ghosh and Sil Hamilton and Carolin Holtermann and Jerry Huang and Kaffee, \{Lucie Aim{\'e}e\} and Janavi Kasera and Tanmay Laud and Anne Lauscher and L{\'o}pez, \{Roberto Luis\} and Jonibek Mansurov and Maraim Masoud and Sagnik Mukherjee and Nurdaulet Mukhituly and Nikita Nangia and Shangrui Nie and Anaelia Ovalle and Giada Pistilli and Esther Ploeger and Jeremy Qin and Dragomir Radev and Vipul Raheja and Beatrice Savoldi and Shanya Sharma and Xudong Shen and Karolina Sta{\'n}czak and Arjun Subramonian and Kaiser Sun and Eliza Szczechla and Torrent, \{Tiago Timponi\} and Deepak Tunuguntla and Emilio Villa-Cueva and Marcelo Viridiano and \{van der Wal\}, Oskar and Adina Yakefu and Kayo Yin and Mike Zhang and Sydney Zink and Aur{\'e}lie N{\'e}v{\'e}ol and Zeerak Talat",

note = "Publisher Copyright: {\textcopyright} 2025 Association for Computational Linguistics.; 2025 Annual Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics : Human Language Technologies,, NAACL-HLT 2025 ; Conference date: 29-04-2025 Through 04-05-2025",

year = "2025",

month = apr,

doi = "10.18653/v1/2025.naacl-long.600",

language = "English",

volume = "1",

pages = "11995--12041",

editor = "Luis Chiruzzo and Alan Ritter and Lu Wang",

booktitle = "Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics",

publisher = "Association for Computational Linguistics",

}

Mitchell, M, Al-Ali, H, Attanasio, G, Baldini, I, Clinciu, M, Clive, J, Delobelle, P, Dey, M, Dhole, K, Dill, T, Djanibekov, A, Djanibekov, T, Doughman, J, Dutt, R, Forde, JZ, Gala, J, Ghosh, A, Hamilton, S, Holtermann, C, Huang, J, Kaffee, LA, Kasera, J, Laud, T, Lauscher, A, López, RL, Mansurov, J, Masoud, M, Mukherjee, S, Mukhituly, N, Nangia, N, Nie, S, Ovalle, A, Pistilli, G, Ploeger, E, Qin, J, Radev, D, Raheja, V, Savoldi, B, Sharma, S, Shen, X, Stańczak, K, Subramonian, A, Sun, K, Szczechla, E, Torrent, TT, Tunuguntla, D, Villa-Cueva, E, Viridiano, M, van der Wal, O, Yakefu, A, Yin, K, Zhang, M, Zink, S, Névéol, A & Talat, Z 2025, SHADES: Towards a Multilingual Assessment of Stereotypes in Large Language Models. in L Chiruzzo, A Ritter & L Wang (eds), Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies. vol. 1, Association for Computational Linguistics, pp. 11995-12041, 2025 Annual Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics, Hybrid, Albuquerque, New Mexico, United States, 29/04/25. https://doi.org/10.18653/v1/2025.naacl-long.600

SHADES: Towards a Multilingual Assessment of Stereotypes in Large Language Models. / Mitchell, Margaret; Al-Ali, Hamdan; Attanasio, Giuseppe et al.
Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies. ed. / Luis Chiruzzo; Alan Ritter; Lu Wang. Vol. 1 Association for Computational Linguistics, 2025. p. 11995-12041.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - SHADES

T2 - 2025 Annual Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics

AU - Mitchell, Margaret

AU - Al-Ali, Hamdan

AU - Attanasio, Giuseppe

AU - Baldini, Ioana

AU - Clinciu, Miruna

AU - Clive, Jordan

AU - Delobelle, Pieter

AU - Dey, Manan

AU - Dhole, Kaustubh

AU - Dill, Timm

AU - Djanibekov, Amirbek

AU - Djanibekov, Tair

AU - Doughman, Jad

AU - Dutt, Ritam

AU - Forde, Jessica Zosa

AU - Gala, Jay

AU - Ghosh, Avijit

AU - Hamilton, Sil

AU - Holtermann, Carolin

AU - Huang, Jerry

AU - Kaffee, Lucie Aimée

AU - Kasera, Janavi

AU - Laud, Tanmay

AU - Lauscher, Anne

AU - López, Roberto Luis

AU - Mansurov, Jonibek

AU - Masoud, Maraim

AU - Mukherjee, Sagnik

AU - Mukhituly, Nurdaulet

AU - Nangia, Nikita

AU - Nie, Shangrui

AU - Ovalle, Anaelia

AU - Pistilli, Giada

AU - Ploeger, Esther

AU - Qin, Jeremy

AU - Radev, Dragomir

AU - Raheja, Vipul

AU - Savoldi, Beatrice

AU - Sharma, Shanya

AU - Shen, Xudong

AU - Stańczak, Karolina

AU - Subramonian, Arjun

AU - Sun, Kaiser

AU - Szczechla, Eliza

AU - Torrent, Tiago Timponi

AU - Tunuguntla, Deepak

AU - Villa-Cueva, Emilio

AU - Viridiano, Marcelo

AU - van der Wal, Oskar

AU - Yakefu, Adina

AU - Yin, Kayo

AU - Zhang, Mike

AU - Zink, Sydney

AU - Névéol, Aurélie

AU - Talat, Zeerak

PY - 2025/4

Y1 - 2025/4

N2 - Large Language Models (LLMs), the bedrock of many “artificial intelligence” (AI) applications, are known to reproduce social biases present in their training data. Yet resources to measure and control this issue are limited. Research identifying and mitigating stereotype biases have primarily been concentrated around English, lagging the rapid advancement of LLMs in multilingual settings. To help further advance the ability to address stereotype bias in AI systems, we introduce a new multilingual dataset: SHADES. Designed for examining culturally-specific stereotypes that may be learned by LLMs, SHADES includes over 300 stereotypes from 37 regions, translated across 16 languages and annotated with multiple features to aid multilingual stereotype analysis. All statements in all languages are paired with templates, to serve as a resource for unlimited generation of new evaluation data. We demonstrate the utility of the dataset in a series of exploratory evaluations that reveal significant differences in how stereotypes are recognized and reflected across models and languages.

AB - Large Language Models (LLMs), the bedrock of many “artificial intelligence” (AI) applications, are known to reproduce social biases present in their training data. Yet resources to measure and control this issue are limited. Research identifying and mitigating stereotype biases have primarily been concentrated around English, lagging the rapid advancement of LLMs in multilingual settings. To help further advance the ability to address stereotype bias in AI systems, we introduce a new multilingual dataset: SHADES. Designed for examining culturally-specific stereotypes that may be learned by LLMs, SHADES includes over 300 stereotypes from 37 regions, translated across 16 languages and annotated with multiple features to aid multilingual stereotype analysis. All statements in all languages are paired with templates, to serve as a resource for unlimited generation of new evaluation data. We demonstrate the utility of the dataset in a series of exploratory evaluations that reveal significant differences in how stereotypes are recognized and reflected across models and languages.

UR - https://www.scopus.com/pages/publications/105027377978

U2 - 10.18653/v1/2025.naacl-long.600

DO - 10.18653/v1/2025.naacl-long.600

M3 - Conference contribution

AN - SCOPUS:105027377978

VL - 1

SP - 11995

EP - 12041

BT - Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics

A2 - Chiruzzo, Luis

A2 - Ritter, Alan

A2 - Wang, Lu

PB - Association for Computational Linguistics

Y2 - 29 April 2025 through 4 May 2025

ER -

Mitchell M, Al-Ali H, Attanasio G, Baldini I, Clinciu M, Clive J et al. SHADES: Towards a Multilingual Assessment of Stereotypes in Large Language Models. In Chiruzzo L, Ritter A, Wang L, editors, Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies. Vol. 1. Association for Computational Linguistics. 2025. p. 11995-12041 doi: 10.18653/v1/2025.naacl-long.600

SHADES: Towards a Multilingual Assessment of Stereotypes in Large Language Models

Abstract

Conference

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this