SemEval-2023 Task 11: Learning With Disagreements (LeWiDi)

Elisa Leonardelli, Gavin Abercrombie, Dina Almanea, Valerio Basile, Tommaso Fornaciari, Barbara Plank, Verena Rieser, Alexandra Uma Massimo Poesio

Research output: Chapter in Book/Report/Conference proceedingConference contribution

27 Citations (Scopus)
15 Downloads (Pure)


The paper contains examples which are offensive in nature. nlp datasets annotated with human judgments are rife with disagreements between the judges. This is especially true for tasks depending on subjective judgments such as sentiment analysis or offensive language detection. Particularly in these latter cases, the nlp community has come to realize that the approach of ‘reconciling’ these different subjective interpretations is inappropriate. Many nlp researchers have therefore concluded that rather than eliminating disagreements from annotated corpora, we should preserve them–indeed, some argue that corpora should aim to preserve all annotator judgments. But this approach to corpus creation for nlp has not yet been widely accepted. The objective of the LeWiDi series of shared tasks is to promote this approach to developing nlp models by providing a unified framework for training and evaluating with such datasets. We report on the second LeWiDi shared task, which differs from the first edition in three crucial respects: (i) it focuses entirely on nlp, instead of both nlp and computer vision tasks in its first edition; (ii) it focuses on subjective tasks, instead of covering different types of disagreements–as training with aggregated labels for subjective nlp tasks is a particularly obvious misrepresentation of the data; and (iii) for the evaluation, we concentrate on soft approaches to evaluation. This second edition of LeWiDi attracted a wide array of participants resulting in 13 shared task submission papers.

Original languageEnglish
Title of host publicationProceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023)
PublisherAssociation for Computational Linguistics
Number of pages15
ISBN (Electronic)9781959429999
Publication statusPublished - Jul 2023
Event17th International Workshop on Semantic Evaluation 2023 - Hybrid, Toronto, Canada
Duration: 13 Jul 202314 Jul 2023


Conference17th International Workshop on Semantic Evaluation 2023
Abbreviated titleSemEval 2023
CityHybrid, Toronto

ASJC Scopus subject areas

  • Computational Theory and Mathematics
  • Computer Science Applications
  • Theoretical Computer Science


Dive into the research topics of 'SemEval-2023 Task 11: Learning With Disagreements (LeWiDi)'. Together they form a unique fingerprint.

Cite this