iLab at SemEval-2023 Task 11 Le-Wi-Di: Modelling Disagreement or Modelling Perspectives?

Research output: Chapter in Book/Report/Conference proceedingConference contribution

3 Citations (Scopus)
25 Downloads (Pure)

Abstract

There are two competing approaches for modelling annotator disagreement: distributional soft-labelling approaches (which aim to capture the level of disagreement) or modelling perspectives of individual annotators or groups thereof. We adapt a multi-task architecture — which has previously shown success in modelling perspectives— to evaluate its performance on the SEMEVAL Task 11. We do so by combining both approaches, i.e. predicting individual annotator perspectives as an interim step towards predicting annotator disagreement. Despite its previous success, we found that a multi-task approach performed poorly on datasets which contained distinct annotator opinions, suggesting that this approach may not always be suitable when modelling perspectives. Furthermore, our results explain that while strongly perspectivist approaches might not achieve state-of-the-art performance according to evaluation metrics used by distributional approaches, our approach allows for a more nuanced understanding of individual perspectives present in the data. We argue that perspectivist approaches are preferable because they enable decision makers to amplify minority views, and that it is important to re-evaluate metrics to reflect this goal.

Original languageEnglish
Title of host publicationProceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023)
PublisherAssociation for Computational Linguistics
Pages1660-1669
Number of pages10
ISBN (Electronic)9781959429999
DOIs
Publication statusPublished - Jul 2023
Event17th International Workshop on Semantic Evaluation 2023 - Hybrid, Toronto, Canada
Duration: 13 Jul 202314 Jul 2023

Conference

Conference17th International Workshop on Semantic Evaluation 2023
Abbreviated titleSemEval 2023
Country/TerritoryCanada
CityHybrid, Toronto
Period13/07/2314/07/23

ASJC Scopus subject areas

  • Computational Theory and Mathematics
  • Computer Science Applications
  • Theoretical Computer Science

Fingerprint

Dive into the research topics of 'iLab at SemEval-2023 Task 11 Le-Wi-Di: Modelling Disagreement or Modelling Perspectives?'. Together they form a unique fingerprint.

Cite this