Temporal and Second Language Influence on Intra-Annotator Agreement and Stability in Hate Speech Labelling

Gavin Abercrombie, Dirk Hovy, Vinodkumar Prabhakaran

Research output: Chapter in Book/Report/Conference proceedingConference contribution

3 Citations (Scopus)
117 Downloads (Pure)


Much work in natural language processing (NLP) relies on human annotation. The majority of this implicitly assumes that annotator’s labels are temporally stable, although the reality is that human judgements are rarely consistent over time. As a subjective annotation task, hate speech labels depend on annotator’s emotional and moral reactions to the language used to convey the message. Studies in Cognitive Science reveal a ‘foreign language effect’, whereby people take differing moral positions and perceive offensive phrases to be weaker in their second languages. Does this affect annotations as well? We conduct an experiment to investigate the impacts of (1) time and (2) different language conditions (English and German) on measurements of intra-annotator agreement in a hate speech labelling task. While we do not observe the expected lower stability in the different language condition, we find that overall agreement is significantly lower than is implicitly assumed in annotation tasks, which has important implications for dataset reproducibility in NLP.

Original languageEnglish
Title of host publicationProceedings of the 17th Linguistic Annotation Workshop (LAW-XVII)
EditorsJakob Prange, Annemarie Friedrich
PublisherAssociation for Computational Linguistics
Number of pages8
ISBN (Electronic)9781959429838
Publication statusPublished - 13 Jul 2023
Event17th Linguistic Annotation Workshop 2023 - Toronto, Canada
Duration: 13 Jul 2023 → …

Publication series

NameProceedings of the Annual Meeting of the Association for Computational Linguistics
ISSN (Print)0736-587X


Conference17th Linguistic Annotation Workshop 2023
Abbreviated titleLAW 2023
Period13/07/23 → …

ASJC Scopus subject areas

  • Computer Science Applications
  • Linguistics and Language
  • Language and Linguistics


Dive into the research topics of 'Temporal and Second Language Influence on Intra-Annotator Agreement and Stability in Hate Speech Labelling'. Together they form a unique fingerprint.

Cite this