Benchmarking Natural Language Understanding Services for Building Conversational Agents

Xingkun Liu, Arash Eshghi, Pawel Swietojanski, Verena Rieser

Research output: Chapter in Book/Report/Conference proceedingConference contribution

46 Citations (Scopus)
151 Downloads (Pure)


We have recently seen the emergence of several publicly available Natural Language Understanding (NLU) toolkits, which map user utterances to structured, but more abstract, Dialogue Act (DA) or Intent specifications, while making this process accessible to the lay developer. In this paper, we present the first wide coverage evaluation and comparison of some of the most popular NLU services, on a large, multi-domain (21 domains) dataset of 25 K user utterances that we have collected and annotated with Intent and Entity Type specifications and which will be released as part of this submission ( ). The results show that on Intent classification Watson significantly outperforms the other platforms, namely, Dialogflow, LUIS and Rasa; though these also perform well. Interestingly, on Entity Type recognition, Watson performs significantly worse due to its low Precision (At the time of producing the camera-ready version of this paper, we noticed the seemingly recent addition of a ‘Contextual Entity’ annotation tool to Watson, much like e.g. in Rasa. We’d threfore like to stress that this paper does not include an evaluation of this feature in Watson NLU.). Again, Dialogflow, LUIS and Rasa perform well on this task.
Original languageEnglish
Title of host publicationIncreasing Naturalness and Flexibility in Spoken Dialogue Interaction
Subtitle of host publication10th International Workshop on Spoken Dialogue Systems
Number of pages19
ISBN (Electronic)9789811593239
ISBN (Print)9789811593222, 9789811593253
Publication statusPublished - 2021
Event10th International Workshop on Spoken Dialogue Systems Technology 2019 - Sicily, Siracusa, Italy
Duration: 24 Apr 201926 Apr 2019

Publication series

NameLecture Notes in Electrical Engineering
ISSN (Print)1876-1100
ISSN (Electronic)1876-1119


Conference10th International Workshop on Spoken Dialogue Systems Technology 2019
Abbreviated titleIWSDS 2019
Internet address

ASJC Scopus subject areas

  • Industrial and Manufacturing Engineering


Dive into the research topics of 'Benchmarking Natural Language Understanding Services for Building Conversational Agents'. Together they form a unique fingerprint.

Cite this