A Holistic Evaluation Methodology for Multi-Party Spoken Conversational Agents

Research output: Chapter in Book/Report/Conference proceedingChapter (peer-reviewed)peer-review

19 Downloads (Pure)

Abstract

While research in multi-party spoken conversation with intelligent embodied agents has made significant progress in sub-tasks like speaker identification and non-verbal cues, there’s a gap in fully autonomous applications users can directly interact with. This lack translates to the absence of a standard methodology for evaluating multi-party conversational speech agents that considers both task-based system performance and user experience. Our research has addressed the former by developing a multimodal robot receptionist for a hospital waiting room whose multiparty conversational ability, nonverbal behaviour, and dialogue management is implemented using Large Language Models (LLM). In this paper, we go on to address the issue of evaluation, describing an experimental methodology and design of task-based user experiments that captures both objective measures of multi-party dialogue performance (such as accurate tracking of user goals) and the users’ subjective experience of multi-party embodied conversations. This paper therefore presents a holistic methodology for the future evaluation of multi-party spoken conversational agents.
Original languageEnglish
Title of host publicationIVA '24: Proceedings of the 24th ACM International Conference on Intelligent Virtual Agents
PublisherAssociation for Computing Machinery
ISBN (Print)9798400706257
DOIs
Publication statusPublished - 26 Dec 2024
Event24th ACM International Conference on Intelligent Virtual Agents 2024 - Glasgow, United Kingdom
Duration: 16 Sept 202419 Sept 2024

Conference

Conference24th ACM International Conference on Intelligent Virtual Agents 2024
Abbreviated titleIVA '24
Country/TerritoryUnited Kingdom
CityGlasgow
Period16/09/2419/09/24

Keywords

  • evaluation methodology
  • user study
  • multi-party interaction
  • conversational agents
  • large language models
  • social robots
  • HRI

Fingerprint

Dive into the research topics of 'A Holistic Evaluation Methodology for Multi-Party Spoken Conversational Agents'. Together they form a unique fingerprint.

Cite this