A Holistic Evaluation Methodology for Multi-Party Spoken Conversational Agents

While research in multi-party spoken conversation with intelligent embodied agents has made significant progress in sub-tasks like speaker identification and non-verbal cues, there’s a gap in fully autonomous applications users can directly interact with. This lack translates to the absence of a standard methodology for evaluating multi-party conversational speech agents that considers both task-based system performance and user experience. Our research has addressed the former by developing a multimodal robot receptionist for a hospital waiting room whose multiparty conversational ability, nonverbal behaviour, and dialogue management is implemented using Large Language Models (LLM). In this paper, we go on to address the issue of evaluation, describing an experimental methodology and design of task-based user experiments that captures both objective measures of multi-party dialogue performance (such as accurate tracking of user goals) and the users’ subjective experience of multi-party embodied conversations. This paper therefore presents a holistic methodology for the future evaluation of multi-party spoken conversational agents.