Multi-party Multimodal Conversations Between Patients, Their Companions, and a Social Robot in a Hospital Memory Clinic

Research output: Chapter in Book/Report/Conference proceedingConference contribution

40 Downloads (Pure)

Abstract

We have deployed an LLM-based spoken dialogue system in a real hospital. The ARI social robot embodies our system, which patients and their companions can have multi-party conversations with together. In order to enable this multi-party ability, multimodality is critical. Our system, therefore, receives speech and video as input, and generates both speech and gestures (arm, head, and eye movements). In this paper, we describe our complex setting and the architecture of our dialogue system. Each component is detailed, and a video of the full system is available with the appropriate components highlighted in real-time. Our system decides when it should take its turn, generates human-like clarification requests when the patient pauses mid-utterance, answers in-domain questions (grounding to the in-prompt knowledge), and responds appropriately to out-of-domain requests (like generating jokes or quizzes). This latter feature is particularly remarkable as real patients often utter unexpected sentences that could not be handled previously.

Original languageEnglish
Title of host publicationProceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations
EditorsNikolaos Aletras, Orphee De Clercq
PublisherAssociation for Computational Linguistics
Pages62-70
Number of pages9
ISBN (Electronic)9798891760912
Publication statusPublished - Mar 2024
Event18th Conference of the European Chapter of the Association for Computational Linguistics 2024 - St. Julian's, Malta
Duration: 17 Mar 202422 Mar 2024

Conference

Conference18th Conference of the European Chapter of the Association for Computational Linguistics 2024
Abbreviated titleEACL 2024
Country/TerritoryMalta
CitySt. Julian's
Period17/03/2422/03/24

ASJC Scopus subject areas

  • Computational Theory and Mathematics
  • Software
  • Linguistics and Language

Fingerprint

Dive into the research topics of 'Multi-party Multimodal Conversations Between Patients, Their Companions, and a Social Robot in a Hospital Memory Clinic'. Together they form a unique fingerprint.

Cite this