ViCA: Combining visual, Social, and Task-orientedconversational AI in a Healthcare Setting

George Pantazopoulos, Jeremy Bruyere, Malvina Nikandrou, Thibaud Boissier, Supun Hemanthage, Binha Kumar Sachish, Vidyul Shah, Christian Dondrup, Oliver Lemon

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Recent developments in computer vision and conversational systems have provided the AI community with novel perspectives towards improving the cognitive capabilities of engaging socially assistive robots. We show how to develop conversational skills for a hospital receptionist robot that incorporates social conversation based on visual information as well as task-based dialog. Fusing the traditional modular conversational system architecture with recent developments in computer vision and scene graph research, our agent (called ĝViCA') supports both visual question answering and social conversational capabilities based on the visual scene. In particular, our agent can provide guidance to users by locating visible objects in the room and can engage in social dialog using visual prompts, such as the user's clothing or possessions. We conduct a comprehensive online evaluation study with 21 participants, showcasing that the ViCA system is perceived as both helpful and entertaining.

Original languageEnglish
Title of host publicationICMI '21: Proceedings of the 2021 International Conference on Multimodal Interaction
PublisherAssociation for Computing Machinery
Pages71-79
Number of pages9
ISBN (Electronic)9781450384810
DOIs
Publication statusPublished - 18 Oct 2021
Event23rd ACM International Conference on Multimodal Interaction 2021 - Montreal, Canada
Duration: 18 Oct 202122 Oct 2021

Conference

Conference23rd ACM International Conference on Multimodal Interaction 2021
Abbreviated titleICMI 2021
Country/TerritoryCanada
CityMontreal
Period18/10/2122/10/21

Keywords

  • conversational agents
  • human robot interaction
  • multimodal interaction

ASJC Scopus subject areas

  • Computer Science Applications
  • Computer Vision and Pattern Recognition
  • Hardware and Architecture
  • Human-Computer Interaction

Fingerprint

Dive into the research topics of 'ViCA: Combining visual, Social, and Task-orientedconversational AI in a Healthcare Setting'. Together they form a unique fingerprint.

Cite this