ViCA: Combining visual, social, and task-oriented conversational AI in a healthcare setting

Georgios Pantazopoulos; Jeremy Bruyere; Maria-Vasiliki Nikandrou; Thibaud Boissier; Supun Hemanthage; Sachish Binha; Vidyul Shah; Christian Dondrup; Oliver Lemon

doi:10.1145/3462244.3479909

ViCA: Combining visual, social, and task-oriented conversational AI in a healthcare setting

Georgios Pantazopoulos, Jeremy Bruyere, Maria-Vasiliki Nikandrou, Thibaud Boissier, Supun Hemanthage, Sachish Binha, Vidyul Shah, Christian Dondrup, Oliver Lemon

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

2 Citations (Scopus)

210 Downloads (Pure)

Abstract

Recent developments in computer vision and conversational systems have provided the AI community with novel perspectives towards improving the cognitive capabilities of engaging socially assistive robots. We show how to develop conversational skills for a hospital receptionist robot that incorporates social conversation based on visual information as well as task-based dialog. Fusing the traditional modular conversational system architecture with recent developments in computer vision and scene graph research, our agent (called ĝViCA') supports both visual question answering and social conversational capabilities based on the visual scene. In particular, our agent can provide guidance to users by locating visible objects in the room and can engage in social dialog using visual prompts, such as the user's clothing or possessions. We conduct a comprehensive online evaluation study with 21 participants, showcasing that the ViCA system is perceived as both helpful and entertaining.

Original language	English
Title of host publication	ICMI '21: Proceedings of the 2021 International Conference on Multimodal Interaction
Publisher	Association for Computing Machinery
Pages	71-79
Number of pages	9
ISBN (Electronic)	9781450384810
DOIs	https://doi.org/10.1145/3462244.3479909
Publication status	Published - 18 Oct 2021
Event	23rd ACM International Conference on Multimodal Interaction 2021 - Montreal, Canada Duration: 18 Oct 2021 → 22 Oct 2021

Conference

Conference	23rd ACM International Conference on Multimodal Interaction 2021
Abbreviated title	ICMI 2021
Country/Territory	Canada
City	Montreal
Period	18/10/21 → 22/10/21

Keywords

conversational agents
human robot interaction
multimodal interaction

ASJC Scopus subject areas

Computer Science Applications
Computer Vision and Pattern Recognition
Hardware and Architecture
Human-Computer Interaction

Access to Document

10.1145/3462244.3479909

Download pdf
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.
Accepted author manuscript, 4.66 MB

Cite this

Pantazopoulos, G., Bruyere, J., Nikandrou, M.-V., Boissier, T., Hemanthage, S., Binha, S., Shah, V., Dondrup, C., & Lemon, O. (2021). ViCA: Combining visual, social, and task-oriented conversational AI in a healthcare setting. In ICMI '21: Proceedings of the 2021 International Conference on Multimodal Interaction (pp. 71-79). Association for Computing Machinery. https://doi.org/10.1145/3462244.3479909

@inproceedings{f01983f5a5a84dddb676a747159d7fa1,

title = "ViCA: Combining visual, social, and task-oriented conversational AI in a healthcare setting",

abstract = "Recent developments in computer vision and conversational systems have provided the AI community with novel perspectives towards improving the cognitive capabilities of engaging socially assistive robots. We show how to develop conversational skills for a hospital receptionist robot that incorporates social conversation based on visual information as well as task-based dialog. Fusing the traditional modular conversational system architecture with recent developments in computer vision and scene graph research, our agent (called ĝViCA') supports both visual question answering and social conversational capabilities based on the visual scene. In particular, our agent can provide guidance to users by locating visible objects in the room and can engage in social dialog using visual prompts, such as the user's clothing or possessions. We conduct a comprehensive online evaluation study with 21 participants, showcasing that the ViCA system is perceived as both helpful and entertaining.",

keywords = "conversational agents, human robot interaction, multimodal interaction",

author = "Georgios Pantazopoulos and Jeremy Bruyere and Maria-Vasiliki Nikandrou and Thibaud Boissier and Supun Hemanthage and Sachish Binha and Vidyul Shah and Christian Dondrup and Oliver Lemon",

note = "Publisher Copyright: {\textcopyright} 2021 ACM.; 23rd ACM International Conference on Multimodal Interaction 2021, ICMI 2021 ; Conference date: 18-10-2021 Through 22-10-2021",

year = "2021",

month = oct,

day = "18",

doi = "10.1145/3462244.3479909",

language = "English",

pages = "71--79",

booktitle = "ICMI '21: Proceedings of the 2021 International Conference on Multimodal Interaction",

publisher = "Association for Computing Machinery",

address = "United States",

}

Pantazopoulos, G, Bruyere, J, Nikandrou, M-V, Boissier, T, Hemanthage, S, Binha, S, Shah, V, Dondrup, C & Lemon, O 2021, ViCA: Combining visual, social, and task-oriented conversational AI in a healthcare setting. in ICMI '21: Proceedings of the 2021 International Conference on Multimodal Interaction. Association for Computing Machinery, pp. 71-79, 23rd ACM International Conference on Multimodal Interaction 2021, Montreal, Quebec, Canada, 18/10/21. https://doi.org/10.1145/3462244.3479909

ViCA: Combining visual, social, and task-oriented conversational AI in a healthcare setting. / Pantazopoulos, Georgios; Bruyere, Jeremy; Nikandrou, Maria-Vasiliki et al.
ICMI '21: Proceedings of the 2021 International Conference on Multimodal Interaction. Association for Computing Machinery, 2021. p. 71-79.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - ViCA: Combining visual, social, and task-oriented conversational AI in a healthcare setting

AU - Pantazopoulos, Georgios

AU - Bruyere, Jeremy

AU - Nikandrou, Maria-Vasiliki

AU - Boissier, Thibaud

AU - Hemanthage, Supun

AU - Binha, Sachish

AU - Shah, Vidyul

AU - Dondrup, Christian

AU - Lemon, Oliver

PY - 2021/10/18

Y1 - 2021/10/18

N2 - Recent developments in computer vision and conversational systems have provided the AI community with novel perspectives towards improving the cognitive capabilities of engaging socially assistive robots. We show how to develop conversational skills for a hospital receptionist robot that incorporates social conversation based on visual information as well as task-based dialog. Fusing the traditional modular conversational system architecture with recent developments in computer vision and scene graph research, our agent (called ĝViCA') supports both visual question answering and social conversational capabilities based on the visual scene. In particular, our agent can provide guidance to users by locating visible objects in the room and can engage in social dialog using visual prompts, such as the user's clothing or possessions. We conduct a comprehensive online evaluation study with 21 participants, showcasing that the ViCA system is perceived as both helpful and entertaining.

AB - Recent developments in computer vision and conversational systems have provided the AI community with novel perspectives towards improving the cognitive capabilities of engaging socially assistive robots. We show how to develop conversational skills for a hospital receptionist robot that incorporates social conversation based on visual information as well as task-based dialog. Fusing the traditional modular conversational system architecture with recent developments in computer vision and scene graph research, our agent (called ĝViCA') supports both visual question answering and social conversational capabilities based on the visual scene. In particular, our agent can provide guidance to users by locating visible objects in the room and can engage in social dialog using visual prompts, such as the user's clothing or possessions. We conduct a comprehensive online evaluation study with 21 participants, showcasing that the ViCA system is perceived as both helpful and entertaining.

KW - conversational agents

KW - human robot interaction

KW - multimodal interaction

UR - http://www.scopus.com/inward/record.url?scp=85118982040&partnerID=8YFLogxK

U2 - 10.1145/3462244.3479909

DO - 10.1145/3462244.3479909

M3 - Conference contribution

AN - SCOPUS:85118982040

SP - 71

EP - 79

BT - ICMI '21: Proceedings of the 2021 International Conference on Multimodal Interaction

PB - Association for Computing Machinery

T2 - 23rd ACM International Conference on Multimodal Interaction 2021

Y2 - 18 October 2021 through 22 October 2021

ER -

ViCA: Combining visual, social, and task-oriented conversational AI in a healthcare setting

Abstract

Conference

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this