Combining Visual and Social Dialogue for Human-Robot Interaction

Nancie Gunson; Daniel Hernandez Garcia; Jose Part; Yanchao Yu; Weronika Sieinska; Christian Dondrup; Oliver Lemon

doi:10.1145/3462244.3481303

Combining Visual and Social Dialogue for Human-Robot Interaction

Nancie Gunson, Daniel Hernandez Garcia, Jose Part, Yanchao Yu, Weronika Sieinska, Christian Dondrup, Oliver Lemon

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

1 Citation (Scopus)

73 Downloads (Pure)

Abstract

We will demonstrate a prototype multimodal conversational AI system that will act as a receptionist in a hospital waiting room, combining visually-grounded dialogue with social conversation. The system supports visual object conversation in the waiting room (e.g. looking for available seats or personal belongings), task-based dialogues regarding navigation and check-in procedures in the hospital, as well as access to the latest news, and a quiz game about coronavirus.
The prototype system therefore demonstrates how to weave together a wide range of natural, daily conversations with end users that vary in complexity; from complex visual dialogue to chitchat and quiz games, to task-oriented domain-specific conversations.
We are currently able to demonstrate the system via a web-based interface. It will soon be deployed on the ARI robot in a hospital waiting room.

Original language	English
Title of host publication	ICMI '21: Proceedings of the 2021 International Conference on Multimodal Interaction
Subtitle of host publication	Montréal QC Canada October 18 - 22, 2021
Editors	Zakia Hammal, Carlos Busso, Catherine Pelachaud, Sharon Oviatt, Albert Ali Salah, Guoying Zhao
Place of Publication	New York
Publisher	Association for Computing Machinery
Pages	841-842
Number of pages	2
ISBN (Print)	9781450384810
DOIs	https://doi.org/10.1145/3462244.3481303
Publication status	Published - 18 Oct 2021
Event	23rd ACM International Conference on Multimodal Interaction 2021 - Montreal, Canada Duration: 18 Oct 2021 → 22 Oct 2021

Conference

Conference	23rd ACM International Conference on Multimodal Interaction 2021
Abbreviated title	ICMI 2021
Country/Territory	Canada
City	Montreal
Period	18/10/21 → 22/10/21

Keywords

Social Dialogue
Visual Dialogue

ASJC Scopus subject areas

Computer Science Applications
Computer Vision and Pattern Recognition
Hardware and Architecture
Human-Computer Interaction

Access to Document

10.1145/3462244.3481303

Download pdf
© ACM 2021. This is the author's version of the work. It is posted here for your personal use. Not for redistribution. The definitive Version of Record was published in ICMI '21: Proceedings of the 2021 International Conference on Multimodal Interaction, https://doi.org/10.1145/3462244.3481303
Accepted author manuscript, 1.56 MB

Cite this

Gunson, N., Hernandez Garcia, D., Part, J., Yu, Y., Sieinska, W., Dondrup, C., & Lemon, O. (2021). Combining Visual and Social Dialogue for Human-Robot Interaction. In Z. Hammal, C. Busso, C. Pelachaud, S. Oviatt, A. A. Salah, & G. Zhao (Eds.), ICMI '21: Proceedings of the 2021 International Conference on Multimodal Interaction: Montréal QC Canada October 18 - 22, 2021 (pp. 841-842). Association for Computing Machinery. https://doi.org/10.1145/3462244.3481303

Gunson, Nancie ; Hernandez Garcia, Daniel ; Part, Jose et al. / Combining Visual and Social Dialogue for Human-Robot Interaction. ICMI '21: Proceedings of the 2021 International Conference on Multimodal Interaction: Montréal QC Canada October 18 - 22, 2021. editor / Zakia Hammal ; Carlos Busso ; Catherine Pelachaud ; Sharon Oviatt ; Albert Ali Salah ; Guoying Zhao. New York : Association for Computing Machinery, 2021. pp. 841-842

@inproceedings{9cb1d06ec4f044abb6d9f659611843d7,

title = "Combining Visual and Social Dialogue for Human-Robot Interaction",

abstract = "We will demonstrate a prototype multimodal conversational AI system that will act as a receptionist in a hospital waiting room, combining visually-grounded dialogue with social conversation. The system supports visual object conversation in the waiting room (e.g. looking for available seats or personal belongings), task-based dialogues regarding navigation and check-in procedures in the hospital, as well as access to the latest news, and a quiz game about coronavirus.The prototype system therefore demonstrates how to weave together a wide range of natural, daily conversations with end users that vary in complexity; from complex visual dialogue to chitchat and quiz games, to task-oriented domain-specific conversations.We are currently able to demonstrate the system via a web-based interface. It will soon be deployed on the ARI robot in a hospital waiting room.",

keywords = "Social Dialogue, Visual Dialogue",

author = "Nancie Gunson and {Hernandez Garcia}, Daniel and Jose Part and Yanchao Yu and Weronika Sieinska and Christian Dondrup and Oliver Lemon",

note = "Funding Information: This research has been funded by the EU H2020 program under grant agreement no. 871245 (http://spring-h2020.eu/). Publisher Copyright: {\textcopyright} 2021 Owner/Author.; 23rd ACM International Conference on Multimodal Interaction 2021, ICMI 2021 ; Conference date: 18-10-2021 Through 22-10-2021",

year = "2021",

month = oct,

day = "18",

doi = "10.1145/3462244.3481303",

language = "English",

isbn = "9781450384810",

pages = "841--842",

editor = "Zakia Hammal and Carlos Busso and Pelachaud, {Catherine } and Oviatt, {Sharon } and Salah, {Albert Ali } and Zhao, {Guoying }",

booktitle = "ICMI '21: Proceedings of the 2021 International Conference on Multimodal Interaction",

publisher = "Association for Computing Machinery",

address = "United States",

}

Gunson, N , Hernandez Garcia, D, Part, J, Yu, Y, Sieinska, W, Dondrup, C & Lemon, O 2021, Combining Visual and Social Dialogue for Human-Robot Interaction. in Z Hammal, C Busso, C Pelachaud, S Oviatt, AA Salah & G Zhao (eds), ICMI '21: Proceedings of the 2021 International Conference on Multimodal Interaction: Montréal QC Canada October 18 - 22, 2021. Association for Computing Machinery, New York, pp. 841-842, 23rd ACM International Conference on Multimodal Interaction 2021, Montreal, Quebec, Canada, 18/10/21. https://doi.org/10.1145/3462244.3481303

Combining Visual and Social Dialogue for Human-Robot Interaction. / Gunson, Nancie ; Hernandez Garcia, Daniel; Part, Jose et al.
ICMI '21: Proceedings of the 2021 International Conference on Multimodal Interaction: Montréal QC Canada October 18 - 22, 2021. ed. / Zakia Hammal; Carlos Busso; Catherine Pelachaud; Sharon Oviatt; Albert Ali Salah; Guoying Zhao. New York: Association for Computing Machinery, 2021. p. 841-842.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Combining Visual and Social Dialogue for Human-Robot Interaction

AU - Gunson, Nancie

AU - Hernandez Garcia, Daniel

AU - Part, Jose

AU - Yu, Yanchao

AU - Sieinska, Weronika

AU - Dondrup, Christian

AU - Lemon, Oliver

PY - 2021/10/18

Y1 - 2021/10/18

N2 - We will demonstrate a prototype multimodal conversational AI system that will act as a receptionist in a hospital waiting room, combining visually-grounded dialogue with social conversation. The system supports visual object conversation in the waiting room (e.g. looking for available seats or personal belongings), task-based dialogues regarding navigation and check-in procedures in the hospital, as well as access to the latest news, and a quiz game about coronavirus.The prototype system therefore demonstrates how to weave together a wide range of natural, daily conversations with end users that vary in complexity; from complex visual dialogue to chitchat and quiz games, to task-oriented domain-specific conversations.We are currently able to demonstrate the system via a web-based interface. It will soon be deployed on the ARI robot in a hospital waiting room.

AB - We will demonstrate a prototype multimodal conversational AI system that will act as a receptionist in a hospital waiting room, combining visually-grounded dialogue with social conversation. The system supports visual object conversation in the waiting room (e.g. looking for available seats or personal belongings), task-based dialogues regarding navigation and check-in procedures in the hospital, as well as access to the latest news, and a quiz game about coronavirus.The prototype system therefore demonstrates how to weave together a wide range of natural, daily conversations with end users that vary in complexity; from complex visual dialogue to chitchat and quiz games, to task-oriented domain-specific conversations.We are currently able to demonstrate the system via a web-based interface. It will soon be deployed on the ARI robot in a hospital waiting room.

KW - Social Dialogue

KW - Visual Dialogue

UR - http://www.scopus.com/inward/record.url?scp=85118990012&partnerID=8YFLogxK

U2 - 10.1145/3462244.3481303

DO - 10.1145/3462244.3481303

M3 - Conference contribution

SN - 9781450384810

SP - 841

EP - 842

BT - ICMI '21: Proceedings of the 2021 International Conference on Multimodal Interaction

A2 - Hammal, Zakia

A2 - Busso, Carlos

A2 - Pelachaud, Catherine

A2 - Oviatt, Sharon

A2 - Salah, Albert Ali

A2 - Zhao, Guoying

PB - Association for Computing Machinery

CY - New York

T2 - 23rd ACM International Conference on Multimodal Interaction 2021

Y2 - 18 October 2021 through 22 October 2021

ER -

Gunson N , Hernandez Garcia D, Part J, Yu Y, Sieinska W, Dondrup C et al. Combining Visual and Social Dialogue for Human-Robot Interaction. In Hammal Z, Busso C, Pelachaud C, Oviatt S, Salah AA, Zhao G, editors, ICMI '21: Proceedings of the 2021 International Conference on Multimodal Interaction: Montréal QC Canada October 18 - 22, 2021. New York: Association for Computing Machinery. 2021. p. 841-842 doi: 10.1145/3462244.3481303

Combining Visual and Social Dialogue for Human-Robot Interaction

Abstract

Conference

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Dataset supporting the paper "Am I Allergic to This? Assisting Sight Impaired People in the Kitchen"

Cite this

Combining Visual and Social Dialogue for Human-Robot Interaction

Abstract

Conference

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Datasets

Dataset supporting the paper "Am I Allergic to This? Assisting Sight Impaired People in the Kitchen"

Cite this