A Multi-party Conversational Social Robot Using LLMs

Angus Addlesee; Neeraj Cherakara; Nivan Nelson; Daniel Hernández García; Nancie Gunson; Weronika Sieińska; Marta Romeo; Christian Dondrup; Oliver Lemon

doi:10.1145/3610978.3641112

A Multi-party Conversational Social Robot Using LLMs

Angus Addlesee, Neeraj Cherakara, Nivan Nelson, Daniel Hernández García, Nancie Gunson, Weronika Sieińska, Marta Romeo, Christian Dondrup, Oliver Lemon

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

9 Citations (Scopus)

Abstract

In this paper, we describe our setting and the architecture of our LLM-based dialogue system embodied in a social robot and able to have multi-party conversations. Each component is detailed, and a video of the full system is available with the appropriate components highlighted in real-time. Our system decides when it should take its turn, generates human-like clarification requests when the patient pauses mid-utterance, answers in-domain questions (grounding to the in-prompt knowledge), and responds appropriately to out-of-domain requests (like generating jokes or quizzes).

Original language	English
Title of host publication	HRI '24: Companion of the 2024 ACM/IEEE International Conference on Human-Robot Interaction
Publisher	Association for Computing Machinery
Pages	1273-1275
Number of pages	3
ISBN (Electronic)	9798400703232
DOIs	https://doi.org/10.1145/3610978.3641112
Publication status	Published - 11 Mar 2024
Event	19th Annual ACM/IEEE International Conference on Human-Robot Interaction 2024 - Boulder, United States Duration: 11 Mar 2024 → 15 Mar 2024

Conference

Conference	19th Annual ACM/IEEE International Conference on Human-Robot Interaction 2024
Abbreviated title	HRI 2024
Country/Territory	United States
City	Boulder
Period	11/03/24 → 15/03/24

Keywords

accessibility
conversational AI
human-robot interaction
large language models
multi-party dialogue
social robots

ASJC Scopus subject areas

Artificial Intelligence
Human-Computer Interaction
Electrical and Electronic Engineering

Access to Document

10.1145/3610978.3641112

Cite this

Addlesee, A., Cherakara, N., Nelson, N., García, D. H., Gunson, N., Sieińska, W., Romeo, M., Dondrup, C., & Lemon, O. (2024). A Multi-party Conversational Social Robot Using LLMs. In HRI '24: Companion of the 2024 ACM/IEEE International Conference on Human-Robot Interaction (pp. 1273-1275). Association for Computing Machinery. https://doi.org/10.1145/3610978.3641112

@inproceedings{fd25270a0b814ab88b9fe4fceea65ed7,

title = "A Multi-party Conversational Social Robot Using LLMs",

abstract = "In this paper, we describe our setting and the architecture of our LLM-based dialogue system embodied in a social robot and able to have multi-party conversations. Each component is detailed, and a video of the full system is available with the appropriate components highlighted in real-time. Our system decides when it should take its turn, generates human-like clarification requests when the patient pauses mid-utterance, answers in-domain questions (grounding to the in-prompt knowledge), and responds appropriately to out-of-domain requests (like generating jokes or quizzes).",

keywords = "accessibility, conversational AI, human-robot interaction, large language models, multi-party dialogue, social robots",

author = "Angus Addlesee and Neeraj Cherakara and Nivan Nelson and Garc{\'i}a, {Daniel Hern{\'a}ndez} and Nancie Gunson and Weronika Siei{\'n}ska and Marta Romeo and Christian Dondrup and Oliver Lemon",

note = "Publisher Copyright: {\textcopyright} 2024 Copyright held by the owner/author(s); 19th Annual ACM/IEEE International Conference on Human-Robot Interaction 2024, HRI 2024 ; Conference date: 11-03-2024 Through 15-03-2024",

year = "2024",

month = mar,

day = "11",

doi = "10.1145/3610978.3641112",

language = "English",

pages = "1273--1275",

booktitle = "HRI '24: Companion of the 2024 ACM/IEEE International Conference on Human-Robot Interaction",

publisher = "Association for Computing Machinery",

address = "United States",

}

Addlesee, A, Cherakara, N, Nelson, N, García, DH , Gunson, N, Sieińska, W, Romeo, M , Dondrup, C & Lemon, O 2024, A Multi-party Conversational Social Robot Using LLMs. in HRI '24: Companion of the 2024 ACM/IEEE International Conference on Human-Robot Interaction. Association for Computing Machinery, pp. 1273-1275, 19th Annual ACM/IEEE International Conference on Human-Robot Interaction 2024, Boulder, Colorado, United States, 11/03/24. https://doi.org/10.1145/3610978.3641112

TY - GEN

T1 - A Multi-party Conversational Social Robot Using LLMs

AU - Addlesee, Angus

AU - Cherakara, Neeraj

AU - Nelson, Nivan

AU - García, Daniel Hernández

AU - Gunson, Nancie

AU - Sieińska, Weronika

AU - Romeo, Marta

AU - Dondrup, Christian

AU - Lemon, Oliver

PY - 2024/3/11

Y1 - 2024/3/11

N2 - In this paper, we describe our setting and the architecture of our LLM-based dialogue system embodied in a social robot and able to have multi-party conversations. Each component is detailed, and a video of the full system is available with the appropriate components highlighted in real-time. Our system decides when it should take its turn, generates human-like clarification requests when the patient pauses mid-utterance, answers in-domain questions (grounding to the in-prompt knowledge), and responds appropriately to out-of-domain requests (like generating jokes or quizzes).

AB - In this paper, we describe our setting and the architecture of our LLM-based dialogue system embodied in a social robot and able to have multi-party conversations. Each component is detailed, and a video of the full system is available with the appropriate components highlighted in real-time. Our system decides when it should take its turn, generates human-like clarification requests when the patient pauses mid-utterance, answers in-domain questions (grounding to the in-prompt knowledge), and responds appropriately to out-of-domain requests (like generating jokes or quizzes).

KW - accessibility

KW - conversational AI

KW - human-robot interaction

KW - large language models

KW - multi-party dialogue

KW - social robots

UR - http://www.scopus.com/inward/record.url?scp=85188129436&partnerID=8YFLogxK

U2 - 10.1145/3610978.3641112

DO - 10.1145/3610978.3641112

M3 - Conference contribution

AN - SCOPUS:85188129436

SP - 1273

EP - 1275

BT - HRI '24: Companion of the 2024 ACM/IEEE International Conference on Human-Robot Interaction

PB - Association for Computing Machinery

T2 - 19th Annual ACM/IEEE International Conference on Human-Robot Interaction 2024

Y2 - 11 March 2024 through 15 March 2024

ER -

A Multi-party Conversational Social Robot Using LLMs

Abstract

Conference

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this