A Learning-based Co-Speech Gesture Generation System for Social Robots

Xiangqi Li; Christian Dondrup

doi:10.1145/3687272.3690915

A Learning-based Co-Speech Gesture Generation System for Social Robots

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

Co-speech gestures enhance both human-human and human-robot interactions. This paper examines the efficacy of a data-driven approach for generating synchronised co-speech gestures in three social robots to improve social interactions. Building on a sequence-to-sequence model, which maps speech to gestures [21], this work uses the Talking With Hands 16.2M dataset [11] to generate natural gestures for face-to-face conversations. Additionally, we address synchronisation issues identified in the original study. The model’s generality is tested on three robots—NAO, Pepper, and ARI. Objective and subjective evaluations, confirm that a data-driven approach effectively generates synchronised co-speech gestures.

Original language	English
Title of host publication	HAI '24: Proceedings of the 12th International Conference on Human-Agent Interaction
Publisher	Association for Computing Machinery
Pages	453-455
Number of pages	3
ISBN (Print)	9798400711787
DOIs	https://doi.org/10.1145/3687272.3690915
Publication status	Published - 24 Nov 2024
Event	12th International Conference on Human-Agent Interaction 2024 - Swansea University, Swansea, United Kingdom Duration: 24 Nov 2024 → 27 Nov 2024 https://hai-conference.net/hai2024/

Conference

Conference	12th International Conference on Human-Agent Interaction 2024
Country/Territory	United Kingdom
City	Swansea
Period	24/11/24 → 27/11/24
Internet address	https://hai-conference.net/hai2024/

Access to Document

10.1145/3687272.3690915

Cite this

@inproceedings{2d75b9fdb7ef458ea396e6087d1a2543,

title = "A Learning-based Co-Speech Gesture Generation System for Social Robots",

abstract = "Co-speech gestures enhance both human-human and human-robot interactions. This paper examines the efficacy of a data-driven approach for generating synchronised co-speech gestures in three social robots to improve social interactions. Building on a sequence-to-sequence model, which maps speech to gestures [21], this work uses the Talking With Hands 16.2M dataset [11] to generate natural gestures for face-to-face conversations. Additionally, we address synchronisation issues identified in the original study. The model{\textquoteright}s generality is tested on three robots—NAO, Pepper, and ARI. Objective and subjective evaluations, confirm that a data-driven approach effectively generates synchronised co-speech gestures.",

author = "Xiangqi Li and Christian Dondrup",

year = "2024",

month = nov,

day = "24",

doi = "10.1145/3687272.3690915",

language = "English",

isbn = "9798400711787",

pages = "453--455",

booktitle = "HAI '24: Proceedings of the 12th International Conference on Human-Agent Interaction",

publisher = "Association for Computing Machinery",

address = "United States",

note = "12th International Conference on Human-Agent Interaction 2024<br/> ; Conference date: 24-11-2024 Through 27-11-2024",

url = "https://hai-conference.net/hai2024/",

}

Li, X & Dondrup, C 2024, A Learning-based Co-Speech Gesture Generation System for Social Robots. in HAI '24: Proceedings of the 12th International Conference on Human-Agent Interaction. Association for Computing Machinery, pp. 453-455, 12th International Conference on Human-Agent Interaction 2024
, Swansea, United Kingdom, 24/11/24. https://doi.org/10.1145/3687272.3690915

TY - GEN

T1 - A Learning-based Co-Speech Gesture Generation System for Social Robots

AU - Li, Xiangqi

AU - Dondrup, Christian

PY - 2024/11/24

Y1 - 2024/11/24

N2 - Co-speech gestures enhance both human-human and human-robot interactions. This paper examines the efficacy of a data-driven approach for generating synchronised co-speech gestures in three social robots to improve social interactions. Building on a sequence-to-sequence model, which maps speech to gestures [21], this work uses the Talking With Hands 16.2M dataset [11] to generate natural gestures for face-to-face conversations. Additionally, we address synchronisation issues identified in the original study. The model’s generality is tested on three robots—NAO, Pepper, and ARI. Objective and subjective evaluations, confirm that a data-driven approach effectively generates synchronised co-speech gestures.

AB - Co-speech gestures enhance both human-human and human-robot interactions. This paper examines the efficacy of a data-driven approach for generating synchronised co-speech gestures in three social robots to improve social interactions. Building on a sequence-to-sequence model, which maps speech to gestures [21], this work uses the Talking With Hands 16.2M dataset [11] to generate natural gestures for face-to-face conversations. Additionally, we address synchronisation issues identified in the original study. The model’s generality is tested on three robots—NAO, Pepper, and ARI. Objective and subjective evaluations, confirm that a data-driven approach effectively generates synchronised co-speech gestures.

U2 - 10.1145/3687272.3690915

DO - 10.1145/3687272.3690915

M3 - Conference contribution

SN - 9798400711787

SP - 453

EP - 455

BT - HAI '24: Proceedings of the 12th International Conference on Human-Agent Interaction

PB - Association for Computing Machinery

T2 - 12th International Conference on Human-Agent Interaction 2024<br/>

Y2 - 24 November 2024 through 27 November 2024

ER -

A Learning-based Co-Speech Gesture Generation System for Social Robots

Abstract

Conference

Access to Document

Fingerprint

Cite this