A Review of Evaluation Techniques for Social Dialogue Systems

Amanda Cercas Curry; Helen Hastie; Verena Rieser

doi:10.1145/3139491.3139504

A Review of Evaluation Techniques for Social Dialogue Systems

Amanda Cercas Curry, Helen Hastie, Verena Rieser

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

10 Citations (Scopus)

Abstract

In contrast with goal-oriented dialogue, social dialogue has no clear measure of task success. Consequently, evaluation of these systems is notoriously hard. In this paper, we review current evaluation methods, focusing on automatic metrics. We conclude that turn-based metrics often ignore the context and do not account for the fact that several replies are valid, while end-of-dialogue rewards are mainly hand-crafted. Both lack grounding in human perceptions.

Original language	English
Title of host publication	ISIAA 2017: Proceedings of the 1st ACM SIGCHI International Workshop on Investigating Social Interactions with Artificial Agents
Publisher	Association for Computing Machinery
Pages	25-26
Number of pages	2
ISBN (Print)	9781450355582
DOIs	https://doi.org/10.1145/3139491.3139504
Publication status	Published - 13 Nov 2017
Event	19th ACM International Conference on Multimodal Interaction 2017 - Glasgow, United Kingdom Duration: 13 Nov 2017 → 17 Nov 2017

Conference

Conference	19th ACM International Conference on Multimodal Interaction 2017
Abbreviated title	ICMI 2017
Country/Territory	United Kingdom
City	Glasgow
Period	13/11/17 → 17/11/17

Access to Document

10.1145/3139491.3139504

Cite this

@inproceedings{099e4a14ef5949bc9f1eb28ea9cee842,

title = "A Review of Evaluation Techniques for Social Dialogue Systems",

abstract = "In contrast with goal-oriented dialogue, social dialogue has no clear measure of task success. Consequently, evaluation of these systems is notoriously hard. In this paper, we review current evaluation methods, focusing on automatic metrics. We conclude that turn-based metrics often ignore the context and do not account for the fact that several replies are valid, while end-of-dialogue rewards are mainly hand-crafted. Both lack grounding in human perceptions.",

author = "\{Cercas Curry\}, Amanda and Helen Hastie and Verena Rieser",

year = "2017",

month = nov,

day = "13",

doi = "10.1145/3139491.3139504",

language = "English",

isbn = "9781450355582",

pages = "25--26",

booktitle = "ISIAA 2017: Proceedings of the 1st ACM SIGCHI International Workshop on Investigating Social Interactions with Artificial Agents",

publisher = "Association for Computing Machinery",

address = "United States",

note = "19th ACM International Conference on Multimodal Interaction 2017, ICMI 2017 ; Conference date: 13-11-2017 Through 17-11-2017",

}

Cercas Curry, A, Hastie, H & Rieser, V 2017, A Review of Evaluation Techniques for Social Dialogue Systems. in ISIAA 2017: Proceedings of the 1st ACM SIGCHI International Workshop on Investigating Social Interactions with Artificial Agents. Association for Computing Machinery, pp. 25-26, 19th ACM International Conference on Multimodal Interaction 2017, Glasgow, United Kingdom, 13/11/17. https://doi.org/10.1145/3139491.3139504

A Review of Evaluation Techniques for Social Dialogue Systems. / Cercas Curry, Amanda; Hastie, Helen; Rieser, Verena.
ISIAA 2017: Proceedings of the 1st ACM SIGCHI International Workshop on Investigating Social Interactions with Artificial Agents. Association for Computing Machinery, 2017. p. 25-26.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - A Review of Evaluation Techniques for Social Dialogue Systems

AU - Cercas Curry, Amanda

AU - Hastie, Helen

AU - Rieser, Verena

PY - 2017/11/13

Y1 - 2017/11/13

N2 - In contrast with goal-oriented dialogue, social dialogue has no clear measure of task success. Consequently, evaluation of these systems is notoriously hard. In this paper, we review current evaluation methods, focusing on automatic metrics. We conclude that turn-based metrics often ignore the context and do not account for the fact that several replies are valid, while end-of-dialogue rewards are mainly hand-crafted. Both lack grounding in human perceptions.

AB - In contrast with goal-oriented dialogue, social dialogue has no clear measure of task success. Consequently, evaluation of these systems is notoriously hard. In this paper, we review current evaluation methods, focusing on automatic metrics. We conclude that turn-based metrics often ignore the context and do not account for the fact that several replies are valid, while end-of-dialogue rewards are mainly hand-crafted. Both lack grounding in human perceptions.

U2 - 10.1145/3139491.3139504

DO - 10.1145/3139491.3139504

M3 - Conference contribution

SN - 9781450355582

SP - 25

EP - 26

BT - ISIAA 2017: Proceedings of the 1st ACM SIGCHI International Workshop on Investigating Social Interactions with Artificial Agents

PB - Association for Computing Machinery

T2 - 19th ACM International Conference on Multimodal Interaction 2017

Y2 - 13 November 2017 through 17 November 2017

ER -

A Review of Evaluation Techniques for Social Dialogue Systems

Abstract

Conference

Access to Document

Fingerprint

Cite this