Datasets and benchmarks for task-oriented log dialogue ranking task

Xinnuo Xu, Yizhe Zhang, Lars Liden, Sungjin Lee

Research output: Contribution to conferencePaperpeer-review

Abstract

Although the data-driven approaches of some recent bot building platforms make it possible for a wide range of users to easily create dialogue systems, those platforms don't offer tools for quickly identifying which log dialogues contain problems. Thus, in this paper, we (1) introduce a new task, log dialogue ranking, where the ranker places problematic dialogues higher (2) provide a collection of human-bot conversations in the restaurant inquiry task labelled with dialogue quality for ranker training and evaluation (3) present a detailed description of the data collection pipeline, which is entirely based on crowd-sourcing (4) finally report a benchmark result of dialogue ranking, which shows the usability of the data and sets a baseline for future studies.

Original languageEnglish
Pages3920-3924
Number of pages5
DOIs
Publication statusPublished - 2020
Event21st Annual Conference of the International Speech Communication Association 2020 - Shanghai, China
Duration: 25 Oct 202029 Oct 2020

Conference

Conference21st Annual Conference of the International Speech Communication Association 2020
Abbreviated titleINTERSPEECH 2020
CountryChina
CityShanghai
Period25/10/2029/10/20

Keywords

  • Dialogue quality
  • Dialogue ranking
  • Dialogue system
  • Language resource

ASJC Scopus subject areas

  • Language and Linguistics
  • Human-Computer Interaction
  • Signal Processing
  • Software
  • Modelling and Simulation

Fingerprint Dive into the research topics of 'Datasets and benchmarks for task-oriented log dialogue ranking task'. Together they form a unique fingerprint.

Cite this