Real user evaluation of a POMDP spoken dialogue system using automatic belief compression

Paul A. Crook, Simon Keizer, Zhuoran Wang, Wenshuo Tang, Oliver Lemon

Research output: Contribution to journalArticle

11 Citations (Scopus)

Abstract

This article describes an evaluation of a POMDP-based spoken dialogue system (SDS), using crowdsourced calls with real users. The evaluation compares a "Hidden Information State" POMDP system which uses a hand-crafted compression of the belief space, with the same system instead using an automatically computed belief space compression. Automatically computed compressions are a way of introducing automation into the design process of statistical SDSs and promise a principled way of reducing the size of the very large belief spaces which often make POMDP approaches intractable. This is the first empirical comparison of manual and automatic approaches on a problem of realistic scale (restaurant, pub and coffee shop domain) with real users. The evaluation took 2193 calls from 85 users. After filtering for minimal user participation the two systems were compared on more than 1000 calls. (C) 2013 Elsevier Ltd. All rights reserved.

Original languageEnglish
Pages (from-to)873-887
Number of pages15
JournalComputer Speech and Language
Volume28
Issue number4
DOIs
Publication statusPublished - Jul 2014

Keywords

  • Spoken dialogue systems
  • Dialogue management
  • Belief compression
  • UPDATE

Cite this