Real user evaluation of a POMDP spoken dialogue system using automatic belief compression

Paul A. Crook, Simon Keizer*, Zhuoran Wang, Wenshuo Tang, Oliver Lemon

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

17 Citations (Scopus)


This article describes an evaluation of a POMDP-based spoken dialogue system (SDS), using crowdsourced calls with real users. The evaluation compares a "Hidden Information State" POMDP system which uses a hand-crafted compression of the belief space, with the same system instead using an automatically computed belief space compression. Automatically computed compressions are a way of introducing automation into the design process of statistical SDSs and promise a principled way of reducing the size of the very large belief spaces which often make POMDP approaches intractable. This is the first empirical comparison of manual and automatic approaches on a problem of realistic scale (restaurant, pub and coffee shop domain) with real users. The evaluation took 2193 calls from 85 users. After filtering for minimal user participation the two systems were compared on more than 1000 calls. (C) 2013 Elsevier Ltd. All rights reserved.

Original languageEnglish
Pages (from-to)873-887
Number of pages15
JournalComputer Speech and Language
Issue number4
Publication statusPublished - Jul 2014


  • Spoken dialogue systems
  • Dialogue management
  • Belief compression


Dive into the research topics of 'Real user evaluation of a POMDP spoken dialogue system using automatic belief compression'. Together they form a unique fingerprint.

Cite this