Adaptive information presentation for spoken dialogue systems: Evaluation with real users

We present evaluation results with human subjects
for a novel data-driven approach to Natural
Language Generation in spoken dialogue
systems. We evaluate a trained Information
Presentation (IP) strategy in a deployed
tourist-information spoken dialogue system.
The IP problem is formulated as statistical decision
making under uncertainty using Reinforcement
Learning, where both content planning
and attribute selection are jointly optimised
based on data collected in a Wizard-of-
Oz study. After earlier work testing and training
this model in simulation, we now present
results from an extensive online user study,
involving 131 users and more than 800 test
dialogues, which explores its contribution to
overall ‘global’ task success. We find that
the trained Information Presentation strategy
significantly improves dialogue task completion,
with up to a 9.7% increase (30% relative)
compared to the deployed dialogue system
which uses conventional, hand-coded presentation
prompts. We also present subjective
evaluation results and discuss the implications
of these results for future work in dialogue
management and NLG.