We present a multi-modal dialogue system for interactive learning of perceptually grounded word meanings from a human tutor. The system integrates an incremental, semantic, and bi-directional grammar framework – Dynamic Syntax and Type Theory with Records (DS-TTR1 , (Eshghi et al., 2012; Kempson et al., 2001)) – with a set of visual classifiers that are learned throughout the interaction and which ground the semantic/contextual representations that it produces (c.f. Kennington & Schlangen (2015)) Our approach extends Dobnik et al. (2012) in integrating perception (vision in this case) and language within a single formal system: Type Theory with Records (TTR (Cooper, 2005)). The combination of deep semantic representations in TTR with an incremental grammar (Dynamic Syntax) allows for complex multi-turn dialogues to be parsed and generated (Eshghi et al., 2015). These include clarification interaction, corrections, ellipsis, and utterance continuations (see e.g. the dialogue in Fig. 1).
|Title of host publication||JerSem|
|Subtitle of host publication||Proceedings of the 20th Workshop on the Semantics and Pragmatics of Dialogue|
|Editors||Julie Hunter, Mandy Simons, Matthew Stone|
|Number of pages||2|
|Publication status||Published - 16 Jul 2016|
|Event||20th Workshop Series on the Semantics and Pragmatics of Dialogue 2016 - New Brunswick, United States|
Duration: 16 Jul 2016 → 18 Jul 2016
|Conference||20th Workshop Series on the Semantics and Pragmatics of Dialogue 2016|
|Period||16/07/16 → 18/07/16|
- Natural language processing
- Robotics, Development, Language action, Social interaction, Learning
- Artificial intelligence
Yu, Y., Eshghi, A., & Lemon, O. (2016). An Incremental Dialogue System for Learning Visually Grounded Language (demonstration system). In J. Hunter, M. Simons, & M. Stone (Eds.), JerSem: Proceedings of the 20th Workshop on the Semantics and Pragmatics of Dialogue (pp. 120-121). (SemDial Proceedings). Rutgers University.