A Knowledge-Grounded Multimodal Search-Based Conversational Agent

Shubham Agarwal; Ondrej Dusek; Ioannis Konstas; Verena Rieser

A Knowledge-Grounded Multimodal Search-Based Conversational Agent

Shubham Agarwal, Ondrej Dusek, Ioannis Konstas, Verena Rieser

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

Multimodal search-based dialogue is a challenging new task: It extends visually grounded question answering systems into multi-turn conversations with access to an external database. We address this new challenge by learning a neural response generation system from the recently released Multimodal Dialogue (MMD) dataset (Saha et al., 2017). We introduce a knowledge-grounded multimodal conversational model where an encoded knowledge base (KB) representation is appended to the decoder input. Our model substantially outperforms strong baselines in terms of text-based similarity measures (over 9 BLEU points, 3 of which are solely due to the use of additional information from the KB).

Original language	English
Title of host publication	Proceedings of the 2018 EMNLP Workshop SCAI
Subtitle of host publication	The 2nd International Workshop on Search-Oriented Conversational AI
Publisher	Association for Computational Linguistics
Pages	59-66
Number of pages	8
ISBN (Electronic)	9781948087759
Publication status	Published - 31 Oct 2018
Event	2nd International Workshop on Search-Oriented Conversational AI - Brussels, Belgium Duration: 31 Oct 2018 → 31 Oct 2018

Workshop

Workshop	2nd International Workshop on Search-Oriented Conversational AI
Country/Territory	Belgium
City	Brussels
Period	31/10/18 → 31/10/18

Access to Document

https://aclweb.org/anthology/W18-5709

Cite this

@inproceedings{ef2e4de8c52b4fe995673e45404c8120,

title = "A Knowledge-Grounded Multimodal Search-Based Conversational Agent",

abstract = "Multimodal search-based dialogue is a challenging new task: It extends visually grounded question answering systems into multi-turn conversations with access to an external database. We address this new challenge by learning a neural response generation system from the recently released Multimodal Dialogue (MMD) dataset (Saha et al., 2017). We introduce a knowledge-grounded multimodal conversational model where an encoded knowledge base (KB) representation is appended to the decoder input. Our model substantially outperforms strong baselines in terms of text-based similarity measures (over 9 BLEU points, 3 of which are solely due to the use of additional information from the KB).",

author = "Shubham Agarwal and Ondrej Dusek and Ioannis Konstas and Verena Rieser",

year = "2018",

month = oct,

day = "31",

language = "English",

pages = "59--66",

booktitle = "Proceedings of the 2018 EMNLP Workshop SCAI",

publisher = "Association for Computational Linguistics",

note = "2nd International Workshop on Search-Oriented Conversational AI ; Conference date: 31-10-2018 Through 31-10-2018",

}

Agarwal, S, Dusek, O, Konstas, I & Rieser, V 2018, A Knowledge-Grounded Multimodal Search-Based Conversational Agent. in Proceedings of the 2018 EMNLP Workshop SCAI: The 2nd International Workshop on Search-Oriented Conversational AI. Association for Computational Linguistics, pp. 59-66, 2nd International Workshop on Search-Oriented Conversational AI, Brussels, Belgium, 31/10/18. <https://aclweb.org/anthology/W18-5709>

A Knowledge-Grounded Multimodal Search-Based Conversational Agent. / Agarwal, Shubham; Dusek, Ondrej; Konstas, Ioannis et al.
Proceedings of the 2018 EMNLP Workshop SCAI: The 2nd International Workshop on Search-Oriented Conversational AI. Association for Computational Linguistics, 2018. p. 59-66.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - A Knowledge-Grounded Multimodal Search-Based Conversational Agent

AU - Agarwal, Shubham

AU - Dusek, Ondrej

AU - Konstas, Ioannis

AU - Rieser, Verena

PY - 2018/10/31

Y1 - 2018/10/31

N2 - Multimodal search-based dialogue is a challenging new task: It extends visually grounded question answering systems into multi-turn conversations with access to an external database. We address this new challenge by learning a neural response generation system from the recently released Multimodal Dialogue (MMD) dataset (Saha et al., 2017). We introduce a knowledge-grounded multimodal conversational model where an encoded knowledge base (KB) representation is appended to the decoder input. Our model substantially outperforms strong baselines in terms of text-based similarity measures (over 9 BLEU points, 3 of which are solely due to the use of additional information from the KB).

AB - Multimodal search-based dialogue is a challenging new task: It extends visually grounded question answering systems into multi-turn conversations with access to an external database. We address this new challenge by learning a neural response generation system from the recently released Multimodal Dialogue (MMD) dataset (Saha et al., 2017). We introduce a knowledge-grounded multimodal conversational model where an encoded knowledge base (KB) representation is appended to the decoder input. Our model substantially outperforms strong baselines in terms of text-based similarity measures (over 9 BLEU points, 3 of which are solely due to the use of additional information from the KB).

M3 - Conference contribution

SP - 59

EP - 66

BT - Proceedings of the 2018 EMNLP Workshop SCAI

PB - Association for Computational Linguistics

T2 - 2nd International Workshop on Search-Oriented Conversational AI

Y2 - 31 October 2018 through 31 October 2018

ER -

A Knowledge-Grounded Multimodal Search-Based Conversational Agent

Abstract

Workshop

Access to Document

Fingerprint

Cite this