GRTr: Generative-Retrieval Transformers for Data-Efficient Dialogue Domain Adaptation

Igor Shalyminov, Alessandro Sordoni, Adam Atkinson, Hannes Schulz

Research output: Contribution to journalArticlepeer-review

2 Downloads (Pure)

Abstract

Domain adaptation has recently become a key problem in dialogue systems research. Deep learning, while being the preferred technique for modeling such systems, works best given massive training data. However, in real-world scenarios, such resources are rarely available for new domains, and the ability to train with a few dialogue examples can be considered essential. Pre-training on large data sources and adapting to the target data has become the standard method for few-shot problems within the deep learning framework. In this paper, we present GRTr, a hybrid generative-retrieval model based on the large-scale general-purpose language model GPT-2 fine-tuned to the multi-domain MetaLWOz dataset. In addition to robust and diverse response generation provided by the GPT-2, our model is able to estimate generation confidence, and is equipped with retrieval logic as a fallback for the cases when the estimate is low. GRTr is the winning entry at the fast domain adaptation task of DSTC-8 in human evaluation (>4% improvement over the 2nd place system). It also attains superior performance to a series of baselines on automated metrics on MetaLWOz and MultiWoz, a multi-domain dataset of goal-oriented dialogues. In this paper, we also conduct a study of GRTr's performance in the setup of limited adaptation data, evaluating the model's overall response prediction performance on MetaLWOz and goal-oriented performance on MultiWoz.

Original languageEnglish
JournalIEEE/ACM Transactions on Audio Speech and Language Processing
Early online date21 Apr 2021
DOIs
Publication statusE-pub ahead of print - 21 Apr 2021

Keywords

  • Adaptation models
  • Context modeling
  • Data models
  • Deep learning
  • dialogue systems
  • domain adaptation
  • Gold
  • natural language processing
  • neural networks
  • Predictive models
  • Task analysis
  • Training

ASJC Scopus subject areas

  • Computer Science (miscellaneous)
  • Acoustics and Ultrasonics
  • Computational Mathematics
  • Electrical and Electronic Engineering

Fingerprint Dive into the research topics of 'GRTr: Generative-Retrieval Transformers for Data-Efficient Dialogue Domain Adaptation'. Together they form a unique fingerprint.

Cite this