GRTr: Generative-Retrieval Transformers for Data-Efficient Dialogue Domain Adaptation

Igor Shalyminov, Alessandro Sordoni, Adam Atkinson, Hannes Schulz

Research output: Contribution to journalArticlepeer-review

2 Citations (Scopus)
115 Downloads (Pure)


Domain adaptation has recently become a key problem in dialogue systems research. Deep learning, while being the preferred technique for modeling such systems, works best given massive training data. However, in real-world scenarios, such resources are rarely available for new domains, and the ability to train with a few dialogue examples can be considered essential. Pre-training on large data sources and adapting to the target data has become the standard method for few-shot problems within the deep learning framework. In this paper, we present grtr, a hybrid generative-retrieval model based on the large-scale general-purpose language model GPT[2] fine-tuned to the multi-domain metalwoz dataset. In addition to robust and diverse response generation provided by the GPT[2], our model is able to estimate generation confidence, and is equipped with retrieval logic as a fallback for the cases when the estimate is low. grtr is the winning entry at the fast domain adaptation task of DSTC-8 in human evaluation (>4% improvement over the 2nd place system). It also attains superior performance to a series of baselines on automated metrics on metalwoz and multiwoz, a multi-domain dataset of goal-oriented dialogues. In this paper, we also conduct a study of grtr's performance in the setup of limited adaptation data, evaluating the model's overall response prediction performance on metalwoz and goal-oriented performance on multiwoz.

Original languageEnglish
Pages (from-to)2484-2492
Number of pages9
JournalIEEE/ACM Transactions on Audio Speech and Language Processing
Early online date21 Apr 2021
Publication statusPublished - 2021


  • Adaptation models
  • Context modeling
  • Data models
  • Deep learning
  • dialogue systems
  • domain adaptation
  • Gold
  • natural language processing
  • neural networks
  • Predictive models
  • Task analysis
  • Training

ASJC Scopus subject areas

  • Computer Science (miscellaneous)
  • Acoustics and Ultrasonics
  • Computational Mathematics
  • Electrical and Electronic Engineering


Dive into the research topics of 'GRTr: Generative-Retrieval Transformers for Data-Efficient Dialogue Domain Adaptation'. Together they form a unique fingerprint.

Cite this