Referenceless Quality Estimation for Natural Language Generation

Ondrej Dusek; Jekaterina Novikova; Verena Rieser

Referenceless Quality Estimation for Natural Language Generation

Ondrej Dusek
, Jekaterina Novikova
, Verena Rieser

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

73 Downloads (Pure)

Abstract

Traditional automatic evaluation measures for natural language generation (NLG) use costly human-authored references to estimate the quality of a system output. In this paper, we propose a referenceless quality estimation (QE) approach based on recurrent neural networks, which predicts a quality score for a NLG system output by comparing it to the source meaning representation only. Our method outperforms traditional metrics and a constant baseline in most respects; we also show that synthetic data helps to increase correlation results by 21% compared to the base system. Our results are comparable to results obtained in similar QE tasks despite the more challenging setting.

Original language	English
Title of host publication	Proceedings of the 34 th International Conference on Machine Learning, Sydney, Australia, PMLR 70, 2017
Publisher	ICML
Publication status	Published - 10 Aug 2017
Event	1st Workshop on Learning to Generate Natural Language - ICML Conference, International Convention Centre, Sydney, Australia Duration: 10 Aug 2017 → 10 Aug 2017 Conference number: 1 https://sites.google.com/site/langgen17/accepted-papers

Workshop

Workshop	1st Workshop on Learning to Generate Natural Language
Abbreviated title	LGNL
Country/Territory	Australia
City	Sydney
Period	10/08/17 → 10/08/17
Internet address	https://sites.google.com/site/langgen17/accepted-papers

Access to Document

Download pdfFinal published version, 271 KBLicence: Other

Cite this

@inproceedings{5a9a8d47cd924630a6f3290b77372185,

title = "Referenceless Quality Estimation for Natural Language Generation",

abstract = "Traditional automatic evaluation measures for natural language generation (NLG) use costly human-authored references to estimate the quality of a system output. In this paper, we propose a referenceless quality estimation (QE) approach based on recurrent neural networks, which predicts a quality score for a NLG system output by comparing it to the source meaning representation only. Our method outperforms traditional metrics and a constant baseline in most respects; we also show that synthetic data helps to increase correlation results by 21\% compared to the base system. Our results are comparable to results obtained in similar QE tasks despite the more challenging setting. ",

author = "Ondrej Dusek and Jekaterina Novikova and Verena Rieser",

year = "2017",

month = aug,

day = "10",

language = "English",

booktitle = "Proceedings of the 34 th International Conference on Machine Learning, Sydney, Australia, PMLR 70, 2017",

publisher = "ICML",

note = "1st Workshop on Learning to Generate Natural Language, LGNL ; Conference date: 10-08-2017 Through 10-08-2017",

url = "https://sites.google.com/site/langgen17/accepted-papers",

}

TY - GEN

T1 - Referenceless Quality Estimation for Natural Language Generation

AU - Dusek, Ondrej

AU - Novikova, Jekaterina

AU - Rieser, Verena

N1 - Conference code: 1

PY - 2017/8/10

Y1 - 2017/8/10

N2 - Traditional automatic evaluation measures for natural language generation (NLG) use costly human-authored references to estimate the quality of a system output. In this paper, we propose a referenceless quality estimation (QE) approach based on recurrent neural networks, which predicts a quality score for a NLG system output by comparing it to the source meaning representation only. Our method outperforms traditional metrics and a constant baseline in most respects; we also show that synthetic data helps to increase correlation results by 21% compared to the base system. Our results are comparable to results obtained in similar QE tasks despite the more challenging setting.

AB - Traditional automatic evaluation measures for natural language generation (NLG) use costly human-authored references to estimate the quality of a system output. In this paper, we propose a referenceless quality estimation (QE) approach based on recurrent neural networks, which predicts a quality score for a NLG system output by comparing it to the source meaning representation only. Our method outperforms traditional metrics and a constant baseline in most respects; we also show that synthetic data helps to increase correlation results by 21% compared to the base system. Our results are comparable to results obtained in similar QE tasks despite the more challenging setting.

UR - https://github.com/tuetschek/ratpred

M3 - Conference contribution

BT - Proceedings of the 34 th International Conference on Machine Learning, Sydney, Australia, PMLR 70, 2017

PB - ICML

T2 - 1st Workshop on Learning to Generate Natural Language

Y2 - 10 August 2017 through 10 August 2017

ER -

Referenceless Quality Estimation for Natural Language Generation

Abstract

Workshop

Access to Document

Other files and links

Fingerprint

Cite this