RankME: Reliable Human Ratings for Natural Language Generation

Jekaterina Novikova; Ondrej Dusek; Verena Rieser

doi:10.18653/v1/N18-2012

RankME: Reliable Human Ratings for Natural Language Generation

Jekaterina Novikova, Ondrej Dusek, Verena Rieser

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

66 Citations (Scopus)

Abstract

Human evaluation for natural language generation (NLG) often suffers from inconsistent user ratings. While previous research tends to attribute this problem to individual user preferences, we show that the quality of human judgements can also be improved by experimental design. We present a novel rank-based magnitude estimation method (RankME), which combines the use of continuous scales and relative assessments. We show that RankME significantly improves the reliability and consistency of human ratings compared to traditional evaluation methods. In addition, we show that it is possible to evaluate NLG systems according to multiple, distinct criteria, which is important for error analysis. Finally, we demonstrate that RankME, in combination with Bayesian estimation of system quality, is a cost-effective alternative for ranking multiple NLG systems.

Original language	English
Title of host publication	Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)
Publisher	Association for Computational Linguistics
Pages	72-78
Number of pages	7
ISBN (Electronic)	9781948087292
DOIs	https://doi.org/10.18653/v1/N18-2012
Publication status	Published - 1 Jun 2018
Event	16th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies - New Orleans, United States Duration: 1 Jun 2018 → 6 Jun 2018

Conference

Conference	16th Annual Conference of the North American Chapter of the Association for Computational Linguistics
Abbreviated title	NAACL HLT 2018
Country/Territory	United States
City	New Orleans
Period	1/06/18 → 6/06/18

Access to Document

10.18653/v1/N18-2012

Cite this

Novikova, J., Dusek, O., & Rieser, V. (2018). RankME: Reliable Human Ratings for Natural Language Generation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) (pp. 72-78). Association for Computational Linguistics. https://doi.org/10.18653/v1/N18-2012

@inproceedings{0a9f3e4f1a1a4befa72eca465bbc5423,

title = "RankME: Reliable Human Ratings for Natural Language Generation",

abstract = "Human evaluation for natural language generation (NLG) often suffers from inconsistent user ratings. While previous research tends to attribute this problem to individual user preferences, we show that the quality of human judgements can also be improved by experimental design. We present a novel rank-based magnitude estimation method (RankME), which combines the use of continuous scales and relative assessments. We show that RankME significantly improves the reliability and consistency of human ratings compared to traditional evaluation methods. In addition, we show that it is possible to evaluate NLG systems according to multiple, distinct criteria, which is important for error analysis. Finally, we demonstrate that RankME, in combination with Bayesian estimation of system quality, is a cost-effective alternative for ranking multiple NLG systems.",

author = "Jekaterina Novikova and Ondrej Dusek and Verena Rieser",

year = "2018",

month = jun,

day = "1",

doi = "10.18653/v1/N18-2012",

language = "English",

pages = "72--78",

booktitle = "Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)",

publisher = "Association for Computational Linguistics",

note = "16th Annual Conference of the North American Chapter of the Association for Computational Linguistics : Human Language Technologies, NAACL HLT 2018 ; Conference date: 01-06-2018 Through 06-06-2018",

}

Novikova, J, Dusek, O & Rieser, V 2018, RankME: Reliable Human Ratings for Natural Language Generation. in Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers). Association for Computational Linguistics, pp. 72-78, 16th Annual Conference of the North American Chapter of the Association for Computational Linguistics, New Orleans, Louisiana, United States, 1/06/18. https://doi.org/10.18653/v1/N18-2012

RankME: Reliable Human Ratings for Natural Language Generation. / Novikova, Jekaterina; Dusek, Ondrej; Rieser, Verena.
Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers). Association for Computational Linguistics, 2018. p. 72-78.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - RankME: Reliable Human Ratings for Natural Language Generation

AU - Novikova, Jekaterina

AU - Dusek, Ondrej

AU - Rieser, Verena

PY - 2018/6/1

Y1 - 2018/6/1

N2 - Human evaluation for natural language generation (NLG) often suffers from inconsistent user ratings. While previous research tends to attribute this problem to individual user preferences, we show that the quality of human judgements can also be improved by experimental design. We present a novel rank-based magnitude estimation method (RankME), which combines the use of continuous scales and relative assessments. We show that RankME significantly improves the reliability and consistency of human ratings compared to traditional evaluation methods. In addition, we show that it is possible to evaluate NLG systems according to multiple, distinct criteria, which is important for error analysis. Finally, we demonstrate that RankME, in combination with Bayesian estimation of system quality, is a cost-effective alternative for ranking multiple NLG systems.

AB - Human evaluation for natural language generation (NLG) often suffers from inconsistent user ratings. While previous research tends to attribute this problem to individual user preferences, we show that the quality of human judgements can also be improved by experimental design. We present a novel rank-based magnitude estimation method (RankME), which combines the use of continuous scales and relative assessments. We show that RankME significantly improves the reliability and consistency of human ratings compared to traditional evaluation methods. In addition, we show that it is possible to evaluate NLG systems according to multiple, distinct criteria, which is important for error analysis. Finally, we demonstrate that RankME, in combination with Bayesian estimation of system quality, is a cost-effective alternative for ranking multiple NLG systems.

U2 - 10.18653/v1/N18-2012

DO - 10.18653/v1/N18-2012

M3 - Conference contribution

SP - 72

EP - 78

BT - Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)

PB - Association for Computational Linguistics

T2 - 16th Annual Conference of the North American Chapter of the Association for Computational Linguistics

Y2 - 1 June 2018 through 6 June 2018

ER -

RankME: Reliable Human Ratings for Natural Language Generation

Abstract

Conference

Access to Document

Fingerprint

Cite this