Why We Need New Evaluation Metrics for NLG

Jekaterina Novikova; Ondrej Dusek; Amanda Cercas Curry; Verena Rieser

doi:10.18653/v1/D17-1237

Why We Need New Evaluation Metrics for NLG

Jekaterina Novikova, Ondrej Dusek, Amanda Cercas Curry, Verena Rieser

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

113 Citations (Scopus)

Abstract

The majority of NLG evaluation relies on automatic metrics, such as BLEU. In this paper, we motivate the need for novel, system- and data-independent automatic evaluation methods: We investigate a wide range of metrics, including state-of-the-art word-based and novel grammar-based ones, and demonstrate that they only weakly reflect human judgements of system outputs as generated by data-driven, end-to-end NLG. We also show that metric performance is data- and system-specific. Nevertheless, our results also suggest that automatic metrics perform reliably at system-level and can support system development by finding cases where a system performs poorly.

Original language	English
Title of host publication	Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing
Publisher	Association for Computational Linguistics
Pages	2231-2242
Number of pages	12
ISBN (Electronic)	978-1-945626-83-8
DOIs	https://doi.org/10.18653/v1/D17-1237
Publication status	Published - 10 Sept 2017
Event	2017 Conference on Empirical Methods in Natural Language Processing - Øksnehallen, Copenhagen, Denmark Duration: 9 Sept 2017 → 11 Sept 2017

Conference

Conference	2017 Conference on Empirical Methods in Natural Language Processing
Abbreviated title	EMNLP 2017
Country/Territory	Denmark
City	Copenhagen
Period	9/09/17 → 11/09/17

Keywords

natural language generation
natural language processing
evaluation
evaluation metrics

Access to Document

10.18653/v1/D17-1237

http://aclweb.org/anthology/D17-1237.pdf

Human Ratings of Natural Language Generation Outputs
Novikova, J. (Creator), Dusek, O. (Creator), Cercas Curry, A. (Creator) & Rieser, V. (Creator), Heriot-Watt University, 2017
https://github.com/jeknov/EMNLP_17_submission
Dataset

Verena Rieser
- V.T.Rieserhw.acuk
- School of Mathematical & Computer Sciences - Professor
- School of Mathematical & Computer Sciences, Computer Science - Professor
Person: Academic (Research & Teaching)

Cite this

@inproceedings{df05d56244dc4e5297239c3ba3df340f,

title = "Why We Need New Evaluation Metrics for NLG",

abstract = "The majority of NLG evaluation relies on automatic metrics, such as BLEU. In this paper, we motivate the need for novel, system- and data-independent automatic evaluation methods: We investigate a wide range of metrics, including state-of-the-art word-based and novel grammar-based ones, and demonstrate that they only weakly reflect human judgements of system outputs as generated by data-driven, end-to-end NLG. We also show that metric performance is data- and system-specific. Nevertheless, our results also suggest that automatic metrics perform reliably at system-level and can support system development by finding cases where a system performs poorly. ",

keywords = "natural language generation, natural language processing, evaluation, evaluation metrics",

author = "Jekaterina Novikova and Ondrej Dusek and {Cercas Curry}, Amanda and Verena Rieser",

year = "2017",

month = sep,

day = "10",

doi = "10.18653/v1/D17-1237",

language = "English",

pages = "2231--2242",

booktitle = "Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing",

publisher = "Association for Computational Linguistics",

note = "2017 Conference on Empirical Methods in Natural Language Processing , EMNLP 2017 ; Conference date: 09-09-2017 Through 11-09-2017",

}

Novikova, J, Dusek, O, Cercas Curry, A & Rieser, V 2017, Why We Need New Evaluation Metrics for NLG. in Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, pp. 2231-2242, 2017 Conference on Empirical Methods in Natural Language Processing , Copenhagen, Denmark, 9/09/17. https://doi.org/10.18653/v1/D17-1237

TY - GEN

T1 - Why We Need New Evaluation Metrics for NLG

AU - Novikova, Jekaterina

AU - Dusek, Ondrej

AU - Cercas Curry, Amanda

AU - Rieser, Verena

PY - 2017/9/10

Y1 - 2017/9/10

N2 - The majority of NLG evaluation relies on automatic metrics, such as BLEU. In this paper, we motivate the need for novel, system- and data-independent automatic evaluation methods: We investigate a wide range of metrics, including state-of-the-art word-based and novel grammar-based ones, and demonstrate that they only weakly reflect human judgements of system outputs as generated by data-driven, end-to-end NLG. We also show that metric performance is data- and system-specific. Nevertheless, our results also suggest that automatic metrics perform reliably at system-level and can support system development by finding cases where a system performs poorly.

AB - The majority of NLG evaluation relies on automatic metrics, such as BLEU. In this paper, we motivate the need for novel, system- and data-independent automatic evaluation methods: We investigate a wide range of metrics, including state-of-the-art word-based and novel grammar-based ones, and demonstrate that they only weakly reflect human judgements of system outputs as generated by data-driven, end-to-end NLG. We also show that metric performance is data- and system-specific. Nevertheless, our results also suggest that automatic metrics perform reliably at system-level and can support system development by finding cases where a system performs poorly.

KW - natural language generation

KW - natural language processing

KW - evaluation

KW - evaluation metrics

U2 - 10.18653/v1/D17-1237

DO - 10.18653/v1/D17-1237

M3 - Conference contribution

SP - 2231

EP - 2242

BT - Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing

PB - Association for Computational Linguistics

T2 - 2017 Conference on Empirical Methods in Natural Language Processing

Y2 - 9 September 2017 through 11 September 2017

ER -

Why We Need New Evaluation Metrics for NLG

Abstract

Conference

Keywords

Access to Document

Fingerprint

Datasets

Human Ratings of Natural Language Generation Outputs

Profiles

Verena Rieser

Cite this