Why We Need New Evaluation Metrics for NLG

Jekaterina Novikova; Ondrej Dusek; Amanda Cercas Curry; Verena Rieser

doi:10.18653/v1/D17-1237

Why We Need New Evaluation Metrics for NLG

Jekaterina Novikova, Ondrej Dusek, Amanda Cercas Curry, Verena Rieser

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

123 Citations (Scopus)

Abstract

The majority of NLG evaluation relies on automatic metrics, such as BLEU. In this paper, we motivate the need for novel, system- and data-independent automatic evaluation methods: We investigate a wide range of metrics, including state-of-the-art word-based and novel grammar-based ones, and demonstrate that they only weakly reflect human judgements of system outputs as generated by data-driven, end-to-end NLG. We also show that metric performance is data- and system-specific. Nevertheless, our results also suggest that automatic metrics perform reliably at system-level and can support system development by finding cases where a system performs poorly.

Original language	English
Title of host publication	Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing
Publisher	Association for Computational Linguistics
Pages	2231-2242
Number of pages	12
ISBN (Electronic)	978-1-945626-83-8
DOIs	https://doi.org/10.18653/v1/D17-1237
Publication status	Published - 10 Sept 2017
Event	2017 Conference on Empirical Methods in Natural Language Processing - Øksnehallen, Copenhagen, Denmark Duration: 9 Sept 2017 → 11 Sept 2017

Conference

Conference	2017 Conference on Empirical Methods in Natural Language Processing
Abbreviated title	EMNLP 2017
Country/Territory	Denmark
City	Copenhagen
Period	9/09/17 → 11/09/17

Keywords

natural language generation
natural language processing
evaluation
evaluation metrics

Access to Document

10.18653/v1/D17-1237

http://aclweb.org/anthology/D17-1237.pdf

Cite this

@inproceedings{df05d56244dc4e5297239c3ba3df340f,

title = "Why We Need New Evaluation Metrics for NLG",

abstract = "The majority of NLG evaluation relies on automatic metrics, such as BLEU. In this paper, we motivate the need for novel, system- and data-independent automatic evaluation methods: We investigate a wide range of metrics, including state-of-the-art word-based and novel grammar-based ones, and demonstrate that they only weakly reflect human judgements of system outputs as generated by data-driven, end-to-end NLG. We also show that metric performance is data- and system-specific. Nevertheless, our results also suggest that automatic metrics perform reliably at system-level and can support system development by finding cases where a system performs poorly. ",

keywords = "natural language generation, natural language processing, evaluation, evaluation metrics",

author = "Jekaterina Novikova and Ondrej Dusek and \{Cercas Curry\}, Amanda and Verena Rieser",

year = "2017",

month = sep,

day = "10",

doi = "10.18653/v1/D17-1237",

language = "English",

pages = "2231--2242",

booktitle = "Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing",

publisher = "Association for Computational Linguistics",

note = "2017 Conference on Empirical Methods in Natural Language Processing , EMNLP 2017 ; Conference date: 09-09-2017 Through 11-09-2017",

}

Novikova, J, Dusek, O, Cercas Curry, A & Rieser, V 2017, Why We Need New Evaluation Metrics for NLG. in Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, pp. 2231-2242, 2017 Conference on Empirical Methods in Natural Language Processing , Copenhagen, Denmark, 9/09/17. https://doi.org/10.18653/v1/D17-1237

TY - GEN

T1 - Why We Need New Evaluation Metrics for NLG

AU - Novikova, Jekaterina

AU - Dusek, Ondrej

AU - Cercas Curry, Amanda

AU - Rieser, Verena

PY - 2017/9/10

Y1 - 2017/9/10

N2 - The majority of NLG evaluation relies on automatic metrics, such as BLEU. In this paper, we motivate the need for novel, system- and data-independent automatic evaluation methods: We investigate a wide range of metrics, including state-of-the-art word-based and novel grammar-based ones, and demonstrate that they only weakly reflect human judgements of system outputs as generated by data-driven, end-to-end NLG. We also show that metric performance is data- and system-specific. Nevertheless, our results also suggest that automatic metrics perform reliably at system-level and can support system development by finding cases where a system performs poorly.

AB - The majority of NLG evaluation relies on automatic metrics, such as BLEU. In this paper, we motivate the need for novel, system- and data-independent automatic evaluation methods: We investigate a wide range of metrics, including state-of-the-art word-based and novel grammar-based ones, and demonstrate that they only weakly reflect human judgements of system outputs as generated by data-driven, end-to-end NLG. We also show that metric performance is data- and system-specific. Nevertheless, our results also suggest that automatic metrics perform reliably at system-level and can support system development by finding cases where a system performs poorly.

KW - natural language generation

KW - natural language processing

KW - evaluation

KW - evaluation metrics

UR - https://www.scopus.com/pages/publications/85049104951

U2 - 10.18653/v1/D17-1237

DO - 10.18653/v1/D17-1237

M3 - Conference contribution

SP - 2231

EP - 2242

BT - Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing

PB - Association for Computational Linguistics

T2 - 2017 Conference on Empirical Methods in Natural Language Processing

Y2 - 9 September 2017 through 11 September 2017

ER -

Why We Need New Evaluation Metrics for NLG

Abstract

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Human Ratings of Natural Language Generation Outputs

Cite this

Why We Need New Evaluation Metrics for NLG

Abstract

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Datasets

Human Ratings of Natural Language Generation Outputs

Cite this