A systematic review of reproducibility research in natural language processing

Anya Belz; Shubham Agarwal; Anastasia Shimorina; Ehud Reiter

A systematic review of reproducibility research in natural language processing

Anya Belz, Shubham Agarwal, Anastasia Shimorina, Ehud Reiter

School of Mathematical & Computer Sciences

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

41 Citations (Scopus)

Abstract

Against the background of what has been termed a reproducibility crisis in science, the NLP field is becoming increasingly interested in, and conscientious about, the reproducibility of its results. The past few years have seen an impressive range of new initiatives, events and active research in the area. However, the field is far from reaching a consensus about how reproducibility should be defined, measured and addressed, with diversity of views currently increasing rather than converging. With this focused contribution, we aim to provide a wide-angle, and as near as possible complete, snapshot of current work on reproducibility in NLP, delineating differences and similarities, and providing pointers to common denominators.

Original language	English
Title of host publication	Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics
Publisher	Association for Computational Linguistics
Pages	381-393
Number of pages	13
ISBN (Electronic)	9781954085022
Publication status	Published - Apr 2021
Event	16th Conference of the European Chapter of the Associationfor Computational Linguistics 2021 - Virtual, Online Duration: 19 Apr 2021 → 23 Apr 2021

Conference

Conference	16th Conference of the European Chapter of the Associationfor Computational Linguistics 2021
Abbreviated title	EACL 2021
City	Virtual, Online
Period	19/04/21 → 23/04/21

ASJC Scopus subject areas

Software
Computational Theory and Mathematics
Linguistics and Language

Access to Document

https://www.aclweb.org/anthology/2021.eacl-main.29.pdfLicence: CC BY

Cite this

@inproceedings{5f0c89b1893b4f87aae1d551e6de6c76,

title = "A systematic review of reproducibility research in natural language processing",

abstract = "Against the background of what has been termed a reproducibility crisis in science, the NLP field is becoming increasingly interested in, and conscientious about, the reproducibility of its results. The past few years have seen an impressive range of new initiatives, events and active research in the area. However, the field is far from reaching a consensus about how reproducibility should be defined, measured and addressed, with diversity of views currently increasing rather than converging. With this focused contribution, we aim to provide a wide-angle, and as near as possible complete, snapshot of current work on reproducibility in NLP, delineating differences and similarities, and providing pointers to common denominators.",

author = "Anya Belz and Shubham Agarwal and Anastasia Shimorina and Ehud Reiter",

note = "Funding Information: We thank our reviewers for their valuable feedback. Shubham Agarwal{\textquoteright}s PhD fees are supported by Adeptmind Inc., Toronto, Canada. Publisher Copyright: {\textcopyright} 2021 Association for Computational Linguistics; 16th Conference of the European Chapter of the Associationfor Computational Linguistics 2021, EACL 2021 ; Conference date: 19-04-2021 Through 23-04-2021",

year = "2021",

month = apr,

language = "English",

pages = "381--393",

booktitle = "Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics",

publisher = "Association for Computational Linguistics",

}

Belz, A, Agarwal, S, Shimorina, A & Reiter, E 2021, A systematic review of reproducibility research in natural language processing. in Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics. Association for Computational Linguistics, pp. 381-393, 16th Conference of the European Chapter of the Associationfor Computational Linguistics 2021, Virtual, Online, 19/04/21. <https://www.aclweb.org/anthology/2021.eacl-main.29.pdf>

A systematic review of reproducibility research in natural language processing. / Belz, Anya; Agarwal, Shubham; Shimorina, Anastasia et al.
Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics. Association for Computational Linguistics, 2021. p. 381-393.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - A systematic review of reproducibility research in natural language processing

AU - Belz, Anya

AU - Agarwal, Shubham

AU - Shimorina, Anastasia

AU - Reiter, Ehud

N1 - Funding Information: We thank our reviewers for their valuable feedback. Shubham Agarwal’s PhD fees are supported by Adeptmind Inc., Toronto, Canada. Publisher Copyright: © 2021 Association for Computational Linguistics

PY - 2021/4

Y1 - 2021/4

N2 - Against the background of what has been termed a reproducibility crisis in science, the NLP field is becoming increasingly interested in, and conscientious about, the reproducibility of its results. The past few years have seen an impressive range of new initiatives, events and active research in the area. However, the field is far from reaching a consensus about how reproducibility should be defined, measured and addressed, with diversity of views currently increasing rather than converging. With this focused contribution, we aim to provide a wide-angle, and as near as possible complete, snapshot of current work on reproducibility in NLP, delineating differences and similarities, and providing pointers to common denominators.

AB - Against the background of what has been termed a reproducibility crisis in science, the NLP field is becoming increasingly interested in, and conscientious about, the reproducibility of its results. The past few years have seen an impressive range of new initiatives, events and active research in the area. However, the field is far from reaching a consensus about how reproducibility should be defined, measured and addressed, with diversity of views currently increasing rather than converging. With this focused contribution, we aim to provide a wide-angle, and as near as possible complete, snapshot of current work on reproducibility in NLP, delineating differences and similarities, and providing pointers to common denominators.

UR - http://www.scopus.com/inward/record.url?scp=85107304722&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:85107304722

SP - 381

EP - 393

BT - Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics

PB - Association for Computational Linguistics

T2 - 16th Conference of the European Chapter of the Associationfor Computational Linguistics 2021

Y2 - 19 April 2021 through 23 April 2021

ER -

A systematic review of reproducibility research in natural language processing

Abstract

Conference

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this