REFORMS: Consensus-based Recommendations for Machine-learning-based Science

Sayash Kapoor*, Emily Cantrell, Kenny Peng, Than Hien Pham, Christopher A. Bail, Odd Erik Gundersen, Jake Hofman, Jessica Hullman, Michael Adam Lones, Momin M. Malik, Priyanka Nanayakkara, Russell A. Poldrack, Inioluwa Deborah Raji, Michael Roberts, Matthew J. Salganik, Marta Serra-Garcia, Brandon M. Stewart, Gilles Vandewiele, Arvind Narayanan

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

166 Downloads (Pure)


Machine learning (ML) methods are proliferating in scientific research. However, the adoption of these methods has been accompanied by failures of validity, reproducibility, and generalizability. These failures can hinder scientific progress, lead to false consensus around invalid claims, and undermine the credibility of ML-based science. ML methods are often applied and fail in similar ways across disciplines. Motivated by this observation, our goal is to provide clear recommendations for conducting and reporting ML-based science. Drawing from an extensive review of past literature, we present the REFORMS checklist (recommendations for machine-learning-based science). It consists of 32 questions and a paired set of guidelines. REFORMS was developed on the basis of a consensus of 19 researchers across computer science, data science, mathematics, social sciences, and biomedical sciences. REFORMS can serve as a resource for researchers when designing and implementing a study, for referees when reviewing papers, and for journals when enforcing standards for transparency and reproducibility.
Original languageEnglish
Article numbereadk3452
JournalScience Advances
Issue number18
Early online date1 May 2024
Publication statusPublished - 3 May 2024


  • Consensus
  • Humans
  • Machine Learning
  • Reproducibility of Results
  • Science


Dive into the research topics of 'REFORMS: Consensus-based Recommendations for Machine-learning-based Science'. Together they form a unique fingerprint.

Cite this