Crowdsourcing hypothesis tests: Making transparent how design choices shape research results

Justin F. Landy; Miaolei Liam; Isabel L. Ding; Domenico Viganola; Warren Tierney; Anna Dreber; Magnus Johannesson; Thomas Pfeiffer; Charles R. Ebersole; Quentin F. Gronau; Alexander Ly; Don Van Den Bergh; Maarten Marsman; Koen Derks; Eric Jan Wagenmakers; Andrew Proctor; Daniel M. Bartels; Christopher W. Bauman; William J. Brady; Felix Cheung; Andrei Cimpian; Simone Dohle; M. Brent Donnellan; Adam Hahn; Michael P. Hall; William Jiménez-Leal; David J. Johnson; Richard E. Lucas; BenoÎt Monin; Andres Montealegre; Elizabeth Mullen; Jun Pang; Jennifer Ray; Diego A. Reinero; Jesse Reynolds; Daniel Shafik Storage; Runkun Su; Christina M. Tworek; Jay J. Van Bavel; Daniel Walco; Julian Wills; Xiaobing Xu; Kai Chi Yam; Xiaoyu Yang; William A. Cunningham; Martin Schweinsberg; Molly Urwitz; Crowdsourcing Hypothesis Tests Collaboration; Eric Luis Uhlmann; Jais Adam-Troian

doi:10.1037/bul0000220

Crowdsourcing hypothesis tests: Making transparent how design choices shape research results

Justin F. Landy^*, Miaolei Liam, Isabel L. Ding, Domenico Viganola, Warren Tierney, Anna Dreber, Magnus Johannesson, Thomas Pfeiffer, Charles R. Ebersole, Quentin F. Gronau, Alexander Ly, Don Van Den Bergh, Maarten Marsman, Koen Derks, Eric Jan Wagenmakers, Andrew Proctor, Daniel M. Bartels, Christopher W. Bauman, William J. Brady, Felix CheungAndrei Cimpian, Simone Dohle, M. Brent Donnellan, Adam Hahn, Michael P. Hall, William Jiménez-Leal, David J. Johnson, Richard E. Lucas, BenoÎt Monin, Andres Montealegre, Elizabeth Mullen, Jun Pang, Jennifer Ray, Diego A. Reinero, Jesse Reynolds, Daniel Shafik Storage, Runkun Su, Christina M. Tworek, Jay J. Van Bavel, Daniel Walco, Julian Wills, Xiaobing Xu, Kai Chi Yam, Xiaoyu Yang, William A. Cunningham, Martin Schweinsberg, Molly Urwitz, Crowdsourcing Hypothesis Tests Collaboration, Eric Luis Uhlmann, Jais Adam-Troian

^*Corresponding author for this work

School of Social Sciences

Research output: Contribution to journal › Article › peer-review

99 Citations (Scopus)

Abstract

To what extent are research results influenced by subjective decisions that scientists make as they design studies? Fifteen research teams independently designed studies to answer five original research questions related to moral judgments, negotiations, and implicit cognition. Participants from 2 separate large samples (total N = 15,000) were then randomly assigned to complete 1 version of each study. Effect sizes varied dramatically across different sets of materials designed to test the same hypothesis: Materials from different teams rendered statistically significant effects in opposite directions for 4 of 5 hypotheses, with the narrowest range in estimates being d = —0.37 to + 0.26. Meta-analysis and a Bayesian perspective on the results revealed overall support for 2 hypotheses and a lack of support for 3 hypotheses. Overall, practically none of the variability in effect sizes was attributable to the skill of the research team in designing materials, whereas considerable variability was attributable to the hypothesis being tested. In a forecasting survey, predictions of other scientists were significantly correlated with study results, both across and within hypotheses. Crowdsourced testing of research hypotheses helps reveal the true consistency of empirical support for a scientific claim.

Original language	English
Pages (from-to)	451-479
Number of pages	29
Journal	Psychological Bulletin
Volume	146
Issue number	5
DOIs	https://doi.org/10.1037/bul0000220
Publication status	Published - 16 Jan 2020

Keywords

conceptual replications
crowdsourcing
forecasting
research robustness
scientific transparency

ASJC Scopus subject areas

General Psychology

Access to Document

10.1037/bul0000220

Cite this

Landy, J. F., Liam, M., Ding, I. L., Viganola, D., Tierney, W., Dreber, A., Johannesson, M., Pfeiffer, T., Ebersole, C. R., Gronau, Q. F., Ly, A., Bergh, D. V. D., Marsman, M., Derks, K., Wagenmakers, E. J., Proctor, A., Bartels, D. M., Bauman, C. W., Brady, W. J., ... Adam-Troian, J. (2020). Crowdsourcing hypothesis tests: Making transparent how design choices shape research results. Psychological Bulletin, 146(5), 451-479. https://doi.org/10.1037/bul0000220

@article{5861f04ce8784b31b5a78db13eb52ff8,

title = "Crowdsourcing hypothesis tests: Making transparent how design choices shape research results",

abstract = "To what extent are research results influenced by subjective decisions that scientists make as they design studies? Fifteen research teams independently designed studies to answer five original research questions related to moral judgments, negotiations, and implicit cognition. Participants from 2 separate large samples (total N = 15,000) were then randomly assigned to complete 1 version of each study. Effect sizes varied dramatically across different sets of materials designed to test the same hypothesis: Materials from different teams rendered statistically significant effects in opposite directions for 4 of 5 hypotheses, with the narrowest range in estimates being d = —0.37 to + 0.26. Meta-analysis and a Bayesian perspective on the results revealed overall support for 2 hypotheses and a lack of support for 3 hypotheses. Overall, practically none of the variability in effect sizes was attributable to the skill of the research team in designing materials, whereas considerable variability was attributable to the hypothesis being tested. In a forecasting survey, predictions of other scientists were significantly correlated with study results, both across and within hypotheses. Crowdsourced testing of research hypotheses helps reveal the true consistency of empirical support for a scientific claim.",

keywords = "conceptual replications, crowdsourcing, forecasting, research robustness, scientific transparency",

author = "Landy, {Justin F.} and Miaolei Liam and Ding, {Isabel L.} and Domenico Viganola and Warren Tierney and Anna Dreber and Magnus Johannesson and Thomas Pfeiffer and Ebersole, {Charles R.} and Gronau, {Quentin F.} and Alexander Ly and Bergh, {Don Van Den} and Maarten Marsman and Koen Derks and Wagenmakers, {Eric Jan} and Andrew Proctor and Bartels, {Daniel M.} and Bauman, {Christopher W.} and Brady, {William J.} and Felix Cheung and Andrei Cimpian and Simone Dohle and Donnellan, {M. Brent} and Adam Hahn and Hall, {Michael P.} and William Jim{\'e}nez-Leal and Johnson, {David J.} and Lucas, {Richard E.} and Beno{\^I}t Monin and Andres Montealegre and Elizabeth Mullen and Jun Pang and Jennifer Ray and Reinero, {Diego A.} and Jesse Reynolds and Walter Sowden and Storage, {Daniel Shafik} and Runkun Su and Tworek, {Christina M.} and {Van Bavel}, {Jay J.} and Daniel Walco and Julian Wills and Xiaobing Xu and Yam, {Kai Chi} and Xiaoyu Yang and Cunningham, {William A.} and Martin Schweinsberg and Molly Urwitz and {Crowdsourcing Hypothesis Tests Collaboration} and Uhlmann, {Eric Luis} and Jais Adam-Troian",

note = "Publisher Copyright: {\textcopyright} 2020, American Psychological Association.",

year = "2020",

month = jan,

day = "16",

doi = "10.1037/bul0000220",

language = "English",

volume = "146",

pages = "451--479",

journal = "Psychological Bulletin",

issn = "0033-2909",

publisher = "American Psychological Association",

number = "5",

}

Landy, JF, Liam, M, Ding, IL, Viganola, D, Tierney, W, Dreber, A, Johannesson, M, Pfeiffer, T, Ebersole, CR, Gronau, QF, Ly, A, Bergh, DVD, Marsman, M, Derks, K, Wagenmakers, EJ, Proctor, A, Bartels, DM, Bauman, CW, Brady, WJ, Cheung, F, Cimpian, A, Dohle, S, Donnellan, MB, Hahn, A, Hall, MP, Jiménez-Leal, W, Johnson, DJ, Lucas, RE, Monin, B, Montealegre, A, Mullen, E, Pang, J, Ray, J, Reinero, DA, Reynolds, J, Storage, DS, Su, R, Tworek, CM, Van Bavel, JJ, Walco, D, Wills, J, Xu, X, Yam, KC, Yang, X, Cunningham, WA, Schweinsberg, M, Urwitz, M, Crowdsourcing Hypothesis Tests Collaboration, Uhlmann, EL & Adam-Troian, J 2020, 'Crowdsourcing hypothesis tests: Making transparent how design choices shape research results', Psychological Bulletin, vol. 146, no. 5, pp. 451-479. https://doi.org/10.1037/bul0000220

TY - JOUR

T1 - Crowdsourcing hypothesis tests: Making transparent how design choices shape research results

AU - Landy, Justin F.

AU - Liam, Miaolei

AU - Ding, Isabel L.

AU - Viganola, Domenico

AU - Tierney, Warren

AU - Dreber, Anna

AU - Johannesson, Magnus

AU - Pfeiffer, Thomas

AU - Ebersole, Charles R.

AU - Gronau, Quentin F.

AU - Ly, Alexander

AU - Bergh, Don Van Den

AU - Marsman, Maarten

AU - Derks, Koen

AU - Wagenmakers, Eric Jan

AU - Proctor, Andrew

AU - Bartels, Daniel M.

AU - Bauman, Christopher W.

AU - Brady, William J.

AU - Cheung, Felix

AU - Cimpian, Andrei

AU - Dohle, Simone

AU - Donnellan, M. Brent

AU - Hahn, Adam

AU - Hall, Michael P.

AU - Jiménez-Leal, William

AU - Johnson, David J.

AU - Lucas, Richard E.

AU - Monin, BenoÎt

AU - Montealegre, Andres

AU - Mullen, Elizabeth

AU - Pang, Jun

AU - Ray, Jennifer

AU - Reinero, Diego A.

AU - Reynolds, Jesse

AU - Sowden, Walter

AU - Storage, Daniel Shafik

AU - Su, Runkun

AU - Tworek, Christina M.

AU - Van Bavel, Jay J.

AU - Walco, Daniel

AU - Wills, Julian

AU - Xu, Xiaobing

AU - Yam, Kai Chi

AU - Yang, Xiaoyu

AU - Cunningham, William A.

AU - Schweinsberg, Martin

AU - Urwitz, Molly

AU - Crowdsourcing Hypothesis Tests Collaboration

AU - Uhlmann, Eric Luis

AU - Adam-Troian, Jais

PY - 2020/1/16

Y1 - 2020/1/16

N2 - To what extent are research results influenced by subjective decisions that scientists make as they design studies? Fifteen research teams independently designed studies to answer five original research questions related to moral judgments, negotiations, and implicit cognition. Participants from 2 separate large samples (total N = 15,000) were then randomly assigned to complete 1 version of each study. Effect sizes varied dramatically across different sets of materials designed to test the same hypothesis: Materials from different teams rendered statistically significant effects in opposite directions for 4 of 5 hypotheses, with the narrowest range in estimates being d = —0.37 to + 0.26. Meta-analysis and a Bayesian perspective on the results revealed overall support for 2 hypotheses and a lack of support for 3 hypotheses. Overall, practically none of the variability in effect sizes was attributable to the skill of the research team in designing materials, whereas considerable variability was attributable to the hypothesis being tested. In a forecasting survey, predictions of other scientists were significantly correlated with study results, both across and within hypotheses. Crowdsourced testing of research hypotheses helps reveal the true consistency of empirical support for a scientific claim.

AB - To what extent are research results influenced by subjective decisions that scientists make as they design studies? Fifteen research teams independently designed studies to answer five original research questions related to moral judgments, negotiations, and implicit cognition. Participants from 2 separate large samples (total N = 15,000) were then randomly assigned to complete 1 version of each study. Effect sizes varied dramatically across different sets of materials designed to test the same hypothesis: Materials from different teams rendered statistically significant effects in opposite directions for 4 of 5 hypotheses, with the narrowest range in estimates being d = —0.37 to + 0.26. Meta-analysis and a Bayesian perspective on the results revealed overall support for 2 hypotheses and a lack of support for 3 hypotheses. Overall, practically none of the variability in effect sizes was attributable to the skill of the research team in designing materials, whereas considerable variability was attributable to the hypothesis being tested. In a forecasting survey, predictions of other scientists were significantly correlated with study results, both across and within hypotheses. Crowdsourced testing of research hypotheses helps reveal the true consistency of empirical support for a scientific claim.

KW - conceptual replications

KW - crowdsourcing

KW - forecasting

KW - research robustness

KW - scientific transparency

UR - http://www.scopus.com/inward/record.url?scp=85081412411&partnerID=8YFLogxK

U2 - 10.1037/bul0000220

DO - 10.1037/bul0000220

M3 - Article

C2 - 31944796

AN - SCOPUS:85081412411

SN - 0033-2909

VL - 146

SP - 451

EP - 479

JO - Psychological Bulletin

JF - Psychological Bulletin

IS - 5

ER -

Crowdsourcing hypothesis tests: Making transparent how design choices shape research results

Abstract

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this