Comparing Human-Only, AI-Assisted, and AI-Led Teams on Assessing Research Reproducibility in Quantitative Social Science

Abel Brodeur; David Valenta; Alexandru Marcoci; Juan Pablo Aparicio; Derek Mikola; Bruno Barbarioli; Rohan Alexander; Lachlan Deer; Tom Stafford; Lars Vilhuber; Gunther Bensch; Mohamed Abdelhady; Yousra Abdelmoula; Ghina Abdul Baki; Tomás Aguirre; Sriraj Aiyer; Shumi Akhtar; Farida Akhtar; Melle R. Albada; Micah Altman; David Angenendt; Zahra Arjmandi Lari; Jorge Armando De León Tejada; Igor Asanov; Anastasiya-Mariya Asanov Noha; Rebecca Ashong; Tobias Auer; Francisco J. Bahamonde-Birke; Bradley J. Baker; Söhnke M. Bartram; Dongqi Bao; Lucija Batinovic; Tommaso Batistoni; Monica Beeder; Louis-Philippe Beland; Carsten Bienz; Christ Billy Aryanto; Cylcia Bolibaugh; Carl Bonander; Ramiro Bravo; Katherine Brennan; Egor Bronnikov; Stephan Bruns; Nino Buliskeria; Sara Caicedo-Silva; Andrea Calef; Solomon Caulker; Simonas Cepenas; Arthur Chatton; Zirou Chen; Ngozi Chioma Ewurum; Anda-Bianca Ciocîrlan; Felix J. Clouth; Jason Collins; Nikolai Cook; Cesar Cornejo; João Craveiro; Jing Cui; Niveditha Chalil Vayalabron; Christian Czymara; Carlos Daniel Bermúdez Jaramillo; Hannes Datta; Lien Denoo; Arshia Dhaliwal; Nency Dhameja; Elodie Djemai; Erwan Dujeancourt; Uğurcan Dündar; Thibaut Duprey; Yasmine Eissa; Youssef El Fassi; Ismail El Fassi; Keaton Ellis; Ali Elminejad; Mahmoud Elsherif; Aysil Emirmahmutoglu; Giulian Etingin-Frati; Emeka Eze; Jan Fabian Dollbaum; Jan Feld Victoria; Andres Felipe Rengifo Jaramillo; Guidon Fenig; Victoria Fernandes; Lenka Fiala; Lukas Fink; Sara Fish; Jack Fitzgerald; Rachel Joy Forshaw; Alexandre Fortier-Chouinard

doi:10.2139/ssrn.5118632

Comparing Human-Only, AI-Assisted, and AI-Led Teams on Assessing Research Reproducibility in Quantitative Social Science

Abel Brodeur
, David Valenta
, Alexandru Marcoci
, Juan Pablo Aparicio
, Derek Mikola
, Bruno Barbarioli
, Rohan Alexander
, Lachlan Deer
, Tom Stafford
, Lars Vilhuber
, Gunther Bensch
, Mohamed Abdelhady
, Yousra Abdelmoula
, Ghina Abdul Baki
, Tomás Aguirre
, Sriraj Aiyer
, Shumi Akhtar
, Farida Akhtar
, Melle R. Albada
, Micah Altman

David Angenendt, Zahra Arjmandi Lari, Jorge Armando De León Tejada, Igor Asanov, Anastasiya-Mariya Asanov Noha, Rebecca Ashong, Tobias Auer, Francisco J. Bahamonde-Birke, Bradley J. Baker, Söhnke M. Bartram, Dongqi Bao, Lucija Batinovic, Tommaso Batistoni, Monica Beeder, Louis-Philippe Beland, Carsten Bienz, Christ Billy Aryanto, Cylcia Bolibaugh, Carl Bonander, Ramiro Bravo, Katherine Brennan, Egor Bronnikov, Stephan Bruns, Nino Buliskeria, Sara Caicedo-Silva, Andrea Calef, Solomon Caulker, Simonas Cepenas, Arthur Chatton, Zirou Chen, Ngozi Chioma Ewurum, Anda-Bianca Ciocîrlan, Felix J. Clouth, Jason Collins, Nikolai Cook, Cesar Cornejo, João Craveiro, Jing Cui, Niveditha Chalil Vayalabron, Christian Czymara, Carlos Daniel Bermúdez Jaramillo, Hannes Datta, Lien Denoo, Arshia Dhaliwal, Nency Dhameja, Elodie Djemai, Erwan Dujeancourt, Uğurcan Dündar, Thibaut Duprey, Yasmine Eissa, Youssef El Fassi, Ismail El Fassi, Keaton Ellis, Ali Elminejad, Mahmoud Elsherif, Aysil Emirmahmutoglu, Giulian Etingin-Frati, Emeka Eze, Jan Fabian Dollbaum, Jan Feld Victoria, Andres Felipe Rengifo Jaramillo, Guidon Fenig, Victoria Fernandes, Lenka Fiala, Lukas Fink, Sara Fish, Jack Fitzgerald, Rachel Joy Forshaw, Alexandre Fortier-Chouinard

Research output: Working paper › Discussion paper

Abstract

This study evaluates the effectiveness of varying levels of human and artificial intelligence (AI) integration in reproducibility assessments. We computationally reproduced quantitative results from published articles in the social sciences with 288 researchers, randomly assigned to 103 teams across three groups — human-only teams, AI-assisted teams and teams whose task was to minimally guide an AI to conduct reproducibility checks (the "AI-led" approach). Findings reveal that when working independently, human teams matched the reproducibility success rates of teams using AI assistance, while both groups substantially outperformed AI-led approaches (with human teams achieving 57 pp higher success rates than AI-led teams). Human teams found significantly more major errors compared to both AI-assisted teams and AI-led teams. AI-assisted teams demonstrated an advantage over more automated approaches, detecting 0.4 more major errors per team than AI-led teams, though still significantly fewer than human-only teams. Finally, both human and AI-assisted teams significantly outperformed AI-led approaches in both proposing and implementing comprehensive robustness checks.

Original language	English
Publisher	IZA Institute of Labor Economics
DOIs	https://doi.org/10.2139/ssrn.5118632
Publication status	Published - 31 Jan 2025

Access to Document

10.2139/ssrn.5118632

Cite this

Brodeur, A., Valenta, D., Marcoci, A., Aparicio, J. P., Mikola, D., Barbarioli, B., Alexander, R., Deer, L., Stafford, T., Vilhuber, L., Bensch, G., Abdelhady, M., Abdelmoula, Y., Baki, G. A., Aguirre, T., Aiyer, S., Akhtar, S., Akhtar, F., Albada, M. R., ... Fortier-Chouinard, A. (2025). Comparing Human-Only, AI-Assisted, and AI-Led Teams on Assessing Research Reproducibility in Quantitative Social Science. IZA Institute of Labor Economics. https://doi.org/10.2139/ssrn.5118632

@techreport{bc32951c1e5f43bda6895a3386e8ab4a,

title = "Comparing Human-Only, AI-Assisted, and AI-Led Teams on Assessing Research Reproducibility in Quantitative Social Science",

abstract = "This study evaluates the effectiveness of varying levels of human and artificial intelligence (AI) integration in reproducibility assessments. We computationally reproduced quantitative results from published articles in the social sciences with 288 researchers, randomly assigned to 103 teams across three groups — human-only teams, AI-assisted teams and teams whose task was to minimally guide an AI to conduct reproducibility checks (the {"}AI-led{"} approach). Findings reveal that when working independently, human teams matched the reproducibility success rates of teams using AI assistance, while both groups substantially outperformed AI-led approaches (with human teams achieving 57 pp higher success rates than AI-led teams). Human teams found significantly more major errors compared to both AI-assisted teams and AI-led teams. AI-assisted teams demonstrated an advantage over more automated approaches, detecting 0.4 more major errors per team than AI-led teams, though still significantly fewer than human-only teams. Finally, both human and AI-assisted teams significantly outperformed AI-led approaches in both proposing and implementing comprehensive robustness checks.",

author = "Abel Brodeur and David Valenta and Alexandru Marcoci and Aparicio, \{Juan Pablo\} and Derek Mikola and Bruno Barbarioli and Rohan Alexander and Lachlan Deer and Tom Stafford and Lars Vilhuber and Gunther Bensch and Mohamed Abdelhady and Yousra Abdelmoula and Baki, \{Ghina Abdul\} and Tom{\'a}s Aguirre and Sriraj Aiyer and Shumi Akhtar and Farida Akhtar and Albada, \{Melle R.\} and Micah Altman and David Angenendt and Lari, \{Zahra Arjmandi\} and \{Armando De Le{\'o}n Tejada\}, Jorge and Igor Asanov and \{Asanov Noha\}, Anastasiya-Mariya and Rebecca Ashong and Tobias Auer and Bahamonde-Birke, \{Francisco J.\} and Baker, \{Bradley J.\} and Bartram, \{S{\"o}hnke M.\} and Dongqi Bao and Lucija Batinovic and Tommaso Batistoni and Monica Beeder and Louis-Philippe Beland and Carsten Bienz and Aryanto, \{Christ Billy\} and Cylcia Bolibaugh and Carl Bonander and Ramiro Bravo and Katherine Brennan and Egor Bronnikov and Stephan Bruns and Nino Buliskeria and Sara Caicedo-Silva and Andrea Calef and Solomon Caulker and Simonas Cepenas and Arthur Chatton and Zirou Chen and Ewurum, \{Ngozi Chioma\} and Anda-Bianca Cioc{\^i}rlan and Clouth, \{Felix J.\} and Jason Collins and Nikolai Cook and Cesar Cornejo and Jo{\~a}o Craveiro and Jing Cui and Vayalabron, \{Niveditha Chalil\} and Christian Czymara and \{Berm{\'u}dez Jaramillo\}, \{Carlos Daniel\} and Hannes Datta and Lien Denoo and Arshia Dhaliwal and Nency Dhameja and Elodie Djemai and Erwan Dujeancourt and Uğurcan D{\"u}ndar and Thibaut Duprey and Yasmine Eissa and \{El Fassi\}, Youssef and \{El Fassi\}, Ismail and Keaton Ellis and Ali Elminejad and Mahmoud Elsherif and Aysil Emirmahmutoglu and Giulian Etingin-Frati and Emeka Eze and Dollbaum, \{Jan Fabian\} and Victoria, \{Jan Feld\} and \{Rengifo Jaramillo\}, \{Andres Felipe\} and Guidon Fenig and Victoria Fernandes and Lenka Fiala and Lukas Fink and Sara Fish and Jack Fitzgerald and Forshaw, \{Rachel Joy\} and Alexandre Fortier-Chouinard",

year = "2025",

month = jan,

day = "31",

doi = "10.2139/ssrn.5118632",

language = "English",

publisher = "IZA Institute of Labor Economics",

type = "WorkingPaper",

institution = "IZA Institute of Labor Economics",

}

Brodeur, A, Valenta, D, Marcoci, A, Aparicio, JP, Mikola, D, Barbarioli, B, Alexander, R, Deer, L, Stafford, T, Vilhuber, L, Bensch, G, Abdelhady, M, Abdelmoula, Y, Baki, GA, Aguirre, T, Aiyer, S, Akhtar, S, Akhtar, F, Albada, MR, Altman, M, Angenendt, D, Lari, ZA, Armando De León Tejada, J, Asanov, I, Asanov Noha, A-M, Ashong, R, Auer, T, Bahamonde-Birke, FJ, Baker, BJ, Bartram, SM, Bao, D, Batinovic, L, Batistoni, T, Beeder, M, Beland, L-P, Bienz, C, Aryanto, CB, Bolibaugh, C, Bonander, C, Bravo, R, Brennan, K, Bronnikov, E, Bruns, S, Buliskeria, N, Caicedo-Silva, S, Calef, A, Caulker, S, Cepenas, S, Chatton, A, Chen, Z, Ewurum, NC, Ciocîrlan, A-B, Clouth, FJ, Collins, J, Cook, N, Cornejo, C, Craveiro, J, Cui, J, Vayalabron, NC, Czymara, C, Bermúdez Jaramillo, CD, Datta, H, Denoo, L, Dhaliwal, A, Dhameja, N, Djemai, E, Dujeancourt, E, Dündar, U, Duprey, T, Eissa, Y, El Fassi, Y, El Fassi, I, Ellis, K, Elminejad, A, Elsherif, M, Emirmahmutoglu, A, Etingin-Frati, G, Eze, E, Dollbaum, JF, Victoria, JF, Rengifo Jaramillo, AF, Fenig, G, Fernandes, V, Fiala, L, Fink, L, Fish, S, Fitzgerald, J, Forshaw, RJ & Fortier-Chouinard, A 2025 'Comparing Human-Only, AI-Assisted, and AI-Led Teams on Assessing Research Reproducibility in Quantitative Social Science' IZA Institute of Labor Economics. https://doi.org/10.2139/ssrn.5118632

TY - UNPB

T1 - Comparing Human-Only, AI-Assisted, and AI-Led Teams on Assessing Research Reproducibility in Quantitative Social Science

AU - Brodeur, Abel

AU - Valenta, David

AU - Marcoci, Alexandru

AU - Aparicio, Juan Pablo

AU - Mikola, Derek

AU - Barbarioli, Bruno

AU - Alexander, Rohan

AU - Deer, Lachlan

AU - Stafford, Tom

AU - Vilhuber, Lars

AU - Bensch, Gunther

AU - Abdelhady, Mohamed

AU - Abdelmoula, Yousra

AU - Baki, Ghina Abdul

AU - Aguirre, Tomás

AU - Aiyer, Sriraj

AU - Akhtar, Shumi

AU - Akhtar, Farida

AU - Albada, Melle R.

AU - Altman, Micah

AU - Angenendt, David

AU - Lari, Zahra Arjmandi

AU - Armando De León Tejada, Jorge

AU - Asanov, Igor

AU - Asanov Noha, Anastasiya-Mariya

AU - Ashong, Rebecca

AU - Auer, Tobias

AU - Bahamonde-Birke, Francisco J.

AU - Baker, Bradley J.

AU - Bartram, Söhnke M.

AU - Bao, Dongqi

AU - Batinovic, Lucija

AU - Batistoni, Tommaso

AU - Beeder, Monica

AU - Beland, Louis-Philippe

AU - Bienz, Carsten

AU - Aryanto, Christ Billy

AU - Bolibaugh, Cylcia

AU - Bonander, Carl

AU - Bravo, Ramiro

AU - Brennan, Katherine

AU - Bronnikov, Egor

AU - Bruns, Stephan

AU - Buliskeria, Nino

AU - Caicedo-Silva, Sara

AU - Calef, Andrea

AU - Caulker, Solomon

AU - Cepenas, Simonas

AU - Chatton, Arthur

AU - Chen, Zirou

AU - Ewurum, Ngozi Chioma

AU - Ciocîrlan, Anda-Bianca

AU - Clouth, Felix J.

AU - Collins, Jason

AU - Cook, Nikolai

AU - Cornejo, Cesar

AU - Craveiro, João

AU - Cui, Jing

AU - Vayalabron, Niveditha Chalil

AU - Czymara, Christian

AU - Bermúdez Jaramillo, Carlos Daniel

AU - Datta, Hannes

AU - Denoo, Lien

AU - Dhaliwal, Arshia

AU - Dhameja, Nency

AU - Djemai, Elodie

AU - Dujeancourt, Erwan

AU - Dündar, Uğurcan

AU - Duprey, Thibaut

AU - Eissa, Yasmine

AU - El Fassi, Youssef

AU - El Fassi, Ismail

AU - Ellis, Keaton

AU - Elminejad, Ali

AU - Elsherif, Mahmoud

AU - Emirmahmutoglu, Aysil

AU - Etingin-Frati, Giulian

AU - Eze, Emeka

AU - Dollbaum, Jan Fabian

AU - Victoria, Jan Feld

AU - Rengifo Jaramillo, Andres Felipe

AU - Fenig, Guidon

AU - Fernandes, Victoria

AU - Fiala, Lenka

AU - Fink, Lukas

AU - Fish, Sara

AU - Fitzgerald, Jack

AU - Forshaw, Rachel Joy

AU - Fortier-Chouinard, Alexandre

PY - 2025/1/31

Y1 - 2025/1/31

N2 - This study evaluates the effectiveness of varying levels of human and artificial intelligence (AI) integration in reproducibility assessments. We computationally reproduced quantitative results from published articles in the social sciences with 288 researchers, randomly assigned to 103 teams across three groups — human-only teams, AI-assisted teams and teams whose task was to minimally guide an AI to conduct reproducibility checks (the "AI-led" approach). Findings reveal that when working independently, human teams matched the reproducibility success rates of teams using AI assistance, while both groups substantially outperformed AI-led approaches (with human teams achieving 57 pp higher success rates than AI-led teams). Human teams found significantly more major errors compared to both AI-assisted teams and AI-led teams. AI-assisted teams demonstrated an advantage over more automated approaches, detecting 0.4 more major errors per team than AI-led teams, though still significantly fewer than human-only teams. Finally, both human and AI-assisted teams significantly outperformed AI-led approaches in both proposing and implementing comprehensive robustness checks.

AB - This study evaluates the effectiveness of varying levels of human and artificial intelligence (AI) integration in reproducibility assessments. We computationally reproduced quantitative results from published articles in the social sciences with 288 researchers, randomly assigned to 103 teams across three groups — human-only teams, AI-assisted teams and teams whose task was to minimally guide an AI to conduct reproducibility checks (the "AI-led" approach). Findings reveal that when working independently, human teams matched the reproducibility success rates of teams using AI assistance, while both groups substantially outperformed AI-led approaches (with human teams achieving 57 pp higher success rates than AI-led teams). Human teams found significantly more major errors compared to both AI-assisted teams and AI-led teams. AI-assisted teams demonstrated an advantage over more automated approaches, detecting 0.4 more major errors per team than AI-led teams, though still significantly fewer than human-only teams. Finally, both human and AI-assisted teams significantly outperformed AI-led approaches in both proposing and implementing comprehensive robustness checks.

U2 - 10.2139/ssrn.5118632

DO - 10.2139/ssrn.5118632

M3 - Discussion paper

BT - Comparing Human-Only, AI-Assisted, and AI-Led Teams on Assessing Research Reproducibility in Quantitative Social Science

PB - IZA Institute of Labor Economics

ER -

Comparing Human-Only, AI-Assisted, and AI-Led Teams on Assessing Research Reproducibility in Quantitative Social Science

Abstract

Access to Document

Fingerprint

Cite this