A metric of knowledge as information compression reflects reproducibility predictions for biomedical experiments

Forecasting the reproducibility of research findings is one of the key challenges of metascience. Above-chance predictions have mainly been achieved by pooling the subjective ratings of experts, and how these predictions are formed remains to be understood. Here, we show that reproducibility forecasts made for the Brazilian Reproducibility Initiative (BRI), a large-scale replication of experiments in the life sciences, are significantly correlated with K, a principled metric of knowledge as information compression. For each study in the BRI sample, we calculated K by dividing the effect size, measured in bits of Shannon entropy, by the descriptive length (a proxy of the complexity) of the study’s methodology, calculated as the optimal Shannon encoding of a conceptual graph representing the replication protocol. We found that experts’ predictions about reproducibility were statistically associated with K values and with the complexity of protocols. This relation was robust to controlling for study methodology and other possible confounding factors. These results suggest that expert raters partially judge the reproducibility of findings by assessing the ratio between the information yielded and the information required by a study, and they support the hypothesis that scientific knowledge may be understood and studied through the lenses of information compression.