A metric of knowledge as information compression reflects reproducibility predictions in biomedical experiments

Forecasting the reproducibility of research findings is one of the key challenges of Metascience. To date, reliable above-chance predictions have mainly been achieved by pooling the subjective ratings of experts through surveys or via prediction markets. Obtaining such data is laborious, however, and unlikely to increase the theoretical understanding of reproducibility.

Here we show that empirical measures of K, a principled metric of knowledge, are correlated with reproducibility forecasts made for the Brazilian Reproducibility Initiative (BRI), an ongoing replication of 60 studies that use three common methodologies in the life sciences. For each study, we calculated the K value by dividing the effect size, measured in bits of Shannon entropy explained, by the descriptive complexity of the study’s methodology, calculated as the optimal Shannon encoding of a conceptual graph representing the replication protocol.

K values were statistically associated with subjective predictions about studies’ replication probabilities and relative effect sizes. This relation was robust to controlling for study methodology, and was a stronger predictor than other plausible covariates, including the originally reported effect size, subjective ratings of a study’s methodological difficulty, and raters’ self-reported expertise. The superior fit of K may be due to its particular structure and metrics, which are rooted in information theory and may more accurately reflect the implicit calculations made by raters.

This is the first evidence that an objective metric hypothesized to quantify scientific knowledge captures subjective judgments about reproducibility. This finding gives the first independent support of K’s underlying assumption that scientific knowledge is a process of information compression, and it may offer a new tool for metaresearch.