Evaluating and improving the reliability of evidence syntheses in conservation and environmental science: A methodology

The volume of primary literature in conservation and environmental science is expanding rapidly. Evidence syntheses that review and combine the findings from research on policy-relevant questions are therefore vital for informing decision-making. However, such syntheses exhibit considerable variation in conduct and reporting, potentially undermining their value to decision-makers. To address this problem, we developed a scoring system - the Collaboration for Environmental Evidence Synthesis Assessment Tool (CEESAT) - that uses detailed criteria and guidelines to evaluate policy-relevant syntheses in conservation and environmental science. The higher the score awarded, the greater the objectivity, comprehensiveness and transparency of the synthesis, hence the greater the confidence in its reliability. We then used 40 review articles to test CEESAT in terms of (i) applicability to different syntheses, (ii) validity of scores awarded, (iii) effectiveness at discriminating between syntheses, and (iv) repeatability of scoring by different assessors. CEESAT was applicable to 36 articles, and scores ranged from 1 to 33 (mean=13.2, median=15, maximum possible=39). Variation in overall scores and in the individual criteria shows that CEESAT discriminates effectively among syntheses, making differences in rigour clear. Scoring was repeatable, indicating that assessments are not overly susceptible to differences in the application and interpretation of guidelines. The detailed rationale and guidelines for each criterion should help improve future syntheses and promote consistent scoring between assessors. Furthermore, scores can be used directly by non-specialists to compare syntheses that investigate key conservation questions and to incorporate reliability and rigour into decision-making.