Evaluating and improving the reliability of evidence syntheses in conservation and environmental science: A methodology

Paul Woodcock, Andrew S. Pullin*, Michel J. Kaiser

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

40 Citations (Scopus)


The volume of primary literature in conservation and environmental science is expanding rapidly. Evidence syntheses that review and combine the findings from research on policy-relevant questions are therefore vital for informing decision-making. However, such syntheses exhibit considerable variation in conduct and reporting, potentially undermining their value to decision-makers. To address this problem, we developed a scoring system - the Collaboration for Environmental Evidence Synthesis Assessment Tool (CEESAT) - that uses detailed criteria and guidelines to evaluate policy-relevant syntheses in conservation and environmental science. The higher the score awarded, the greater the objectivity, comprehensiveness and transparency of the synthesis, hence the greater the confidence in its reliability. We then used 40 review articles to test CEESAT in terms of (i) applicability to different syntheses, (ii) validity of scores awarded, (iii) effectiveness at discriminating between syntheses, and (iv) repeatability of scoring by different assessors. CEESAT was applicable to 36 articles, and scores ranged from 1 to 33 (mean=13.2, median=15, maximum possible=39). Variation in overall scores and in the individual criteria shows that CEESAT discriminates effectively among syntheses, making differences in rigour clear. Scoring was repeatable, indicating that assessments are not overly susceptible to differences in the application and interpretation of guidelines. The detailed rationale and guidelines for each criterion should help improve future syntheses and promote consistent scoring between assessors. Furthermore, scores can be used directly by non-specialists to compare syntheses that investigate key conservation questions and to incorporate reliability and rigour into decision-making.

Original languageEnglish
Pages (from-to)54-62
Number of pages9
JournalBiological Conservation
Publication statusPublished - Aug 2014


  • Environmental policy
  • Evidence base
  • Evidence synthesis
  • Meta-analysis
  • Review methodology
  • Systematic review

ASJC Scopus subject areas

  • Ecology, Evolution, Behavior and Systematics
  • Nature and Landscape Conservation


Dive into the research topics of 'Evaluating and improving the reliability of evidence syntheses in conservation and environmental science: A methodology'. Together they form a unique fingerprint.

Cite this