Testing the reliability and validity of risk assessment methods in Human Factors and Ergonomics

Adam Hulme, Neville A. Stanton, Guy H. Walker, Patrick Waterson, Paul M. Salmon

Research output: Contribution to journalArticlepeer-review

26 Citations (Scopus)
157 Downloads (Pure)

Abstract

There is growing interest in the use of systems-based risk assessment methods in Human Factors and Ergonomics (HFE). The purpose of this study was to test the intra-rater reliability and criterion-referenced concurrent validity of three systems-based risk assessment approaches: (i) the Systems-Theoretic Process Analysis (STPA) method; (ii) the Event Analysis of Systemic Teamwork Broken Links (EAST-BL) method; and, (iii) the Network Hazard Analysis and Risk Management System (Net-HARMS) method. Reliability and validity measures were obtained using the Signal Detection Theory (SDT) paradigm. Whilst STPA identified the highest number of risks, the findings indicate a weak to moderate level of reliability and validity for STPA, EAST-BL and Net-HARMS. There were no statistically significant differences between the methods across analyses. The results suggest that there is merit to the continued use of systems-based risk assessment methods following a series of methodological extensions that aim to enhance the reliability and validity of future applications. Practitioner summary The three risk assessment methods produced weak to moderate levels of stability and accuracy regarding their capability to predict risks. There is a pressing need to further test the reliability and validity of safety methods in Human Factors and Ergonomics.

Original languageEnglish
Pages (from-to)407-428
Number of pages22
JournalErgonomics
Volume65
Issue number3
Early online date12 Aug 2021
DOIs
Publication statusPublished - 4 Mar 2022

Keywords

  • EAST-BL
  • Net-HARMS
  • Risk assessment
  • STPA
  • reliability validity

ASJC Scopus subject areas

  • Human Factors and Ergonomics
  • Physical Therapy, Sports Therapy and Rehabilitation

Fingerprint

Dive into the research topics of 'Testing the reliability and validity of risk assessment methods in Human Factors and Ergonomics'. Together they form a unique fingerprint.

Cite this