Testing the Reliability and Validity of Net-HARMS: A New Systems-Based Risk Assessment Method in HFE

Adam Hulme*, Neville A. Stanton, Guy H. Walker, Patrick Waterson, Paul M. Salmon

*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceedingConference contribution

2 Citations (Scopus)


There is growing interest in the use of systems-based risk assessment (RA) methods in human factors and ergonomics (HFE). Despite this, there has been a lack of formal reliability and validity testing undertaken to determine whether systems-based RA methods have the capacity to reliably and accurately identify potential risks within complex systems. The purpose of this study was to test the intra-rater reliability (within subject stability) and criterion-related validity (‘gold standard’ performance) of the Networked Hazard Analysis and Risk Management System (Net-HARMS). Net-HARMS is a new and innovative systems-based RA method that supports analysts with the identification of emergent risks. Emergent risks represent new risks that are created when risks from across a complex system interact with one another. Reliability and validity measures for Net-HARMS were obtained using the Signal Detection Theory (SDT) paradigm. Matthews Correlation Coefficient (MCC) was used to analyze the complete SDT data to measure the strength of the correlation between risks. Findings indicate a weak to moderate level of reliability and validity for Net-HARMS based on the MCC score. The results suggest that there is merit to the continued use of Net-HARMS following a series of methodological recommendations that aim to enhance the reliability and validity of future applications.

Original languageEnglish
Title of host publicationProceedings of the 21st Congress of the International Ergonomics Association (IEA 2021)
EditorsNancy L. Black, W. Patrick Neumann, Ian Noy
Number of pages9
ISBN (Electronic)9783030746025
ISBN (Print)9783030746018
Publication statusPublished - 1 May 2021
Event21st Congress of the International Ergonomics Association 2021 - Virtual, Online
Duration: 13 Jun 202118 Jun 2021

Publication series

NameLecture Notes in Networks and Systems
ISSN (Print)2367-3370
ISSN (Electronic)2367-3389


Conference21st Congress of the International Ergonomics Association 2021
Abbreviated titleIEA 2021
CityVirtual, Online


  • Net-HARMS
  • Reliability and validity
  • Risk assessment
  • Signal detection theory

ASJC Scopus subject areas

  • Control and Systems Engineering
  • Signal Processing
  • Computer Networks and Communications


Dive into the research topics of 'Testing the Reliability and Validity of Net-HARMS: A New Systems-Based Risk Assessment Method in HFE'. Together they form a unique fingerprint.

Cite this