A Safety Framework for Critical Systems Utilising Deep Neural Networks

Xingyu Zhao, Alec Banks, James Sharp, Valentin Robu, David Flynn, Michael Fisher, Xiaowei Huang

Research output: Chapter in Book/Report/Conference proceedingConference contribution

23 Citations (Scopus)
65 Downloads (Pure)


Increasingly sophisticated mathematical modelling processes from Machine Learning are being used to analyse complex data. However, the performance and explainability of these models within practical critical systems requires a rigorous and continuous verification of their safe utilisation. Working towards addressing this challenge, this paper presents a principled novel safety argument framework for critical systems that utilise deep neural networks. The approach allows various forms of predictions, e.g., future reliability of passing some demands, or confidence on a required reliability level. It is supported by a Bayesian analysis using operational data and the recent verification and validation techniques for deep learning. The prediction is conservative - it starts with partial prior knowledge obtained from lifecycle activities and then determines the worst-case prediction. Open challenges are also identified.
Original languageEnglish
Title of host publicationComputer Safety, Reliability, and Security. SAFECOMP 2020
EditorsAntónio Casimiro, Pedro Ferreira, Frank Ortmeier, Friedemann Bitsch
Number of pages16
ISBN (Electronic)9783030545499
ISBN (Print)9783030545482
Publication statusPublished - 2020
Event39th International Conference on Computer Safety, Reliability and Security 2020 - Lisbon, Portugal
Duration: 16 Sept 202018 Sept 2020

Publication series

NameLecture Notes in Computer Science
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349


Conference39th International Conference on Computer Safety, Reliability and Security 2020
Abbreviated titleSAFECOMP 2020


  • Assurance arguments
  • Bayesian inference
  • Deep learning verification
  • Quantitative claims
  • Reliability claims
  • Safe AI
  • Safety cases

ASJC Scopus subject areas

  • Theoretical Computer Science
  • Computer Science(all)


Dive into the research topics of 'A Safety Framework for Critical Systems Utilising Deep Neural Networks'. Together they form a unique fingerprint.

Cite this