A Safety Framework for Critical Systems Utilising Deep Neural Networks

Xingyu Zhao, Alec Banks, James Sharp, Valentin Robu, David Flynn, Michael Fisher, Xiaowei Huang

Research output: Chapter in Book/Report/Conference proceedingConference contribution

1 Citation (Scopus)
3 Downloads (Pure)

Abstract

Increasingly sophisticated mathematical modelling processes from Machine Learning are being used to analyse complex data. However, the performance and explainability of these models within practical critical systems requires a rigorous and continuous verification of their safe utilisation. Working towards addressing this challenge, this paper presents a principled novel safety argument framework for critical systems that utilise deep neural networks. The approach allows various forms of predictions, e.g., future reliability of passing some demands, or confidence on a required reliability level. It is supported by a Bayesian analysis using operational data and the recent verification and validation techniques for deep learning. The prediction is conservative - it starts with partial prior knowledge obtained from lifecycle activities and then determines the worst-case prediction. Open challenges are also identified.
Original languageEnglish
Title of host publicationComputer Safety, Reliability, and Security. SAFECOMP 2020
EditorsAntónio Casimiro, Pedro Ferreira, Frank Ortmeier, Friedemann Bitsch
PublisherSpringer
Pages244-259
Number of pages16
ISBN (Electronic)9783030545499
ISBN (Print)9783030545482
DOIs
Publication statusPublished - 2020
Event39th International Conference on Computer Safety, Reliability and Security 2020 - Lisbon, Portugal
Duration: 16 Sep 202018 Sep 2020

Publication series

NameLecture Notes in Computer Science
Volume12234
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

Conference39th International Conference on Computer Safety, Reliability and Security 2020
Abbreviated titleSAFECOMP 2020
CountryPortugal
CityLisbon
Period16/09/2018/09/20

Keywords

  • Assurance arguments
  • Bayesian inference
  • Deep learning verification
  • Quantitative claims
  • Reliability claims
  • Safe AI
  • Safety cases

ASJC Scopus subject areas

  • Theoretical Computer Science
  • Computer Science(all)

Fingerprint Dive into the research topics of 'A Safety Framework for Critical Systems Utilising Deep Neural Networks'. Together they form a unique fingerprint.

  • Cite this

    Zhao, X., Banks, A., Sharp, J., Robu, V., Flynn, D., Fisher, M., & Huang, X. (2020). A Safety Framework for Critical Systems Utilising Deep Neural Networks. In A. Casimiro, P. Ferreira, F. Ortmeier, & F. Bitsch (Eds.), Computer Safety, Reliability, and Security. SAFECOMP 2020 (pp. 244-259). (Lecture Notes in Computer Science; Vol. 12234). Springer. https://doi.org/10.1007/978-3-030-54549-9_16