On Reliability Assessment When a Software-based System Is Replaced by a Thought-to-be-Better One

Bev Littlewood, Kizito Salako, Lorenzo Strigini, Xingyu Zhao

Research output: Contribution to journalArticle

Abstract

The failure history of pre-existing systems can inform a reliability assessment of a new system. Such assessments – consisting of arguments based on evidence from older systems – are attractive and have been used for quite some time for, typically, mechanical/hardware-only systems. But their application to software-based systems brings some challenges. In this paper, we present a conservative, Bayesian approach to software reliability assessment – one that combines reliability evidence from an old system with an assessor’s confidence in a newer system being an improved replacement for the old one. We demonstrate, via different scenarios, what a thought-to-be-better replacement formally means in practice, and what it allows one to believe about actual reliability improvement. The results can be used directly in a reliability assessment, or to caution system stakeholders and industry regulators against using other models that give optimistic assessments. For instance, even if one is certain that some new software must be more reliable than an old product, using the reliability distribution for the old software as a prior distribution when assessing the new system gives optimistic, not conservative, predictions for the posterior reliability of the new system after seeing operational testing evidence.
Original languageEnglish
Article number106752
JournalReliability Engineering and System Safety
Volume197
Early online date12 Nov 2019
DOIs
Publication statusE-pub ahead of print - 12 Nov 2019

Fingerprint

Software reliability
Hardware
Testing
Industry

Cite this

@article{dd01216acb8f469f905a267f30d27001,
title = "On Reliability Assessment When a Software-based System Is Replaced by a Thought-to-be-Better One",
abstract = "The failure history of pre-existing systems can inform a reliability assessment of a new system. Such assessments – consisting of arguments based on evidence from older systems – are attractive and have been used for quite some time for, typically, mechanical/hardware-only systems. But their application to software-based systems brings some challenges. In this paper, we present a conservative, Bayesian approach to software reliability assessment – one that combines reliability evidence from an old system with an assessor’s confidence in a newer system being an improved replacement for the old one. We demonstrate, via different scenarios, what a thought-to-be-better replacement formally means in practice, and what it allows one to believe about actual reliability improvement. The results can be used directly in a reliability assessment, or to caution system stakeholders and industry regulators against using other models that give optimistic assessments. For instance, even if one is certain that some new software must be more reliable than an old product, using the reliability distribution for the old software as a prior distribution when assessing the new system gives optimistic, not conservative, predictions for the posterior reliability of the new system after seeing operational testing evidence.",
author = "Bev Littlewood and Kizito Salako and Lorenzo Strigini and Xingyu Zhao",
year = "2019",
month = "11",
day = "12",
doi = "10.1016/j.ress.2019.106752",
language = "English",
volume = "197",
journal = "Reliability Engineering and System Safety",
issn = "0951-8320",
publisher = "Elsevier Limited",

}

On Reliability Assessment When a Software-based System Is Replaced by a Thought-to-be-Better One. / Littlewood, Bev; Salako, Kizito; Strigini, Lorenzo; Zhao, Xingyu.

In: Reliability Engineering and System Safety, Vol. 197, 106752, 05.2020.

Research output: Contribution to journalArticle

TY - JOUR

T1 - On Reliability Assessment When a Software-based System Is Replaced by a Thought-to-be-Better One

AU - Littlewood, Bev

AU - Salako, Kizito

AU - Strigini, Lorenzo

AU - Zhao, Xingyu

PY - 2019/11/12

Y1 - 2019/11/12

N2 - The failure history of pre-existing systems can inform a reliability assessment of a new system. Such assessments – consisting of arguments based on evidence from older systems – are attractive and have been used for quite some time for, typically, mechanical/hardware-only systems. But their application to software-based systems brings some challenges. In this paper, we present a conservative, Bayesian approach to software reliability assessment – one that combines reliability evidence from an old system with an assessor’s confidence in a newer system being an improved replacement for the old one. We demonstrate, via different scenarios, what a thought-to-be-better replacement formally means in practice, and what it allows one to believe about actual reliability improvement. The results can be used directly in a reliability assessment, or to caution system stakeholders and industry regulators against using other models that give optimistic assessments. For instance, even if one is certain that some new software must be more reliable than an old product, using the reliability distribution for the old software as a prior distribution when assessing the new system gives optimistic, not conservative, predictions for the posterior reliability of the new system after seeing operational testing evidence.

AB - The failure history of pre-existing systems can inform a reliability assessment of a new system. Such assessments – consisting of arguments based on evidence from older systems – are attractive and have been used for quite some time for, typically, mechanical/hardware-only systems. But their application to software-based systems brings some challenges. In this paper, we present a conservative, Bayesian approach to software reliability assessment – one that combines reliability evidence from an old system with an assessor’s confidence in a newer system being an improved replacement for the old one. We demonstrate, via different scenarios, what a thought-to-be-better replacement formally means in practice, and what it allows one to believe about actual reliability improvement. The results can be used directly in a reliability assessment, or to caution system stakeholders and industry regulators against using other models that give optimistic assessments. For instance, even if one is certain that some new software must be more reliable than an old product, using the reliability distribution for the old software as a prior distribution when assessing the new system gives optimistic, not conservative, predictions for the posterior reliability of the new system after seeing operational testing evidence.

U2 - 10.1016/j.ress.2019.106752

DO - 10.1016/j.ress.2019.106752

M3 - Article

VL - 197

JO - Reliability Engineering and System Safety

JF - Reliability Engineering and System Safety

SN - 0951-8320

M1 - 106752

ER -