Assessing the Safety and Reliability of Autonomous Vehicles from Road Testing

Xingyu Zhao; Valentin Robu; David Flynn; Kizito Salako; Lorenzo Strigini

Assessing the Safety and Reliability of Autonomous Vehicles from Road Testing

Xingyu Zhao, Valentin Robu, David Flynn, Kizito Salako, Lorenzo Strigini

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

There is an urgent societal need to assess whether autonomous vehicles (AVs) are safe enough. From published quantitative safety and reliability assessments of AVs, we know that, given the goal of predicting very low rates of accidents,
road testing alone requires infeasible numbers of miles to be driven. However, previous analyses do not consider any knowledge prior to road testing – knowledge which could bring substantial advantages if the AV design allows strong expectations of safety before road testing. We present the advantages of a new variant of Conservative Bayesian Inference (CBI), which uses prior knowledge while avoiding optimistic biases. We then study the trend of disengagements (take-overs by human drivers) by applying Software Reliability Growth Models (SRGMs) to data from Waymo’s public road testing over 51 months, in view of the practice of software updates during this testing. Our approach is to not trust any specific SRGM, but to assess forecast accuracy
and then improve forecasts. We show that, coupled with accuracy
assessment and recalibration techniques, SRGMs could be a valuable test planning aid.

Original language	English
Title of host publication	The 30th International Symposium on Software Reliability Engineering
Publication status	Accepted/In press - 20 Jul 2019
Event	30th International Symposium on Software Reliability Engineering 2019 - Berlin, Germany Duration: 28 Oct 2019 → 31 Oct 2019

Conference

Conference	30th International Symposium on Software Reliability Engineering 2019
Abbreviated title	ISSRE 2019
Country	Germany
City	Berlin
Period	28/10/19 → 31/10/19

Keywords

Autonomous vehicles
software reliability
Bayesian inference
Conservative Bayesian Inference
reliability claims
Statistical analysis
Autonomous systems

Access to Document

open access link to publication.

Embargoed Document

Download pdf
Accepted author manuscript, 501 KB
Embargo ends: 5/08/29

Fingerprint Dive into the research topics of 'Assessing the Safety and Reliability of Autonomous Vehicles from Road Testing'. Together they form a unique fingerprint.

Cite this

Zhao, X., Robu, V., Flynn, D., Salako, K., & Strigini, L. (Accepted/In press). Assessing the Safety and Reliability of Autonomous Vehicles from Road Testing. In The 30th International Symposium on Software Reliability Engineering

@inproceedings{be9a428678df4e4196a715dbd2153426,

title = "Assessing the Safety and Reliability of Autonomous Vehicles from Road Testing",

abstract = "There is an urgent societal need to assess whether autonomous vehicles (AVs) are safe enough. From published quantitative safety and reliability assessments of AVs, we know that, given the goal of predicting very low rates of accidents,road testing alone requires infeasible numbers of miles to be driven. However, previous analyses do not consider any knowledge prior to road testing – knowledge which could bring substantial advantages if the AV design allows strong expectations of safety before road testing. We present the advantages of a new variant of Conservative Bayesian Inference (CBI), which uses prior knowledge while avoiding optimistic biases. We then study the trend of disengagements (take-overs by human drivers) by applying Software Reliability Growth Models (SRGMs) to data from Waymo{\textquoteright}s public road testing over 51 months, in view of the practice of software updates during this testing. Our approach is to not trust any specific SRGM, but to assess forecast accuracyand then improve forecasts. We show that, coupled with accuracyassessment and recalibration techniques, SRGMs could be a valuable test planning aid.",

keywords = "Autonomous vehicles, software reliability, Bayesian inference, Conservative Bayesian Inference, reliability claims, Statistical analysis, Autonomous systems",

author = "Xingyu Zhao and Valentin Robu and David Flynn and Kizito Salako and Lorenzo Strigini",

year = "2019",

month = jul,

day = "20",

language = "English",

booktitle = "The 30th International Symposium on Software Reliability Engineering",

note = "30th International Symposium on Software Reliability Engineering 2019, ISSRE 2019 ; Conference date: 28-10-2019 Through 31-10-2019",

}

TY - GEN

T1 - Assessing the Safety and Reliability of Autonomous Vehicles from Road Testing

AU - Zhao, Xingyu

AU - Robu, Valentin

AU - Flynn, David

AU - Salako, Kizito

AU - Strigini, Lorenzo

PY - 2019/7/20

Y1 - 2019/7/20

N2 - There is an urgent societal need to assess whether autonomous vehicles (AVs) are safe enough. From published quantitative safety and reliability assessments of AVs, we know that, given the goal of predicting very low rates of accidents,road testing alone requires infeasible numbers of miles to be driven. However, previous analyses do not consider any knowledge prior to road testing – knowledge which could bring substantial advantages if the AV design allows strong expectations of safety before road testing. We present the advantages of a new variant of Conservative Bayesian Inference (CBI), which uses prior knowledge while avoiding optimistic biases. We then study the trend of disengagements (take-overs by human drivers) by applying Software Reliability Growth Models (SRGMs) to data from Waymo’s public road testing over 51 months, in view of the practice of software updates during this testing. Our approach is to not trust any specific SRGM, but to assess forecast accuracyand then improve forecasts. We show that, coupled with accuracyassessment and recalibration techniques, SRGMs could be a valuable test planning aid.

AB - There is an urgent societal need to assess whether autonomous vehicles (AVs) are safe enough. From published quantitative safety and reliability assessments of AVs, we know that, given the goal of predicting very low rates of accidents,road testing alone requires infeasible numbers of miles to be driven. However, previous analyses do not consider any knowledge prior to road testing – knowledge which could bring substantial advantages if the AV design allows strong expectations of safety before road testing. We present the advantages of a new variant of Conservative Bayesian Inference (CBI), which uses prior knowledge while avoiding optimistic biases. We then study the trend of disengagements (take-overs by human drivers) by applying Software Reliability Growth Models (SRGMs) to data from Waymo’s public road testing over 51 months, in view of the practice of software updates during this testing. Our approach is to not trust any specific SRGM, but to assess forecast accuracyand then improve forecasts. We show that, coupled with accuracyassessment and recalibration techniques, SRGMs could be a valuable test planning aid.

KW - Autonomous vehicles

KW - software reliability

KW - Bayesian inference

KW - Conservative Bayesian Inference

KW - reliability claims

KW - Statistical analysis

KW - Autonomous systems

UR - https://openaccess.city.ac.uk/id/eprint/22872/

M3 - Conference contribution

BT - The 30th International Symposium on Software Reliability Engineering

T2 - 30th International Symposium on Software Reliability Engineering 2019

Y2 - 28 October 2019 through 31 October 2019

ER -