Modelling and Simulation of Detection Rates of Emergent Behaviours in System Integration Test Regimes
Keywords:Bayes’ theorem, emergent behaviour, experimental design, statistical inference, system integration testing
AbstractSystem level testing generally lacks coverage due to cost of performing realistic tests on the “system as a whole”. This lack in test coverage gives rise to seemingly emergent aviour at system level. The interactions between multiple sub-systems lead to “the whole being greater than the sum of its parts”, which is a famous saying dated back to the time of the Greek philosopher Aristotle. Either we should test more extensively at system level, or we should test smarter. The company needs to validate its current test regime to see if the current way of testing detects the emergent behaviours in question. We seek to validate the company’s system integration test regime to see if it can detect a given set of emergent behaviours. This paper aims to find the probabilities of detecting specified types of emergent behaviour in the way the company performs system integration testing today and compare that to alternative test regimes. A model is set-up to find the probabilities of the emergent behaviour types in the different test regimes, and to simulate the corresponding detection rates and related uncertainties. The results show that the company could benefit from changing to an alternative test regime, which has higher probability of detecting a given set of unwanted behaviours emerging through system integration testing.
R. Allenby and A. Slomson. How to count: An introduction to combinatorics. CRC Press. 2010.
C. Cenedese and A. L. Gordon. Ocean Current. Encyclopedia Britannica. 2021. https://www.britannica.com/science/ocean-current.
K. Dunn. Process Improvements Using Data. 2021. https://learnche.org/pid.
R. A. Haugen and M. Mansouri. Applying Systems Thinking to Frame and Explore a Test System for Product Verification; a Case Study in Large Defence Projects. In Proceedings – 30th INCOSE International Symposium, INCOSE IS 2020, 20-22 July, 2020, virtual, pages 78-93, 2020. doi:10.1002/j.2334-5837.2020.00709.x
K. A. Kjeldaas, R. A. Haugen, and E. Syverud. Challenges in Detecting Emergent Behavior in System Testing. In Proceedings – 31st INCOSE International Symposium, INCOSE IS 2021, 17-22 July, 2021, virtual, pager 1211-1228, 2021. doi:10.1002/j.2334-5837.2021.00896.x
B. Lambert. A Student’s Guide to Bayesian Statistics, 1st ed. SAGE Publications Ltd. 2018.
S. Mittal, S. Diallo, and A. Tolk. Emergent Behavior in Complex Systems Engineering: A Modelling and Simulation Approach. Wiley. 2018.
D. C. Montgomery. Design and Analysis of Experiments, 8th ed. Wiley. 2017.
D. S. Sivia and J. Skilling. Data Analysis: A Bayesian tutorial, 2nd ed. Oxford Science Publications. 2006
Copyright (c) 2022 Rune Andre Haugen, Ali Ghaderi
This work is licensed under a Creative Commons Attribution 4.0 International License.