Validation of Dynamic Simulation Models using Metamorphic Testing and Given-When-Then Patterns

Abstract

As the maritime industry evolves, there is a focus onsimulation-driven design, testing, and validation usingnovel technology solutions. Simulation models designed torepresent the behaviour and features of real systems areincreasingly available for testing during the early phaseof the full development, but in many cases, their testingsuffers from the availability of test oracles. Metamorphictesting has become increasingly used in differentapplication domains as an approach to test systems when anexplicit test oracle is unavailable. In order to increaseits adoption by domain experts, we combine metamorphictesting with Behaviour Driven development for theverification and validation of simulation models. Thetool-based approach facilitates automated test generationbased on domain-specific custom metamorphic transformationsto generate meaningful test inputs for metamorphic inputrelations. The method also uses features and scenariosextracted from system requirements and domain expertise todefine metamorphic output relations. By automating testgeneration based on system behaviours as features,scenarios, metamorphic transformations, and outputcandidates for metamorphic relations in a Gherkin-likeformat, the tool enables practitioners to verify modelsbased on domain-specific constraints and metamorphicrelation checks. Our preliminary evaluation shows that thetool can detect MR violations in the simulation modelsunder test and that automated test generation providesimproved coverage.