Improving System Safety in Aviation: Supporting STPA with AI Models

Abstract

Background: System safety in aeronautics is critical, as it directly affects aircraft reliability, efficiency, safety, and security. Given the complexity of modern aviation systems and the potential consequences of failures, a structured and proactive safety approach is essential. System-Theoretic Process Analysis (STPA) is a modern hazard analysis method designed to identify and mitigate risks. Unlike traditional methods that focus primarily on component failures, STPA accounts for both failures and unsafe interactions among system elements, including human operators, software, and organizational factors. Problem: Despite its effectiveness, STPA poses challenges in practical application. The process is time-consuming and requires extensive expertise in system safety, control theory, and system dynamics. Analysts must heavily rely on expert judgment to define losses, hazards, safety constraints, and unsafe control actions. Additionally, training in STPA is resource-intensive, making automation an appealing solution to streamline the process. Goal: To address these challenges, we developed two AI-driven pipelines to automate the initial steps of STPA, reducing reliance on expert knowledge and enhancing efficiency. Method: The first pipeline leverages a fine-tuned Llama3.1-8B model to extract losses, hazards, and constraints from ConOps documents. The second pipeline, BERT Error Detection for STPA (BEDS), improves accuracy by classifying, verifying, detecting errors, and suggesting potential corrections for the extracted elements. Results: The first pipeline was trained using 134 ConOps documents paired with corresponding STPA safety analysis elements. The dataset comprised 35 authentic documents from the CORDIS repository and 99 AI-generated examples. The model achieved a mean precision of 79.73%, recall of 81.09%, and an F1-score of 80.22%. For the second pipeline, 1,084 sentences were extracted from values identified during the first step of STPA. Three classifiers were developed: the sentence identifier achieved a mean accuracy of 95.20%, the incorrect sentence detector 88.61%, and the sentence error identifier 83.44%. While the pipelines were designed to work together, they can also be used independently. Conclusion: This study tackles the challenges of applying STPA in aeronautics by introducing two automated pipelines to streamline the initial process steps. The first pipeline, powered by a fine-tuned Llama3.1-8B model, extracts losses, hazards, and constraints from ConOps documents. The second pipeline, BEDS, verifies and corrects these elements with high accuracy. The results demonstrate strong precision and recall scores, highlighting the potential to reduce both the time and expertise required for STPA analysis in complex aviation systems.