Information extraction from operator interface images using computer vision and machine learning

Abstract

In the process of system upgrades or migrations, the utilization of existing layouts and object structures for designing new Human Machine Interfaces (HMI) can significantly save time and effort. Operator interface images, commonly referred to as HMI´s, contain valuable information crucial to industrial operations, but access to source code or design files can be limited. Modern frameworks for object detection and text recognition offer a solution by extracting information directly from images. However, these methods require time-consuming data acquisition and manual effort to initiate. This paper proposes a novel approach utilizing traditional Computer Vision (CV) and Machine Learning (ML) techniques to extract objects from images. The extracted objects are used as training data to transfer learn a ResNet model for multi-label image classification. The combination of this model with techniques such as sliding window, pyramid scaling, and non-maximum suppression forms the basis for a semi-automated annotation tool. This tool generates training data for more optimized object detection methods, specifically the YOLO (You Only Look Once) one-stage object detector. The semi-automated annotation tool allows engineers to manually refine the training data and export state-of-the-art training images for YOLO. The YOLO model achieves an impressive mean Average Precision at IoU 50% (mAP50) score of 95.5% when transfer learned on the annotated data. Additionally, an Optical Character Recognition (OCR) engine is utilized to extract text information from preprocessed images, followed by postprocessing to filter tag data. An algorithm is then employed to link objects and tags together. The final solution is implemented in software designed to optimize user interaction, resulting in an analysis document in Excel format, which can be easily exported for end-user access. With the novel use of this software to automate image analysis, the time required to analyze HMI images prior to migration or rebuild can be reduced by an estimate of 90%.