CN112801072B - A computer vision-based elevator non-leveling door opening fault identification device and method - Google Patents
A computer vision-based elevator non-leveling door opening fault identification device and methodDownload PDFInfo
- Publication number
- CN112801072B CN112801072BCN202110400660.9ACN202110400660ACN112801072BCN 112801072 BCN112801072 BCN 112801072BCN 202110400660 ACN202110400660 ACN 202110400660ACN 112801072 BCN112801072 BCN 112801072B
- Authority
- CN
- China
- Prior art keywords
- elevator
- acceleration
- sill
- image
- door opening
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/40—Scenes; Scene-specific elements in video content
- G06V20/41—Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items
- G—PHYSICS
- G01—MEASURING; TESTING
- G01D—MEASURING NOT SPECIALLY ADAPTED FOR A SPECIFIC VARIABLE; ARRANGEMENTS FOR MEASURING TWO OR MORE VARIABLES NOT COVERED IN A SINGLE OTHER SUBCLASS; TARIFF METERING APPARATUS; MEASURING OR TESTING NOT OTHERWISE PROVIDED FOR
- G01D21/00—Measuring or testing not otherwise provided for
- G01D21/02—Measuring two or more variables by means not covered by a single other subclass
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions
- G06F17/10—Complex mathematical operations
- G06F17/18—Complex mathematical operations for evaluating statistical data, e.g. average values, frequency distributions, probability functions, regression analysis
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
- G06F18/2413—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on distances to training or reference patterns
- G06F18/24133—Distances to prototypes
- G06F18/24137—Distances to cluster centroïds
- G06F18/2414—Smoothing the distance, e.g. radial basis function networks [RBFN]
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/10—Image acquisition
- G06V10/12—Details of acquisition arrangements; Constructional details thereof
- G06V10/14—Optical characteristics of the device performing the acquisition or on the illumination arrangements
- G06V10/141—Control of illumination
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/34—Smoothing or thinning of the pattern; Morphological operations; Skeletonisation
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Life Sciences & Earth Sciences (AREA)
- General Engineering & Computer Science (AREA)
- Evolutionary Computation (AREA)
- Mathematical Physics (AREA)
- Artificial Intelligence (AREA)
- Software Systems (AREA)
- Evolutionary Biology (AREA)
- Bioinformatics & Computational Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Computational Linguistics (AREA)
- Multimedia (AREA)
- Health & Medical Sciences (AREA)
- Computational Mathematics (AREA)
- Biophysics (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Biomedical Technology (AREA)
- Mathematical Analysis (AREA)
- Mathematical Optimization (AREA)
- Pure & Applied Mathematics (AREA)
- Probability & Statistics with Applications (AREA)
- Operations Research (AREA)
- Algebra (AREA)
- Databases & Information Systems (AREA)
- Indicating And Signalling Devices For Elevators (AREA)
Abstract
Translated fromChinese
Description
Translated fromChinese技术领域technical field
本发明涉及电梯故障识别领域，特别是涉及一种基于计算机视觉的电梯非平层开门故障识别装置及方法。The invention relates to the field of elevator fault identification, in particular to a computer vision-based elevator non-leveling door opening fault identification device and method.
背景技术Background technique
近年来，我国电梯数量快速增加，与之相对应的电梯事故也越来越多，其中，电梯门系统故障是引发电梯事故的主要原因。电梯门系统故障有多种，主要包括：非平层开门(非平层指的是轿门地坎相对于厅门地坎的高差绝对值大于10mm，非平层开门指的是电梯在非平层的位置打开轿门和厅门)、非同步开门、开关门失效、防止门夹人装置失效等类型。非平层开门故障是造成人员伤亡的最主要原因之一，也是现代电梯安全保障技术最难彻底解决的问题之一。因此对非平层开门进行检测和识别具有重要的现实意义。In recent years, the number of elevators in my country has increased rapidly, and the corresponding elevator accidents have also increased. Among them, the failure of elevator door system is the main cause of elevator accidents. There are many kinds of elevator door system faults, mainly including: non-leveling door opening (non-leveling refers to the absolute value of the height difference between the car door sill and the hall door sill is greater than 10mm, non-leveling door opening refers to the elevator in non-leveling Leveling position to open the car door and hall door), non-synchronized door opening, door opening and closing failure, failure of door jamming device failure, etc. Non-leveling door opening failure is one of the main causes of casualties, and it is also one of the most difficult problems for modern elevator safety technology to solve completely. Therefore, it is of great practical significance to detect and identify non-leveling door openings.
传统的非平层开门故障检测方法主要依赖于人工维护排查，这种方式不仅效率很低，而且发现故障隐患周期过久，影响电梯安全运行。The traditional non-leveling door opening fault detection method mainly relies on manual maintenance and investigation, which is not only inefficient, but also has a long period of finding hidden faults, which affects the safe operation of the elevator.
近几年，随着计算机技术的发展，一些研究尝试通过计算机视觉技术来实现识别电梯开关门。In recent years, with the development of computer technology, some researches try to realize the recognition of elevator door opening and closing through computer vision technology.
如：发明专利：电梯平层故障的检测方法和系统（CN201710666494），提出一种电梯平层故障的检测方法和系统，在电梯轿门开启时，获取电梯轿内地坎与电梯所在楼层的厅门地坎所在区域的目标图像，计算目标图像中电梯轿内地坎和厅门地坎之间的视角差值，在视角差值大于或小于预设值时，判定电梯平层故障。For example: Invention patent: Detection method and system for elevator leveling fault (CN201710666494), which proposes a detection method and system for elevator leveling fault. When the elevator car door is opened, the elevator car interior sill and the hall door of the floor where the elevator is located are obtained. The target image of the area where the sill is located, calculate the difference between the viewing angle between the elevator car sill and the hall door sill in the target image, and determine the elevator leveling failure when the viewing angle difference is greater than or less than the preset value.
该专利使用的图像识别电梯是否平层的方法存在以下问题：The method used in the patent to identify whether the elevator is leveling has the following problems:
第一，目标图像难以获取，将摄像头安装在轿厢内上部，地坎或门扇很容易被轿厢出入口乘客遮挡；First, the target image is difficult to obtain. The camera is installed in the upper part of the car, and the sill or door leaf is easily blocked by passengers at the entrance and exit of the car;
第二，图像检测方法难以实现，采用层面地坎与厅门地坎视交差值的方法，开门时如果厅门地坎位于较低位置时，即被轿厢地坎遮挡，算法失效；Second, the image detection method is difficult to implement. The method of the apparent intersection value between the floor sill and the hall door sill is used. If the hall door sill is at a lower position when opening the door, it will be blocked by the car sill, and the algorithm will fail;
第三，图像识别效率和准确率低，电梯门和地坎种类规格繁多，目前采用的图像识别技术较为简单，无法快速、准确地识别目标图像；Third, the image recognition efficiency and accuracy are low, and there are many types and specifications of elevator doors and sills. The currently used image recognition technology is relatively simple, and cannot quickly and accurately identify the target image;
第四，故障识别过程发生在整个电梯的运行过程中，带来了不必要的算力消耗。Fourth, the fault identification process occurs during the entire elevator operation, which brings unnecessary consumption of computing power.
发明内容SUMMARY OF THE INVENTION
本发明的目的在于克服现有技术的不足，提供一种基于计算机视觉的电梯非平层开门故障识别装置和方法，能够实时在线监测电梯非平层开门故障，减少非平层目标样本的人工标定工作量，实现地坎槽模型和非平层模型的快速训练，提高对电梯非平层图像和开关门图像的目标检测准确率，具有在低照度和高照度环境下的图像降噪能力，提高整体电梯故障智能检测和识别能力。The purpose of the present invention is to overcome the deficiencies of the prior art, and to provide a computer vision-based elevator non-leveling door-opening fault identification device and method, which can monitor the elevator non-leveling door-opening faults online in real time and reduce the manual calibration of non-leveling target samples workload, realize the rapid training of the sill model and the non-leveling model, improve the target detection accuracy of the elevator non-leveling image and door opening and closing image, and have the ability to reduce image noise in low-light and high-light environments. The overall elevator fault intelligent detection and identification ability.
本发明实施例提供了一种基于计算机视觉的电梯非平层开门故障识别装置，其包括视频采集模块、加速度监测模块、模型训练模块以及检测识别模块；其中，The embodiment of the present invention provides a computer vision-based elevator non-leveling door opening fault identification device, which includes a video acquisition module, an acceleration monitoring module, a model training module, and a detection and identification module; wherein,
所述视频采集模块，包括摄像装置，用于采集电梯地坎图像；所述视频采集模块与检测识别模块电性连接；所述摄像装置的镜头设置于电梯轿门和厅门夹缝，垂直并正对于地坎槽，以同时拍摄厅门地坎槽和轿门地坎槽；The video capture module includes a camera device for capturing images of the elevator sill; the video capture module is electrically connected to the detection and identification module; the camera lens of the video capture device is arranged in the gap between the elevator car door and the hall door, and is vertical and positive. For the sill groove, the hall door sill groove and the sedan door sill groove can be photographed at the same time;
所述加速度监测模块，包括加速度传感器以及微处理器；所述加速度监测模块用于对电梯运行加速度进行监控，以预判电梯是否进入减速停梯状态，加速度监测模块与检测识别模块电性连接；The acceleration monitoring module includes an acceleration sensor and a microprocessor; the acceleration monitoring module is used to monitor the running acceleration of the elevator to predict whether the elevator enters a deceleration stop state, and the acceleration monitoring module is electrically connected to the detection and identification module;
所述模型训练模块，包括地坎槽目标检测模型训练单元和非平层开门分类模型训练单元；所述地坎槽目标检测模型训练单元用于预先训练地坎槽检测模型，供检测识别模块加载使用；所述非平层开门分类模型训练单元用于预先训练非平层开门分类模型，以供检测识别模块加载使用；The model training module includes a sill target detection model training unit and a non-level door classification model training unit; the sill target detection model training unit is used to pre-train the sill detection model for loading by the detection and identification module use; the non-leveling door classification model training unit is used for pre-training the non-leveling door classification model for the detection and identification module loading;
所述检测识别模块，用于在根据所述加速度监测模块确定所述电梯进入减速停梯状态时，对电梯地坎图像做自适应处理并加载由所述模型训练模块训练好的模型进行预测以及识别，以判断电梯是否发生非平层开门故障。The detection and recognition module is used to perform adaptive processing on the elevator sill image and load the model trained by the model training module to predict and predict when the elevator enters the deceleration stop state according to the acceleration monitoring module. Identification to determine whether the elevator has a non-leveling door open fault.
优选地，所述微处理器具体用于，利用方差、极差以及极差对应的索引距离占加速度序列长度的占比对完成滤波平滑处理后的加速度曲线进行加速度检测以判断电梯是否停梯；Preferably, the microprocessor is specifically used to perform acceleration detection on the acceleration curve after the filtering and smoothing process is completed by using the ratio of the variance, the range, and the index distance corresponding to the range to the length of the acceleration sequence to determine whether the elevator stops;
其中，所述方差是用来度量加速度序列的波动程度，表达的是加速度的平稳性，用
其中，
所述极差是用来衡量数值变动的最大范围，是测定数值变动较为方便的指标，是最大值与最小值之间的差距，用
其中，
所述极差对应的索引距离占加速度序列长度的占比的是刻画加速度骤变程度，描述的是最大值和最小值所在序列位置的间距占整个运行加速度序列的比例，用r表示，如公式（3）所示：The ratio of the index distance corresponding to the range to the length of the acceleration sequence is to describe the degree of sudden acceleration of acceleration, which describes the ratio of the distance between the sequence positions of the maximum value and the minimum value to the entire running acceleration sequence, which is represented by r, as shown in the formula (3) shows:
其中，
其中，当方差满足大于1000、极差满足大于500和/或极差对应的索引距离占加速度序列长度的占比小于10%，则认为电梯即将停梯；否则微处理器判断为加速度正常，电梯处于正常运行。Among them, when the variance is greater than 1000, the range is greater than 500, and/or the index distance corresponding to the range accounts for less than 10% of the length of the acceleration sequence, it is considered that the elevator is about to stop; otherwise, the microprocessor determines that the acceleration is normal, and the elevator in normal operation.
优选地，所述加速度传感器具有多个采集时间间隔；通过遍历所述加速度传感器的采集时间间隔，计算每个采集时间间隔下微处理器判断的准确度，并计算准确度与加速度传感器在单位时间内采集的信息量的比值，当所述比值达到最大值时，将当前对应的采集时间间隔设置为默认的采集时间间隔。Preferably, the acceleration sensor has a plurality of acquisition time intervals; by traversing the acquisition time intervals of the acceleration sensor, the accuracy of the judgment of the microprocessor at each acquisition time interval is calculated, and the accuracy is calculated with the acceleration sensor in unit time. When the ratio reaches the maximum value, the current corresponding collection time interval is set as the default collection time interval.
优选地，所述模型训练模块中的地坎槽检测模型训练单元，其训练方法包括如下步骤：Preferably, the sill groove detection model training unit in the model training module, its training method comprises the following steps:
步骤41：制作地坎槽检测训练集，使用LabelImg对含有地坎槽的轿厢图片进行标注，具体为框出地坎槽图像并标注类别；Step 41: make a sill detection training set, use LabelImg to label the car picture containing the sill, specifically frame the sill image and label the category;
步骤42：搭建目标检测模型结构，采用迁移学习方法使用在 ILSVRC数据集上训练过的模型作为网络backbone的初始权重并随机化除所述初始权重外的其他参数，在步骤41建立的地坎槽检测训练集上进行训练，损失函数包括类别分类损失函数loss_cls和地坎槽边界框位置回归损失函数loss_bbox；Step 42: Build the structure of the target detection model, use the transfer learning method to use the model trained on the ILSVRC data set as the initial weight of the network backbone and randomize other parameters except the initial weight. The sill slot established in step 41 Training is performed on the detection training set. The loss function includes the category classification loss function loss_cls and the sill trough bounding box position regression loss function loss_bbox;
步骤43：设置停止策略，当达到指定的迭代次数，或者模型在验证集的预测精度达到设定值时停止训练。Step 43: Set a stopping strategy, when the specified number of iterations is reached, or the model stops training when the prediction accuracy of the validation set reaches the set value.
优选地，所述模型训练模块中的非平层开门图像分类模型训练单元，用于训练非平层开门识别模型，对地坎槽图像做二分类，以识别平层状态；Preferably, the non-leveling door opening image classification model training unit in the model training module is used to train the non-leveling door opening recognition model, and the sill and groove images are classified into two to identify the leveling state;
训练非平层开门分类模型的方法包括如下步骤：The method for training a non-level door classification model includes the following steps:
步骤51：制作地坎槽图像分类训练集，具体方法为开门状态下，采集厅门地坎槽和轿门地坎槽高度差绝对值小于等于10mm的地坎槽图像作为平层开门数据集，其他为非平层开门数据集；Step 51: Create a training set of sill image classification. The specific method is to collect the sill image with the absolute value of the height difference between the hall door sill and the car door sill less than or equal to 10mm in the open state as the leveling door opening data set. Others are non-leveling open door datasets;
步骤52：下载并加载Inception V3模型作为预训练模型；Step 52: Download and load the Inception V3 model as a pre-training model;
步骤53：执行迁移学习策略，去掉输出全连接层和softmax层，增加新的二分类的全连接层和softmax分类层，并增删预定中间网络层，作为改进的基于inception结构的深度卷积神经网络；深度卷积神经网络含有Inception结构通过多卷积核融合提取图像不同尺度信息得到更好的表征，Inception结构含有1×1卷积以降维构建参数更少的稀疏网络结构；所述预定中间网络层包括inception块、1×1卷积；Step 53: Execute the transfer learning strategy, remove the output fully connected layer and the softmax layer, add a new two-category fully connected layer and a softmax classification layer, and add or delete a predetermined intermediate network layer as an improved deep convolutional neural network based on the inception structure ; The deep convolutional neural network contains the Inception structure to obtain better characterization by extracting information at different scales of the image through the fusion of multiple convolution kernels. The Inception structure contains 1 × 1 convolution to reduce the dimension to build a sparse network structure with fewer parameters; the predetermined intermediate network Layers include inception blocks, 1×1 convolutions;
步骤54：加载步骤53得到的深度卷积神经网络，创建训练脚本文件；Step 54: load the deep convolutional neural network obtained in step 53, and create a training script file;
步骤55：对地坎槽图像分类训练集样本做预处理，输入训练的参数batch_size，epoch，iteration，batch_size表示一次参数更新过程中使用的所需样本数量，1个iteration等于使用batch_size个样本训练1次，epoch为所有样本总共迭代的次数；Step 55: Preprocess the samples of the sill and trough image classification training set, input the training parameters batch_size, epoch, iteration, batch_size represents the required number of samples used in a parameter update process, 1 iteration is equal to using batch_size samples to train 1 times, epoch is the total number of iterations for all samples;
步骤56：设置停止策略，当迭代达到指定的epoch数和iteration数，或者模型在验证集的预测精度达到设定值时停止训练；Step 56: Set a stopping strategy, when the iteration reaches the specified epoch number and iteration number, or the model stops training when the prediction accuracy of the validation set reaches the set value;
步骤57：增设checkpoint机制，在每个epoch模型训练过程中达到最优时，输出模型文件；Step 57: Add a checkpoint mechanism, and output the model file when the optimum is reached in the training process of each epoch model;
步骤58：根据batch_size加载样本做数据预处理后执行训练过程。Step 58: Load the samples according to batch_size for data preprocessing and then execute the training process.
优选地，地坎槽图像分类训练集中的训练样本通过对原始样本做进行随机裁剪、翻转以及特征变换的方法获得；特征变换如公示（4）（5）（6）所示：Preferably, the training samples in the sill image classification training set are obtained by randomly cropping, flipping, and transforming the original samples; the transforming features are shown in public announcements (4) (5) (6):
其中，
优选地，所述检测识别模块中的图像光照自适应矫正单元，具体用于对视频采集模块采集到的输入图像做光照自适应矫正，处理环境光照度过暗、过亮时的地坎槽图像，以提高光照均匀度并增强图像质量，自适应处理方法如公式（7）（8）所示：Preferably, the image illumination adaptive correction unit in the detection and recognition module is specifically configured to perform illumination adaptive correction on the input image collected by the video acquisition module, and process the sill image when the ambient light is too dark or too bright, In order to improve illumination uniformity and enhance image quality, the adaptive processing method is shown in formula (7) (8):
其中，
加载目标检测模型对地坎槽进行检测，当检测到地坎槽时判断为开门；根据检测到地坎槽长度和其总长度比值判断电梯开门程度α，检测到的地坎槽的实际长度和其实际总长度比值，等于其对应的像素比；当α为0时电梯未开门；当开门程度α大于等于设定的阈值时，识别判断是否开门；所述检测识别模块中的图像角度自适应矫正单元，其具体用于对检测到的地坎槽图像进行角度变换以适应不同镜头、不同型号的门系统、不同类型的开门方式带来的角度畸变，如公示（9）（10）（11）（12）所示：Load the target detection model to detect the sill slot, when the sill slot is detected, it is judged to open the door; according to the ratio of the detected sill slot length and its total length, the elevator door opening degree α is judged, and the actual length of the detected sill slot and the total length are determined. Its actual total length ratio is equal to its corresponding pixel ratio; when α is 0, the elevator does not open the door; when the degree of door opening α is greater than or equal to the set threshold, it is recognized and judged whether to open the door; the image angle in the detection and recognition module is adaptive Correction unit, which is specifically used to perform angle transformation on the detected sill image to adapt to the angle distortion caused by different lenses, different types of door systems, and different types of door opening methods, such as public announcements (9) (10) (11) ) (12) as shown:
其中
优选地，所述检测识别模块中的非平层开门识别单元，具体用于加载非平层开门分类模型，判断厅门、轿门是否非平层并给出置信度β，当β大于设定的阈值时判断为非平层开门。Preferably, the non-leveling door opening identification unit in the detection and identification module is specifically used to load the non-leveling door opening classification model, determine whether the hall door and the car door are non-leveling, and give a confidence level β, when β is greater than the set When the threshold value is reached, it is judged as non-leveling open door.
本发明实施例还提供了一种基于深度学习的非平层开门故障识别方法，其包括如下步骤：The embodiment of the present invention also provides a deep learning-based non-leveling door opening fault identification method, which includes the following steps:
步骤101：视频采集模块启动；加速度监测模块采集电梯轿厢的加速度信号，并将加速度信号传输到微处理器；微处理器接收加速度信号，进行滤波平滑处理，并对完成滤波平滑处理后的加速度曲线进行分析，若判断为进入减速停梯阶段时进入步骤102，若判断为正常行驶则继续进行加速度监测；Step 101: the video acquisition module is started; the acceleration monitoring module collects the acceleration signal of the elevator car, and transmits the acceleration signal to the microprocessor; the microprocessor receives the acceleration signal, performs filtering and smoothing processing, and performs filtering and smoothing processing on the acceleration signal The curve is analyzed, if it is judged that it is entering the stage of deceleration and stopping, go to step 102, and if it is judged that it is normal driving, continue to carry out acceleration monitoring;
步骤102：检测识别模块接收加速度监测模块信息，并根据监测到准备停梯的时间开始截取电梯轿厢运行视频；检测识别模块对截取的视频进行帧处理，对每帧图像进行光照自适应矫正，进入步骤103；Step 102: The detection and identification module receives the information of the acceleration monitoring module, and starts to intercept the running video of the elevator car according to the monitoring time when the elevator is ready to stop; the detection and identification module performs frame processing on the intercepted video, and performs adaptive illumination correction on each frame of image, Enter step 103;
步骤103：检测识别模块加载地坎槽检测模型，对步骤102得到的图像进行目标检测，当检测到地坎槽时即可判断为开门，进入步骤104；Step 103: The detection and recognition module loads the sill detection model, and performs target detection on the image obtained in step 102. When the sill is detected, it can be judged that the door is opened, and the process goes to step 104;
步骤104：检测识别模块对步骤103检测到的地坎槽图像做角度自适应矫正，智能处理不同镜头、不同型号的门系统、不同类型的开门方式等因素带来的图像角度畸变，进入步骤105；Step 104: The detection and recognition module performs angle adaptive correction on the sill image detected in step 103, intelligently handles the image angle distortion caused by factors such as different lenses, different types of door systems, different types of door opening methods, etc., and proceeds to step 105 ;
步骤105：检测识别模块加载非平层开门分类模型，判断步骤104处理后的图像是否为非平层开门，并给出置信度β，当β大于设定的阈值时判断为非平层开门故障；否则回到步骤101。Step 105: The detection and recognition module loads the non-leveling door opening classification model, determines whether the image processed in step 104 is non-leveling door opening, and gives a confidence level β, when β is greater than the set threshold, it is judged as a non-leveling door opening failure ; otherwise, go back to step 101.
本发明提供的一种基于计算机视觉的电梯非平层开门故障识别装置及方法，能够全覆盖到各种类型的电梯门系统，满足电梯行业快速发展的需要，实现对于电梯非平层开门的智能识别预测，克服了传统检测方式的不足，减少非平层目标样本的人工标定工作量，提高对电梯非平层图像和开关门图像的目标检测准确率和识别速度，模型识别精度达95.3%。本装置采用领先的电梯故障识别技术，实现极强的兼容性。The invention provides a computer vision-based elevator non-leveling door opening fault identification device and method, which can fully cover various types of elevator door systems, meet the needs of the rapid development of the elevator industry, and realize intelligent elevator non-leveling door opening. Recognition prediction overcomes the shortcomings of traditional detection methods, reduces the workload of manual calibration of non-leveling target samples, and improves the target detection accuracy and recognition speed of elevator non-leveling images and door opening and closing images, and the model recognition accuracy reaches 95.3%. The device adopts the leading elevator fault identification technology to achieve strong compatibility.
本发明具有如下有益效果：The present invention has the following beneficial effects:
1、充分考虑到各类电梯的传感器差异性，采用视频识别的检测方式，避免改造原有电梯传感器装置的复杂问题，识别装置安装更加简易方便。1. Fully considering the sensor differences of various elevators, the detection method of video recognition is adopted to avoid the complicated problem of modifying the original elevator sensor device, and the installation of the recognition device is simpler and more convenient.
2、充分考虑到门系统的型号、规格的差异性，利用深度学习的方式预先训练好地坎槽模型和非平层开门模型，可以大量减少人工标定非平层图像的工作量。2. Fully considering the differences in the model and specification of the door system, using the deep learning method to pre-train the sill model and the non-leveling door model can greatly reduce the workload of manually calibrating non-leveling images.
3、充分考虑到不同镜头、不同型号的门系统、不同类型的电梯开门方式等带来的镜头拍摄角度畸变，在装置中设置了图像角度自适应矫正单元，可对检测到的地坎槽图像进行角度自动变换。3. Fully considering the lens shooting angle distortion caused by different lenses, different types of door systems, and different types of elevator door opening methods, etc., an image angle adaptive correction unit is set up in the device, which can correct the detected sill image. Perform automatic angle transformation.
4、针对电梯复杂的环境照度对图像造成的影响，本装置设置了图像光照自适应矫正单元，具体用于对视频采集模块采集到的输入图像做光照自适应矫正，以提高光照均匀度并增强图像质量。4. In view of the influence of the complex environmental illumination of the elevator on the image, the device is equipped with an image illumination adaptive correction unit, which is specifically used to perform illumination adaptive correction on the input image collected by the video acquisition module, so as to improve the illumination uniformity and enhance the Image Quality.
5、充分考虑到电梯门和地坎容易被乘客或设备遮挡的情况，合理应对，将镜头设置于电梯轿门和厅门夹缝，可以同时拍摄厅门地坎槽和轿门地坎槽，避免被遮挡，根据检测到地坎槽长度和其总长度比值判断电梯开门程度。5. Fully consider the situation that the elevator doors and sills are easily blocked by passengers or equipment, and deal with it reasonably. Set the camera in the gap between the elevator car door and the hall door. You can shoot the hall door sill and the car door sill at the same time. If it is blocked, the door opening degree of the elevator is judged according to the ratio of the detected sill groove length to its total length.
6、为了提高对电梯非平层图像和开关门图像的目标检测准确率和识别速度，分别制作地坎槽检测训练集、地坎槽图像分类训练集；并进行地坎槽检测模型训练，通过对原始样本做进行随机裁剪、翻转以及特征变换的方法获得训练非平层开门识别模型，对地坎槽图像做二分类，快速识别平层或非平层。6. In order to improve the target detection accuracy and recognition speed of elevator non-leveling images and door opening and closing images, respectively make a sill detection training set and a sill image classification training set; and train the sill detection model. The method of random cropping, flipping and feature transformation of the original sample is used to obtain a training non-leveling door recognition model, and the sill and trough images are classified into two categories to quickly identify the leveling or non-leveling.
7、针对检测识别模块的算力消耗问题，配置加速度监测模块，当发现电梯进入减速停梯状态时，检测识别模块开始工作，否则待机，以节约算力。7. In view of the computing power consumption of the detection and recognition module, configure the acceleration monitoring module. When the elevator is found to enter the deceleration stop state, the detection and recognition module starts to work, otherwise it will stand by to save computing power.
8、采用深度学习模型，检测识别的准确性有保证，相比传统的利用传感器测距等方式，具有更好的鲁棒性和适用性。8. Using a deep learning model, the accuracy of detection and recognition is guaranteed, and it has better robustness and applicability than traditional methods such as using sensors to measure distance.
9、充分考虑到产业化和市场化方面问题，本方法在普通单目相机作为图像采集装置的基础上，通过优化图像识别模型和算法来实现测距功能，相比采用深度相机等其他图像测距方式，具有成本更低、体积更小、算力需求更少、图像处理速度更快等优势，具有良好的产业化，也更易于现场安装。9. Fully considering the industrialization and marketization issues, this method realizes the ranging function by optimizing the image recognition model and algorithm on the basis of the ordinary monocular camera as the image acquisition device. The distance method has the advantages of lower cost, smaller size, less computing power requirements, and faster image processing speed. It has good industrialization and is easier to install on site.
附图说明Description of drawings
为了更清楚地说明本发明的技术方案，下面将对实施方式中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本发明的一些实施方式，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。In order to illustrate the technical solutions of the present invention more clearly, the following will briefly introduce the accompanying drawings used in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention, which are common in the art. As far as technical personnel are concerned, other drawings can also be obtained based on these drawings without any creative effort.
图1为本发明实施例提供的电梯非平层开门故障识别装置的结构示意图。FIG. 1 is a schematic structural diagram of an elevator non-leveling door opening fault identification device provided by an embodiment of the present invention.
图2为本发明实施例提供的电梯非平层开门故障识别方法的流程图。FIG. 2 is a flowchart of a method for identifying an elevator non-leveling door opening fault provided by an embodiment of the present invention.
图3为本发明实施例的样本特征预处理运算方式举例。FIG. 3 is an example of a sample feature preprocessing operation manner according to an embodiment of the present invention.
图4为本发明实施例检测识别模块的工作过程示意图。FIG. 4 is a schematic diagram of a working process of a detection and identification module according to an embodiment of the present invention.
图5为本发明实施例视频采集模块和加速度监测模块安装位置示意图。FIG. 5 is a schematic diagram of installation positions of a video acquisition module and an acceleration monitoring module according to an embodiment of the present invention.
具体实施方式Detailed ways
下面将结合本发明实施例中的附图，对本发明实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例仅仅是本发明一部分实施例，而不是全部的实施例。基于本发明中的实施例，本领域普通技术人员在没有作出创造性劳动前提下所获得的所有其他实施例，都属于本发明保护的范围。The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.
请参阅图1，本发明实施例提供了一种基于计算机视觉的电梯非平层开门故障识别装置，包括视频采集模块10、加速度监测模块20、模型训练模块30以及检测识别模块40；其中，Referring to FIG. 1, an embodiment of the present invention provides a computer vision-based elevator non-leveling door opening fault identification device, including a
如图5所示，在本实施例中，所述视频采集模块10设置于电梯轿厢1内的顶部，所述视频采集模块10可为摄像装置，如微型相机或者摄像头等，其镜头3设置于电梯轿门6和厅门7的夹缝，垂直于地坎槽，正对地坎槽进行拍摄，从而可以同时拍摄厅门地坎槽5和轿门地坎槽4。As shown in FIG. 5 , in this embodiment, the
在本实施例中，镜头具体安装位置根据电梯开门类型而定，且与电梯门锁打开的位置一致，如当电梯为中分开门时，镜头安装在门的正中间位置；电梯为旁开门时，镜头安装在轿门的开门一端。在本实施例中，所述摄像装置优选的采用红外广角摄像头，能够适应光线微弱的环境，其中摄像头的镜头正对地坎槽。In this embodiment, the specific installation position of the lens is determined according to the type of elevator door opening, and is consistent with the position where the elevator door lock is opened. , the lens is installed on the open end of the car door. In this embodiment, the camera device preferably adopts an infrared wide-angle camera, which can adapt to a weak light environment, wherein the lens of the camera is facing the sill groove.
进一步地，摄像装置可以采用补光灯和光照度传感器，当光照度低于设定限度时，补光灯自动打开，对地坎槽补充光照。Further, the camera device can use a fill light and an illuminance sensor. When the illuminance is lower than a set limit, the fill light is automatically turned on to supplement the illumination for the sill groove.
本实施例中，视频采集模块10与检测识别模块40电性连接，视频采集模块40能够将实时采集的视频信号通过连接线路传输到检测识别模块40并进行存储。In this embodiment, the
所述加速度监测模块20，包括加速度传感器以及微处理器，微处理器可为单片机；所述加速度监测模块20用于对电梯运行加速度进行监控，以预判电梯是否进入减速状态，加速度监测模块20与检测识别模块40电性连接。The
所述加速度传感器采集加速度的时间间隔可为60ms、40ms、 25ms。其中随着加速度采样频率的增大，微处理器对于检测加速度进行电梯即将停梯的判断就愈为准确，并且对硬件存储要求和信息传输要求指数级上升；特别在40ms处出现拐点，此时微处理器判断的准确度与加速度传感器在单位时间内采集的信息量的比值达到最大值，因此在本实施例中加速度传感器采用40ms的采集时间间隔，也就是25HZ的采集频率，加速度的单位为cm/s2。The time interval at which the acceleration sensor collects the acceleration may be 60ms, 40ms, or 25ms. Among them, with the increase of the acceleration sampling frequency, the microprocessor will be more accurate for detecting the acceleration to judge that the elevator is about to stop, and the requirements for hardware storage and information transmission increase exponentially; The ratio between the accuracy of the microprocessor's judgment and the amount of information collected by the acceleration sensor per unit time reaches the maximum value. Therefore, in this embodiment, the acceleration sensor adopts a collection time interval of 40ms, that is, the collection frequency of 25HZ. The unit of acceleration is cm/s2 .
所述微处理器利用方差、极差以及极差对应的索引距离占加速度序列长度的占比对完成滤波平滑处理后的加速度曲线进行加速度检测以判断电梯是否停梯。所述方差是用来度量加速度序列的波动程度，表达的是加速度的平稳性，用
其中，
所述极差是用来衡量数值变动的最大范围，是测定数值变动较为方便的指标，是最大值与最小值之间的差距，用
其中，
所述极差对应的索引距离占加速度序列长度的占比是刻画的加速度骤变程度，描述的是最大值和最小值所在序列位置的间距占整个运行加速度序列的比例，用
其中，
在本实施例中方差满足大于1000、极差满足大于500和/或极差对应的索引距离占加速度序列长度的占比小于10%，则认为电梯即将停梯；否则微处理器判断为加速度正常。其中方差、极差以及极差对应的索引距离占加速度序列长度的占比之间的关系为“和”时，说明条件苛刻，判断为电梯即将停梯的情况会更为精确，但是会出现漏查的情况；关系为“或”时，说明条件松弛，能够全面排查可能为电梯即将停梯的情况，但是对于判断为电梯即将停梯情况的精度会降低。所述加速度序列长度表示电梯轿厢的速度从零开始到恢复为零的时间间隔内的加速度数据长度；极差对应的索引距离表示加速度序列中加速度的最大值和最小值之间的加速度数据长度；所述数据长度为数据的数量，电梯轿厢的速度由速度传感器采集，速度传感器设置于电梯轿厢。In this embodiment, if the variance is greater than 1000, the range is greater than 500, and/or the index distance corresponding to the range accounts for less than 10% of the length of the acceleration sequence, it is considered that the elevator is about to stop; otherwise, the microprocessor determines that the acceleration is normal . When the relationship between the variance, the range and the ratio of the index distance corresponding to the range to the length of the acceleration sequence is "sum", it means that the conditions are harsh, and it will be more accurate to judge that the elevator is about to stop, but there will be leakage. When the relationship is "or", it means that the conditions are relaxed, and it is possible to comprehensively check the situation that the elevator is about to stop, but the accuracy of judging the situation that the elevator is about to stop will be reduced. The length of the acceleration sequence represents the length of the acceleration data in the time interval when the speed of the elevator car starts from zero and returns to zero; the index distance corresponding to the extreme difference represents the length of the acceleration data between the maximum value and the minimum value of the acceleration in the acceleration sequence The data length is the number of data, the speed of the elevator car is collected by a speed sensor, and the speed sensor is arranged in the elevator car.
所述模型训练模块30，包括地坎槽目标检测模型训练单元和非平层开门图像分类模型训练单元；用于预先训练地坎槽检测模型和非平层开门识别模型，以供检测模块加载使用。The
所述检测识别模块40，包括图像光照自适应矫正单元，地坎槽检测单元，图像角度自适应矫正单元，非平层开门识别单元；用于对图像做自适应处理并加载训练好的模型进行预测、识别。The detection and recognition module 40 includes an image illumination adaptive correction unit, a sill groove detection unit, an image angle adaptive correction unit, and a non-leveling door recognition unit; it is used to perform adaptive processing on the image and load the trained model for Predict, identify.
如图2所示，所述视频采集模块10与检测识别模块40电性连接；模型训练模块30和检测识别模块40都设置于后端服务器。As shown in FIG. 2 , the
所述模型训练模块30中的地坎槽检测模型训练单元，其训练方法包括如下步骤：The sill groove detection model training unit in the
步骤1：制作地坎槽检测训练集，使用LabelImg对含有地坎槽的轿厢图片进行批量标注，具体为框出地坎槽图像并标注类别；Step 1: Create a training set for sill detection, and use LabelImg to label the pictures of the car containing the sill in batches, specifically frame the sill image and label the category;
步骤2：搭建目标检测模型结构，采用迁移学习方法使用在 ILSVRC数据集上训练过的模型作为网络backbone的初始权重并随机化除初始权重外的其他参数，在步骤1建立的数据集上进行训练，损失函数包括类别分类损失函数loss_cls和地坎槽边界框位置回归损失函数loss_bbox。本实施例中类别损失函数采用二值交叉熵，边框位置回归损失函数采用L2损失函数；Step 2: Build the target detection model structure, use the transfer learning method to use the model trained on the ILSVRC data set as the initial weight of the network backbone and randomize other parameters except the initial weight, and train on the data set established in
步骤3：设置停止策略，当达到指定的迭代次数，或者模型在验证集的预测精度达到设定值时停止训练。Step 3: Set a stopping strategy, when the specified number of iterations is reached, or the model stops training when the prediction accuracy of the validation set reaches the set value.
在本实施例中地坎槽图像目标检测模型采用YoLoV3, YoLoV3用于特征提取的backbone网络为darknet53网络，输出层有三个全卷积网络作为输出层，每个输出层在不同维度的特征图上输出结果，其输出维度为In this embodiment, the sill image target detection model adopts YoLoV3, the backbone network used by YoLoV3 for feature extraction is darknet53 network, and the output layer has three fully convolutional networks as output layers, and each output layer is on a feature map of different dimensions. The output result, whose output dimension is
其中S是S×S特征图的尺度大小，B表示每个格子输出的边框数目，C是检测的类别数，5是输出的4个位置边框坐标和属于该物体的置信度得分。本实施例中只有地坎槽一个类别因此C为1，输入图像大小为416×416，预测((52×52)+(26×26)+ (13×13)) ×3=10647个边界框。where S is the scale of the S×S feature map, B is the number of bounding boxes output by each grid, C is the number of detected categories, and 5 is the coordinates of the
本实施例中，数据集可采用摄像头采集开门期间电梯轿厢图片后用标注工具LabelImg对含有地坎槽的轿厢图片进行位置和类别的标注后生成。在获得数据集后，对其做数据清理后按照8:2随机划分为训练集和验证集。使用在ILSVRC 图像数据集上训练过的darknet53 模型作为网络backbone的初始权重，并用he_normal方法初始化除初始权重外的其他参数。训练中使用的损失函数包括类别损失函数以及边框位置回归损失函数，类别损失函数采用二值交叉熵，边框位置回归损失函数采用L2损失函数。在训练集上做训练并在验证集上验证以根据实时精度设置停止策略。In this embodiment, the data set can be generated by using a camera to collect pictures of the elevator car during door opening, and then using the labeling tool LabelImg to label the picture of the car containing the sill with the location and category. After the data set is obtained, it is randomly divided into training set and validation set according to 8:2 after data cleaning. Use the darknet53 model trained on the ILSVRC image dataset as the initial weights of the network backbone, and initialize other parameters except the initial weights with the he_normal method. The loss functions used in training include the category loss function and the border position regression loss function. The category loss function adopts binary cross entropy, and the border position regression loss function adopts the L2 loss function. Do train on the training set and validate on the validation set to set the stopping policy based on real-time accuracy.
所述模型训练模块30中的非平层开门图像分类模型训练单元，用于训练非平层开门识别模型，对地坎槽图像做二分类，以识别平层或非平层；The non-leveling door opening image classification model training unit in the
Inception结构显著增加了每一步的神经单元数目，且在不同尺度上进行了预处理聚合，替代了人工确定卷积层中过滤器的类型，让网络自我学习参数。而Inception V3通过分解卷积核尺寸、改变降低特征图尺寸来达到堆积更多的inception模块提升计算资源的利用，其总计47层。The Inception structure significantly increases the number of neural units in each step, and performs preprocessing aggregation at different scales, instead of manually determining the type of filters in the convolutional layer, allowing the network to learn parameters by itself. Inception V3 achieves stacking more inception modules to improve the utilization of computing resources by decomposing the size of the convolution kernel and changing and reducing the size of the feature map, with a total of 47 layers.
训练非平层开门识别模型的方法包括如下步骤：The method for training a non-leveling door recognition model includes the following steps:
步骤1：制作地坎槽图像分类训练集，具体方法为开门状态下，采集厅门地坎槽和轿门地坎槽高度差绝对值小于等于10mm的地坎槽图像作为平层开门数据集，其他为非平层开门数据集；Step 1: Create a training set of sill image classification. The specific method is to collect the sill image with the absolute value of the height difference between the hall door sill and the car door sill less than or equal to 10mm in the open state as the leveling door opening data set. Others are non-leveling open door datasets;
步骤2：下载并加载Inception V3模型作为预训练模型；Step 2: Download and load the Inception V3 model as a pretrained model;
步骤3：执行迁移学习策略，去掉输出全连接层和softmax层，增加新的二分类的全连接层和softmax分类层，并可选地增删部分中间网络层如部分inception块、1×1卷积，作为改进的基于inception结构的深度卷积神经网络；网络含有Inception结构通过多卷积核融合提取图像不同尺度信息得到更好的表征，Inception结构含有1×1卷积以降维构建参数更少的稀疏网络结构。Step 3: Execute the transfer learning strategy, remove the output fully connected layer and softmax layer, add a new two-category fully connected layer and softmax classification layer, and optionally add or delete some intermediate network layers such as some inception blocks, 1×1 convolution , as an improved deep convolutional neural network based on the inception structure; the network contains the Inception structure to extract information at different scales of the image through multi-convolution kernel fusion, and the Inception structure contains 1 × 1 convolution to reduce dimensionality. Sparse network structure.
步骤4：加载步骤3得到的深度卷积神经网络，创建训练脚本文件；Step 4: Load the deep convolutional neural network obtained in
步骤5：对地坎槽图像分类训练集样本做预处理，输入训练的参数batch_size，epoch，iteration等，batch_size表示一次参数更新过程中使用的所需样本数量，1个iteration等于使用batch_size个样本训练1次，epoch为所有样本总共迭代的次数。本实施例batch_size取64，epoch取100。Step 5: Preprocess the samples of the sill and trough image classification training set, and input the training parameters batch_size, epoch, iteration, etc. batch_size represents the required number of samples used in one parameter update process, and 1 iteration is equal to using batch_size samples for
步骤6：设置停止策略，当迭代达到指定的epoch数和iteration数，或者模型在验证集的预测精度达到设定值时停止训练。Step 6: Set the stopping strategy. When the iteration reaches the specified number of epochs and iterations, or the model stops training when the prediction accuracy of the validation set reaches the set value.
步骤7：增设checkpoint机制，在每个epoch深度学习过程中达到最优时，输出模型文件；Step 7: Add a checkpoint mechanism, and output the model file when the optimum is reached in the deep learning process of each epoch;
步骤8：根据batch_size加载样本做数据预处理后执行训练过程。Step 8: Load samples according to batch_size for data preprocessing and execute the training process.
所述Softmax函数为归一化指数函数，是逻辑函数的推广，用于多分类。Softmax函数将多个神经元的输出映射到(0,1)区间内，表示输出概率，公式如下：The Softmax function is a normalized exponential function, which is a generalization of a logistic function and is used for multi-classification. The Softmax function maps the outputs of multiple neurons to the (0,1) interval, representing the output probability. The formula is as follows:
其中，
所述的非平层开门图像分类模型训练单元，训练样本通过对原始样本做进行随机裁剪、翻转以及特征变换的方法获得；特征变换如公式（4）（5）（6）所示：In the non-leveling door image classification model training unit, the training samples are obtained by randomly cropping, flipping and feature transformation on the original samples; the feature transformations are shown in formulas (4) (5) (6):
其中，
在本实施例中假设设定：In this example it is assumed that:
非平层开门样本总数量为800张，平层开门的样本总数量为1000张；The total number of samples for non-leveling doors is 800, and the total number of samples for leveling doors is 1000;
batch_size =64，训练集：测试集=8：2；batch_size = 64, training set: test set = 8: 2;
测试集的混淆矩阵如下：The confusion matrix for the test set is as follows:
y=0代表真实负例的样本数量，y=1代表真实正例的样本数量；yhat=0代表推断出为负例的样本数量，yhat=1代表推断出为正例的样本数量。y=0 represents the number of samples of true negative examples, y=1 represents the number of samples of true positive examples; yhat=0 represents the number of samples inferred as negative examples, and yhat=1 represents the number of samples inferred as positive examples.
其中测试集的总数=（800+1000）*0.2=192+9+9+151=360；The total number of test sets = (800+1000)*0.2=192+9+9+151=360;
模型识别精度=（192+151）/360≈95.3%，满足模型预测精度的设定值要求。Model recognition accuracy=(192+151)/360≈95.3%, which meets the set value requirements of model prediction accuracy.
所述检测识别模块中的图像光照自适应矫正单元，具体用于对视频采集模块采集到的输入图像（即地坎槽图像）做光照自适应矫正，处理昏暗电梯的低照度图像，或强灯照射、观光电梯等的高照度图像，特别是金属地坎槽反光引起的过曝、高反光图像，以提高光照均匀度并增强图像质量，自适应处理方法如公式（7）（8）所示：The image illumination self-adaptive correction unit in the detection and recognition module is specifically used to perform self-adaptive correction of illumination on the input image (ie the sill image) collected by the video acquisition module, and to process low-illumination images of dim elevators, or strong lights. High-illuminance images of illumination, sightseeing elevators, etc., especially over-exposed and high-reflection images caused by reflection of metal sill grooves, in order to improve illumination uniformity and enhance image quality, the adaptive processing method is shown in formula (7) (8) :
其中，
所述检测识别模块40中的地坎槽检测单元，加载目标检测模型对地坎槽进行检测；当检测到地坎槽时即可判断为开门，进一步的，可根据检测到地坎槽长度和其总长度比值判断电梯开门程度α，检测到的地坎槽的实际长度和其实际总长度比值，等于其对应的像素比；当α为0时电梯未开门；当开门程度α大于等于设定的阈值时对地坎槽图像识别判断是否非平层开门。本实施例中设定α阈值为0.1。The sill groove detection unit in the detection and identification module 40 loads the target detection model to detect the sill groove; when the sill groove is detected, it can be judged as opening the door, and further, according to the detected sill groove length and Its total length ratio judges the elevator door opening degree α, the ratio between the actual length of the detected sill groove and its actual total length is equal to its corresponding pixel ratio; when α is 0, the elevator does not open the door; when the door opening degree α is greater than or equal to the setting When the threshold value is set, the image recognition of the sill groove is used to judge whether the door is not open on a leveling level. In this embodiment, the α threshold is set to 0.1.
所述检测识别模块40中的图像角度自适应矫正单元，其具体用于对检测到的地坎槽图像进行角度变换以适应不同镜头、不同门系统、不同开门方式等带来的角度畸变，如公式（9）（10）（11）（12）所示：The image angle adaptive correction unit in the detection and recognition module 40 is specifically used to perform angle transformation on the detected sill image to adapt to the angle distortion caused by different lenses, different door systems, different door opening methods, etc. Formulas (9) (10) (11) (12) are shown as:
其中，
所述检测识别模块40中的非平层开门识别单元，具体用于加载非平层开门分类模型，判断厅门、轿门是否非平层并给出置信度β，当β大于设定的阈值时判断为非平层开门。本实施例中置信度的阈值设为90%。The non-leveling door opening identification unit in the detection and identification module 40 is specifically used to load the non-leveling door opening classification model, determine whether the hall door and the car door are not leveled, and give a confidence level β, when β is greater than the set threshold value When it is judged as non-leveling door opening. In this embodiment, the threshold of confidence is set to 90%.
在检测时，加载训练好的检测模型对图像中的地坎槽进行检测，将图片归一化之后输入到训练好的网络中可得到输出，并对输出做转换得到实际的位置边界框。输出转换公式如下：During detection, load the trained detection model to detect the sill grooves in the image, normalize the image and input it into the trained network to get the output, and convert the output to get the actual location bounding box. The output conversion formula is as follows:
其中，
最后对得到的边框采用NMS算法滤除重叠的框，保留最终有效框。检测到的地坎槽图像随即作为非平层开门识别模型的输入。检测识别模块的工作流程如图4所示。Finally, the NMS algorithm is used for the obtained frame to filter out the overlapping frames, and the final valid frame is retained. The detected sill image is then used as the input of the non-leveling door recognition model. The workflow of the detection and identification module is shown in Figure 4.
请参阅图2，本发明实施例提供了一种基于计算机视觉的电梯非平层故障识别方法，包括如下步骤：Referring to FIG. 2, an embodiment of the present invention provides a computer vision-based elevator non-leveling fault identification method, including the following steps:
步骤1：图像采集模块启动；加速度监测模块采集电梯轿厢的加速度信号，并将加速度信号传输到微处理器；微处理器接收加速度信号，进行滤波平滑处理，并对完成滤波平滑处理后的加速度曲线进行分析，若判断为进入减速阶段准备停梯时进入步骤2，若判断为正常行驶则继续进行加速度采集；Step 1: The image acquisition module is started; the acceleration monitoring module collects the acceleration signal of the elevator car, and transmits the acceleration signal to the microprocessor; Analyze the curve, if it is judged to enter the deceleration stage and prepare to stop the elevator, go to step 2, and if it is judged to be driving normally, continue to collect acceleration;
步骤2：检测识别模块接收加速度信息，并根据加速度监测到准备停梯的时间截取电梯轿厢运行视频；对截取的视频进行帧处理，对每帧图像进行光照自适应矫正，进入步骤3；Step 2: The detection and identification module receives the acceleration information, and intercepts the running video of the elevator car according to the time when the acceleration is monitored to prepare to stop the elevator; frame processing is performed on the intercepted video, and the illumination adaptive correction is performed on each frame of image, and the
步骤3：检测识别模块加载地坎槽检测模型，对步骤2得到的图像进行目标检测，当检测到地坎槽时即可判断为开门，进入步骤4；Step 3: The detection and recognition module loads the sill detection model, and performs target detection on the image obtained in step 2. When the sill is detected, it can be judged that the door is opened, and the
步骤4：检测识别模块对步骤3检测到的地坎槽图像做角度自适应矫正，以处理不同镜头、不同型号的门系统、不同类型的开门方式等带来的角度畸变，进入步骤5；Step 4: The detection and recognition module performs angle adaptive correction on the sill image detected in
步骤5：检测识别模块加载非平层开门分类模型，判断步骤4处理后的图像是否为非平层开门，并给出置信度
本发明实施例提供的一种基于计算机视觉的电梯非平层故障识别装置和方法，能够全覆盖到各种类型的电梯门系统，满足电梯行业快速发展的需要，实现对于电梯非平层开门的智能识别预测，克服了传统检测方式的不足，减少非平层目标样本的人工标定工作量，提高对电梯非平层图像和开关门图像的目标检测准确率和识别速度，模型识别精度达95.3%。本实施例采用领先的电梯故障识别技术，实现极强的兼容性。A computer vision-based elevator non-leveling fault identification device and method provided by the embodiments of the present invention can fully cover various types of elevator door systems, meet the needs of the rapid development of the elevator industry, and realize the detection of elevator non-leveling door openings. Intelligent recognition and prediction overcomes the shortcomings of traditional detection methods, reduces the workload of manual calibration of non-leveling target samples, and improves the target detection accuracy and recognition speed of elevator non-leveling images and door opening and closing images, and the model recognition accuracy reaches 95.3% . This embodiment adopts the leading elevator fault identification technology to achieve extremely strong compatibility.
需说明的是，以上所描述的装置实施例仅仅是示意性的，其中所述作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部模块来实现本实施例方案的目的。另外，本发明提供的装置实施例附图中，模块之间的连接关系表示它们之间具有通信连接，具体可以实现为一条或多条通信总线或信号线。本领域普通技术人员在不付出创造性劳动的情况下，即可以理解并实施。It should be noted that the device embodiments described above are only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical unit, that is, it can be located in one place, or it can be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution in this embodiment. In addition, in the drawings of the apparatus embodiments provided by the present invention, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines. Those of ordinary skill in the art can understand and implement it without creative effort.
以上所述，仅为本发明较佳的具体实施方式，但本发明的保护范围并不局限于此，任何熟悉本技术领域的技术人员在本发明揭露的技术范围内，可轻易想到的变化或替换，都应涵盖在本发明的保护范围之内。因此，本发明的保护范围应该以权利要求的保护范围为准。The above description is only a preferred embodiment of the present invention, but the protection scope of the present invention is not limited to this. Substitutions should be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.
Claims (9)
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110400660.9ACN112801072B (en) | 2021-04-14 | 2021-04-14 | A computer vision-based elevator non-leveling door opening fault identification device and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110400660.9ACN112801072B (en) | 2021-04-14 | 2021-04-14 | A computer vision-based elevator non-leveling door opening fault identification device and method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112801072A CN112801072A (en) | 2021-05-14 |
| CN112801072Btrue CN112801072B (en) | 2021-11-02 |
Family
ID=75811344
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110400660.9AActiveCN112801072B (en) | 2021-04-14 | 2021-04-14 | A computer vision-based elevator non-leveling door opening fault identification device and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112801072B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113936250A (en)* | 2021-10-18 | 2022-01-14 | 日立楼宇技术（广州）有限公司 | A method, device, computer equipment and medium for calculating the opening and closing amplitude of an elevator door |
| CN114572787B (en)* | 2022-03-04 | 2023-06-06 | 陈小平 | Elevator non-flat-bed door fault identification device and method based on computer vision |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107416630A (en)* | 2017-09-05 | 2017-12-01 | 广州日滨科技发展有限公司 | The detection method and system of the improper closing of elevator |
| CN107601199A (en)* | 2017-08-07 | 2018-01-19 | 广州日滨科技发展有限公司 | The detection method and system of elevator components failure |
Family Cites Families (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2011011872A (en)* | 2009-07-02 | 2011-01-20 | Mitsubishi Electric Building Techno Service Co Ltd | Failure detecting device for elevator |
| CN107539855B (en)* | 2017-08-07 | 2019-09-06 | 日立楼宇技术（广州）有限公司 | Method and system for detecting elevator leveling faults |
| CN108358004B (en)* | 2018-02-05 | 2019-05-21 | 苏州钜立智能系统有限公司 | Elevator operation monitoring method based on color identification technology |
| CN108341313A (en)* | 2018-03-28 | 2018-07-31 | 灏景（厦门）文化传媒股份有限公司 | A kind of method for early warning and its multimedia Forewarning Terminal of elevator safety operation environmental monitoring |
| JP2020104983A (en)* | 2018-12-27 | 2020-07-09 | 株式会社日立製作所 | Device and method for elevator operation control |
| CN110127475B (en)* | 2019-03-27 | 2020-12-25 | 浙江新再灵科技股份有限公司 | Elevator taking personnel classification and elevator taking rule analysis method and system thereof |
| CN110407051A (en)* | 2019-08-02 | 2019-11-05 | 杭州岁丰信息技术有限公司 | A kind of identification lift running safety monitoring of audio-video and pacify system |
| CN111144247B (en)* | 2019-12-16 | 2023-10-13 | 浙江大学 | A method for retrograde detection of escalator passengers based on deep learning |
| CN111792472B (en)* | 2020-06-03 | 2023-12-08 | 上海三菱电梯有限公司 | Control method and control system for elevator leveling position |
| CN111731962B (en)* | 2020-06-29 | 2022-04-22 | 浙江新再灵科技股份有限公司 | Opening and closing fault detection method and detection early warning system for door |
| CN112209193A (en)* | 2020-07-22 | 2021-01-12 | 上海城建职业学院 | Intelligent elevator system |
| CN112347862A (en)* | 2020-10-20 | 2021-02-09 | 浙江新再灵科技股份有限公司 | Elevator door fault real-time detection method based on machine vision |
| CN112357707B (en)* | 2020-10-21 | 2022-06-21 | 日立楼宇技术（广州）有限公司 | An elevator detection method, device, robot and storage medium |
| CN112591572A (en)* | 2020-12-03 | 2021-04-02 | 日立楼宇技术（广州）有限公司 | Elevator leveling precision verification system, method and device and controller |
- 2021
- 2021-04-14CNCN202110400660.9Apatent/CN112801072B/enactiveActive
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107601199A (en)* | 2017-08-07 | 2018-01-19 | 广州日滨科技发展有限公司 | The detection method and system of elevator components failure |
| CN107416630A (en)* | 2017-09-05 | 2017-12-01 | 广州日滨科技发展有限公司 | The detection method and system of the improper closing of elevator |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112801072A (en) | 2021-05-14 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111754498A (en) | A Conveyor Belt Idler Detection Method Based on YOLOv3 | |
| CN111079640B (en) | Vehicle type identification method and system based on automatic amplification sample | |
| CN111784633A (en) | An automatic detection algorithm for insulator defects for power inspection video | |
| CN107220603A (en) | Vehicle checking method and device based on deep learning | |
| CN109711322A (en) | A kind of people's vehicle separation method based on RFCN | |
| CN112801072B (en) | A computer vision-based elevator non-leveling door opening fault identification device and method | |
| CN112508946B (en) | Cable tunnel anomaly detection method based on antagonistic neural network | |
| CN111160100A (en) | A Lightweight Deep Model Aerial Vehicle Detection Method Based on Sample Generation | |
| CN111881833B (en) | Vehicle detection method, device, equipment and storage medium | |
| CN112801071B (en) | A system and method for elevator asynchronous door opening recognition based on deep learning | |
| CN116311212B (en) | Ship number identification method and device based on high-speed camera and in motion state | |
| CN114255339A (en) | A method, device and storage medium for identifying breakpoints of power transmission wires | |
| CN109086803A (en) | A kind of haze visibility detection system and method based on deep learning and the personalized factor | |
| CN113343926A (en) | Driver fatigue detection method based on convolutional neural network | |
| CN106570885A (en) | Background modeling method based on brightness and texture fusion threshold value | |
| CN115187878A (en) | Wind turbine blade defect detection method based on UAV image analysis | |
| CN115953719A (en) | Multi-target recognition computer image processing system | |
| CN110163081A (en) | SSD-based real-time regional intrusion detection method, system and storage medium | |
| CN109359518A (en) | A kind of infrared video moving object recognition method, system and alarm device | |
| CN111860141B (en) | A real-time pedestrian detection system and method based on scene structure | |
| CN113822165A (en) | Method for detecting vehicle window state of road bayonet and counting persons in vehicle | |
| CN118962704A (en) | A drone hanging line method and related device based on safety distance detection | |
| CN112183287A (en) | A People Counting Method for Mobile Robots in Complex Background | |
| CN117876353A (en) | Electronic equipment surface scratch detecting system based on visual recognition | |
| CN116503664A (en) | Classification method for distinguishing high-similarity target objects |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |