技术领域Technical field
本发明涉及智能仪表的自动识别技术，尤其涉及一种倾斜指针式仪表自动读数方法。The present invention relates to automatic identification technology of intelligent meters, and in particular to an automatic reading method for tilted pointer meters.
背景技术Background technique
随着深度学习、模式识别的发展，深度学习算法在工业仪表图像识别处理领域的应用越来越广泛。工业场景中采集仪表图像时是通过对摄像头的预置位进行调整，使得一个摄像头尽可能多的拍摄到多个仪表的画面，通过后续深度卷积神经网络识别的方式，提取出指针，进而计算指针角度，计算指针读数图像识别的方式，得到仪表的读数。由于摄像头安装以及需要对多个仪表的拍摄，拍摄的仪表往往存在一定的倾斜。在现有的矫正倾斜仪表图像的方法中，透射变换是最主要的方式。透射变换作用于指针提取之后，计算指针角度之前，其原理是基于二维空间对图像进行坐标变换，利用特征点的匹配和对应的坐标关系，构建透射矩阵，在进行大幅度的透射变换时，可能会产生图像拉伸、变形或剪裁的问题，这就使得计算指针角度的时候会产生偏差，引起误差，使得仪表读数不准确。With the development of deep learning and pattern recognition, deep learning algorithms are increasingly used in the field of industrial instrument image recognition processing. When collecting instrument images in industrial scenes, the preset position of the camera is adjusted so that one camera can capture as many images of multiple instruments as possible. Through subsequent deep convolutional neural network recognition, the pointer is extracted and then calculated. Pointer angle, calculate pointer reading image recognition method, get the meter reading. Due to the installation of the camera and the need to photograph multiple instruments, the instruments being photographed often have a certain tilt. Among the existing methods for correcting tilted instrument images, transmission transformation is the most important method. The principle of transmission transformation after pointer extraction and before calculation of pointer angle is to perform coordinate transformation on the image based on two-dimensional space, and use the matching of feature points and corresponding coordinate relationships to construct a transmission matrix. When performing large-scale transmission transformation, There may be problems with image stretching, deformation or clipping, which will cause deviations when calculating the pointer angle, causing errors and making meter readings inaccurate.
通过透射变换对倾斜的图像进行矫正时，可能需要进行图像的拉伸、压缩或裁剪，以使得倾斜的对象变得垂直或水平。这样可能会造成图像中的某些区域被裁剪或变形，导致丢失细节或造成图像质量下降。尤其在进行大角度倾斜矫正时，由于透射变换是基于二维坐标的变换，不可避免地会引入一定程度的图像形变，特别是在图像的边缘部分。同时，在进行透射变换时，至少需要4对匹配的特征点来估计透射矩阵，然而，在某些情况下，可能很难找到足够数量和质量的匹配点，尤其是在复杂的场景或图像中，这可能导致透射变换无法准确估计，从而影响矫正效果。When correcting a tilted image through transmission transformation, it may be necessary to stretch, compress, or crop the image to make the tilted object vertical or horizontal. This may cause some areas of the image to be cropped or distorted, resulting in a loss of detail or a loss of image quality. Especially when performing large-angle tilt correction, since the transmission transformation is based on the transformation of two-dimensional coordinates, a certain degree of image deformation will inevitably be introduced, especially at the edges of the image. At the same time, when performing transmission transformation, at least 4 pairs of matching feature points are needed to estimate the transmission matrix. However, in some cases, it may be difficult to find a sufficient number and quality of matching points, especially in complex scenes or images , which may cause the transmission transformation to be inaccurately estimated, thus affecting the correction effect.
为此，如何解决倾斜状态下仪表识别的问题成为当前亟需解决的技术问题。For this reason, how to solve the problem of instrument identification in a tilted state has become an urgent technical problem that needs to be solved.
发明内容Contents of the invention
(一)要解决的技术问题(1) Technical problems to be solved
针对现有技术的不足，本发明实施例提供一种倾斜指针式仪表自动读数方法。In view of the shortcomings of the existing technology, embodiments of the present invention provide an automatic reading method for a tilted pointer meter.
(二)技术方案(2) Technical solutions
为了达到上述目的，本发明采用的主要技术方案包括：In order to achieve the above objectives, the main technical solutions adopted by the present invention include:
第一方面，本发明实施例提供一种倾斜指针式仪表自动读数方法，包括：In a first aspect, an embodiment of the present invention provides an automatic reading method for a tilted pointer instrument, including:
步骤101、基于预先训练的深度卷积神经网络对待识别的仪表倾斜图进行识别，获得深度卷积神经网络识别输出的角度值一ang_Net，以及获取角度值一ang_Net所属的三维坐标[x_Net y_Net z_Net]；Step 101. Recognize the instrument tilt map to be recognized based on the pre-trained deep convolutional neural network, obtain the angle value ang_Net recognized by the deep convolutional neural network, and obtain the three-dimensional coordinates [x_Net y_Net z_Net] to which the angle value ang_Net belongs. ;
步骤102、基于预先设置的仪表正面图和待识别的仪表倾斜图，选择至少三个特征点进行旋转识别方式，获取仪表正面图旋转至仪表倾斜图的状态时的旋转角度(θx，θy，θz)和旋转矩阵(rotMX，rotMY，rotMZ)；Step 102. Based on the preset front view of the instrument and the tilt chart of the instrument to be recognized, select at least three feature points for rotation recognition, and obtain the rotation angle (θx , θy) when the front view of the instrument is rotated to the state of the tilt chart of the instrument. , θz ) and rotation matrix (rotMX , rotMY , rotMZ );
步骤103、根据所述角度值一ang_Net所属的三维坐标[x_Net y_Net z_Net]、仪表正面图旋转至仪表倾斜图的状态时的旋转角度(θx，θy，θz)和旋转矩阵(rotMX，rotMY，rotMZ)，获取仪表倾斜图中仪表指针处于正面时的角度值θpoint；Step 103. According to the three-dimensional coordinate [x_Net y_Net z_Net] to which ang_Net belongs, the rotation angle (θx , θy , θz ) and the rotation matrix (rotM , rotMY , rotMZ ), obtain the angle value θpoint when the instrument pointer is on the front side in the instrument tilt diagram;
步骤104、根据仪表正面图时仪表的基本参数和所述仪表指针处于正面时的角度值θpoint，获得待识别的仪表倾斜图中仪表读数R。Step 104: Obtain the instrument reading R in the tilt diagram of the instrument to be identified based on the basic parameters of the instrument when the instrument is in front view and the angle value θpoint when the instrument pointer is in the front view.
可选地，所述步骤104包括：Optionally, the step 104 includes:
根据下述公式获取所述仪表读数R；Obtain the instrument reading R according to the following formula;
其中，R是仪表读数，Vmin,Vmax分别为仪表的最小读数和最大读数，θmin,θmax分别为仪表最小读数和水平线夹角角度，仪表最大读数和水平线夹角角度，θpoint为仪表指针处于正面时的角度值即处于正面时与水平线夹角角度。Among them, R is the instrument reading, Vmin and Vmax are the minimum and maximum readings of the instrument respectively, θmin and θmax are respectively the angle between the minimum reading of the instrument and the horizontal line, the maximum reading of the instrument and the angle between the horizontal line, θpoint is The angle value when the instrument pointer is facing forward is the angle between the meter pointer and the horizontal line.
可选地，所述步骤103包括：Optionally, the step 103 includes:
根据下述公式获取仪表指针处于正面时的角度值θpoint；Obtain the angle value θpoint when the instrument pointer is at the front according to the following formula;
front_x＝xn*rotMx*rotMY*rotMZfront_x＝xn *rotMx *rotMY *rotMZ
front_y＝yn*rotMx*rotMY*rotMZfront_y＝yn *rotMx *rotMY *rotMZ
front_z＝0front_z=0
其中，仪表倾斜图中倾斜仪表的指针顶点处于正面时的坐标front_x,front_y,front_z；Among them, the coordinates front_x, front_y, front_z when the pointer vertex of the tilted instrument is in the front direction in the instrument tilt diagram;
[x_Net y_Net z_Net]是仪表倾斜图中倾斜仪表指针顶点的三维坐标，[xn yn zn]是指针顶点在旋转斜面上面的坐标；[x_Net y_Net z_Net] is the three-dimensional coordinates of the pointer vertex of the tilted instrument in the instrument tilt diagram, [xn yn zn ] is the coordinates of the pointer vertex on the rotation slope;
所述旋转斜面为旋转角度(θx，θy，θz)所构成的平面；The rotation slope is a plane formed by rotation angles (θx , θy , θz );
nl是通过[x_Net y_Net z_Net]得到的旋转斜面的法线。nl is the normal of the rotation slope obtained through [x_Net y_Net z_Net].
可选地，所述步骤103包括：Optionally, the step 103 includes:
根据下述公式获取旋转面的法线nl；Obtain the normal nl of the rotating surface according to the following formula;
up＝[0 1 0]up＝[0 1 0]
r＝[1 0 0]r＝[1 0 0]
up_R＝rotMX×rotMY×rotMZ×upup_R＝rotMX ×rotMY ×rotMZ ×up
r_R＝rotMX×rotMY×rotMZ×rr_R＝rotMX ×rotMY ×rotMZ ×r
nl＝-(r_R×up_R)nl=-(r_R×up_R)
其中，up和r分别为仪表正面图的y轴上的正向单位坐标，x轴上的正向单位坐标，up_R和r_R分别是仪表倾斜图旋转后y轴上的正向单位坐标、x轴上的正向单位坐标。Among them, up and r are respectively the positive unit coordinates on the y-axis and the x-axis of the front view of the instrument. up_R and r_R are respectively the positive unit coordinates on the y-axis and the x-axis of the instrument tilt diagram after rotation. The forward unit coordinate on.
可选地，所述步骤101之前，所述方法还包括：Optionally, before step 101, the method further includes:
获取训练数据集，采用训练数据集训练深度神经网络模型；Obtain the training data set and use the training data set to train the deep neural network model;
训练数据集包括多个标注有特征点的仪表正面图和标注有对应仪表正面图中特征点的仪表倾斜图；The training data set includes multiple instrument front views marked with feature points and instrument tilt images marked with corresponding feature points in the instrument front view;
训练过程的损失函数为：The loss function of the training process is:
Loss＝λ1*Angle Loss+λ2*lbox Loss+λ3*lobj Loss+λ4*lcls Loss；Loss＝λ1*Angle Loss+λ2*lbox Loss+λ3*lobj Loss+λ4*lcls Loss;
其中，Angle Loss用于衡量深度神经网络模型预测的指针角度与实际指针角度之间的差异；lbox Loss则用于测量深度神经网络模型的位置误差；lobj Loss用于度量深度神经网络模型的置信度误差，lcls Loss用于衡量深度神经网络模型的类别误差；λ1，λ2，λ3，λ4分别为四个权重参数λ1，λ2，λ3，λ4；Among them, Angle Loss is used to measure the difference between the pointer angle predicted by the deep neural network model and the actual pointer angle; lbox Loss is used to measure the position error of the deep neural network model; lobj Loss is used to measure the confidence of the deep neural network model. Error, lcls Loss is used to measure the category error of the deep neural network model; λ1, λ2, λ3, and λ4 are the four weight parameters λ1, λ2, λ3, and λ4 respectively;
和/或，and / or,
采用训练数据集训练深度神经网络模型的训练过程包括：The training process of using a training data set to train a deep neural network model includes:
对每一图进行预处理，并将预处理后的640*640*3的仪表图像经过n次卷积，进行深度卷积和逐点卷积、深度可分离卷积处理，得到的是40*40*64大小的特征图，对40*40*64的特征图执行Softmax操作，得到特征图中每个像素点分别属于各个类别的概率，再经过argmax运算将概率转换成每个像素具体所属的类别，最后将40*40*64尺寸的特征图还原得到640*640*3大小的特征图，并预测输出是多个特征点的像素坐标，根据预测出的多个特征点的像素坐标，计算出指针基于图像底边的斜率，根据斜率获得指针角度。Preprocess each image, and pass the preprocessed 640*640*3 instrument image through n times of convolution, perform depth convolution, point-by-point convolution, and depth-separable convolution processing, and the result is 40* For a feature map of size 40*64, perform the Softmax operation on the feature map of 40*40*64 to obtain the probability that each pixel in the feature map belongs to each category, and then use the argmax operation to convert the probability into the specific value of each pixel. Category, finally restore the 40*40*64 size feature map to a 640*640*3 size feature map, and predict that the output is the pixel coordinates of multiple feature points. Based on the predicted pixel coordinates of multiple feature points, calculate The out pointer is based on the slope of the bottom edge of the image, and the pointer angle is obtained based on the slope.
可选地，所述步骤101包括：Optionally, the step 101 includes:
根据下述公式获取角度值一ang_Net所属的三维坐标Obtain the angle value - the three-dimensional coordinates of ang_Net according to the following formula
[x_Net y_Net z_Net]；[x_Net y_Net z_Net];
z_Net＝-1，z_Net表示仪表倾斜图是仪表正面图通过旋转后的投影生成的，z＝0表示仪表正面图。z_Net=-1, z_Net indicates that the instrument tilt diagram is generated by rotating the projection of the instrument front view, z=0 indicates the instrument front view.
可选地，所述步骤102包括：Optionally, the step 102 includes:
步骤102-1、对仪表正面图和仪表倾斜图分别标注对应的三个特征点；针对正面图像中特征点的二维坐标值和仪表倾斜图中特征点的坐标，基于下述公式获取各自的第一角度值；Step 102-1. Mark the corresponding three feature points on the front view of the instrument and the tilt chart of the instrument respectively; for the two-dimensional coordinate values of the feature points in the front image and the coordinates of the feature points in the tilt chart of the instrument, obtain their respective coordinates based on the following formula first angle value;
其中，ang是特征点的角度值，x，y是特征点的坐标值；Among them, ang is the angle value of the feature point, x, y are the coordinate values of the feature point;
步骤102-2、根据各自的第一角度值，基于下述公式获取仪表正面图和仪表倾斜图中每一特征点的三维坐标；Step 102-2: Obtain the three-dimensional coordinates of each feature point in the instrument front view and the instrument tilt diagram based on the respective first angle values and the following formula;
z＝0 (公式3)；z=0 (Formula 3);
步骤102-3、基于仪表正面图中每一特征点的三维坐标和仪表倾斜图中每一特征点的三维坐标，获取仪表正面图至仪表倾斜图的旋转矩阵(rotMX，rotMY，rotMZ)；Step 102-3. Based on the three-dimensional coordinates of each feature point in the instrument front view and the three-dimensional coordinates of each feature point in the instrument tilt diagram, obtain the rotation matrix (rotMX , rotMY , rotMZ from the instrument front view to the instrument tilt diagram) );
其中：θx，θy，θz是从(0～180°)进行遍历取值的。Among them: θx , θy , θz are traversed from (0 to 180°).
可选地，所述步骤102还包括：Optionally, step 102 also includes:
采用遍历方式获取θx，θy，θz；具体包括：Use traversal method to obtain θx , θy , θz ; details include:
根据下述公式5求出仪表正面图中三个特征点旋转后的坐标(x_R、y_R、z_R)，把旋转后的三个坐标点投影到一个z＝0的平面上，通过公式6计算出这三个特征点在z＝0的平面上的角度rotAng1，rotAng2，rotAng3；Calculate the rotated coordinates (x_R, y_R, z_R) of the three characteristic points in the front view of the instrument according to the following formula 5, project the three rotated coordinate points onto a plane with z=0, and calculate by formula 6 The angles rotAng1, rotAng2, rotAng3 of these three feature points on the plane z=0;
其中rotAng1对应(x_R1,y_R1)，rotAng2对应(x_R2,y_R2)，rotAng3对应(x_R3,y_R3)；Among them, rotAng1 corresponds to (x_R1, y_R1), rotAng2 corresponds to (x_R2, y_R2), and rotAng3 corresponds to (x_R3, y_R3);
如果旋转之后的三个特征点的角度都与仪表倾斜图上的三个特征点的第一角度值在误差范围内，则得到三个旋转角；If the angles of the three feature points after rotation are within the error range with the first angle values of the three feature points on the instrument tilt diagram, then three rotation angles are obtained;
即：(x1,y1,z1)-》(x_R1,y_R1,z_R1)，That is: (x1,y1,z1)-》(x_R1,y_R1,z_R1),
(x2,y2,z2)-》(x_R2,y_R2,z_R2)，(x2,y2,z2)-》(x_R2,y_R2,z_R2),
(x3,y3,z3)-》(x_R3,y_R3,z_R3)，(x3,y3,z3)-》(x_R3,y_R3,z_R3),
R1，R2，R3分别表示旋转的三个特征点，x_R1，y_R1,z_R1分别表示仪表正面图的三个特征点分别旋转后的坐标x轴坐标，y轴坐标和z轴坐标，R1, R2, and R3 respectively represent the three characteristic points of rotation, x_R1, y_R1, and z_R1 respectively represent the coordinates of the three characteristic points of the front view of the instrument after rotation, x-axis coordinate, y-axis coordinate, and z-axis coordinate.
x_R＝x*rotMz*rotMY*rotMXx_R＝x*rotMz *rotMY *rotMX
y_R＝y*rotMz*rotMY*rotMX (公式5)y_R＝y*rotMz *rotMY *rotMX (Formula 5)
z_R＝z*rotMz*rotMY*rotMXz_R＝z*rotMz *rotMY *rotMX
公式5中，x_R,y_R,z_R分别代表仪表正面图的特征点通过旋转后的x坐标，y坐标，z坐标，x,y,z分别表示仪表正面图的特征点坐标，rotMz,rotMy,rotMx分别表示绕z轴的旋转矩阵，绕y轴的旋转矩阵，绕x轴的旋转矩阵；In Formula 5, x_R, y_R, and z_R respectively represent the x coordinate, y coordinate, and z coordinate of the characteristic point in the front view of the instrument after rotation. x, y, z respectively represent the coordinates of the characteristic point in the front view of the instrument, rotMz, rotMy, rotMx. Respectively represent the rotation matrix around the z-axis, the rotation matrix around the y-axis, and the rotation matrix around the x-axis;
θ-thr＜rotAng＜θ+thr (公式7)θ-thr＜rotAng＜θ+thr (Formula 7)
如果ang4-thr<rotAng1<ang4+thr,ang5-thr<rotAng2<ang5+thr,ang6-thr<rotAng3<ang6+thr，则此时的θx，θy，θz就是最终得到的三个旋转角。If ang4-thr<rotAng1<ang4+thr, ang5-thr<rotAng2<ang5+thr, ang6-thr<rotAng3<ang6+thr, then θx , θy , θz at this time are the three final rotations horn.
第二方面，本发明实施例还提供一种电子设备，包括：存储器和处理器，所述存储器中存储有计算机程序，所述处理器执行所述存储器中存储的计算机程序，并执行第一方面任一所述的一种倾斜指针式仪表自动读数方法的步骤。In a second aspect, embodiments of the present invention further provide an electronic device, including: a memory and a processor. A computer program is stored in the memory. The processor executes the computer program stored in the memory and executes the first aspect. The steps of any one of the automatic reading methods of a tilted pointer instrument.
(三)有益效果(3) Beneficial effects
本发明的目的是针对仪表倾斜的问题，对现有的深度卷积神经网络进行改进，通过叠加特征提取层并将旋转角度嵌入到网络的损失函数中的方法提升识别的精度，利用改进后的深度卷积神经网络进行指针提取，然后再利用本发明的倾斜矫正算法然后进行三维图像还原，计算出三维下的旋转角度，实现三维图像的还原，解决在倾斜状态下的仪表识别问题，提高了识别精度。The purpose of this invention is to improve the existing deep convolutional neural network to solve the problem of instrument tilt, improve the accuracy of recognition by overlaying feature extraction layers and embedding the rotation angle into the loss function of the network, and using the improved The deep convolutional neural network performs pointer extraction, and then uses the tilt correction algorithm of the present invention to restore the three-dimensional image, calculate the rotation angle in three dimensions, realize the restoration of the three-dimensional image, solve the problem of instrument identification in the tilted state, and improve the Recognition accuracy.
附图说明Description of drawings
图1为本发明一实施例提供的训练数据集中任一标注的倾斜仪表的实例图；Figure 1 is an example diagram of any labeled tilt meter in the training data set provided by an embodiment of the present invention;
图2a为仪表正面图中标注特征点的示意图，Figure 2a is a schematic diagram of the characteristic points marked in the front view of the instrument.
图2b为仪表倾斜图中标注特征点的示意图；Figure 2b is a schematic diagram of the characteristic points marked in the instrument tilt diagram;
图3a为仪表正面图中三个特征点角度及横轴的示意图；Figure 3a is a schematic diagram of the angles and horizontal axes of the three characteristic points in the front view of the instrument;
图3b为仪表倾斜图中三个特征点角度及横轴的示意图；Figure 3b is a schematic diagram of the three characteristic point angles and the horizontal axis in the instrument tilt diagram;
图4为本发明一实施例提供的一种倾斜指针式仪表自动读数方法的流程示意图。FIG. 4 is a schematic flowchart of an automatic reading method for a tilted pointer meter provided by an embodiment of the present invention.
具体实施方式Detailed ways
为了更好的解释本发明，以便于理解，下面结合附图，通过具体实施方式，对本发明作详细描述。In order to better explain the present invention and facilitate understanding, the present invention will be described in detail below through specific embodiments in conjunction with the accompanying drawings.
实施例一Embodiment 1
如图1所示，图1示出了本发明一实施例提供的一种倾斜指针式仪表自动读数方法的流程示意图，本发明的方法的执行主体为控制设备，本实施例的方法可包括下述的步骤：As shown in Figure 1, Figure 1 shows a schematic flow chart of an automatic reading method for a tilted pointer instrument provided by an embodiment of the present invention. The execution subject of the method of the present invention is a control device. The method of this embodiment may include the following The steps described:
步骤101、基于预先训练的深度卷积神经网络对待识别的仪表倾斜图进行识别，获得深度卷积神经网络识别输出的角度值一ang_Net，以及获取角度值一ang_Net所属的三维坐标[x_Net y_Net z_Net]；Step 101. Recognize the instrument tilt map to be recognized based on the pre-trained deep convolutional neural network, obtain the angle value ang_Net recognized by the deep convolutional neural network, and obtain the three-dimensional coordinates [x_Net y_Net z_Net] to which the angle value ang_Net belongs. ;
步骤102、基于预先设置的仪表正面图和待识别的仪表倾斜图，以选择至少三个特征点进行旋转识别方式，获取仪表正面图旋转至仪表倾斜图的状态时的旋转角度(θx，θy，θz)和旋转矩阵(rotMX，rotMY，rotMZ)；Step 102: Based on the preset front view of the instrument and the tilt chart of the instrument to be recognized, select at least three feature points for rotation recognition, and obtain the rotation angle (θx , θ when the front view of the instrument is rotated to the state of the tilt chart of the instrument)y , θz ) and rotation matrix (rotMX , rotMY , rotMZ );
步骤103、根据所述角度值一ang_Net所属的三维坐标[x_Net y_Net z_Net]、仪表正面图旋转至仪表倾斜图的状态时的旋转角度(θx，θy，θz)和旋转矩阵(rotMX，rotMY，rotMZ)，获取仪表倾斜图中仪表指针处于正面时的角度值θpoint；Step 103. According to the three-dimensional coordinate [x_Net y_Net z_Net] to which ang_Net belongs, the rotation angle (θx , θy , θz ) and the rotation matrix (rotM , rotMY , rotMZ ), obtain the angle value θpoint when the instrument pointer is on the front side in the instrument tilt diagram;
步骤104、根据仪表正面图时仪表的基本参数和所述仪表指针处于正面时的角度值θpoint，获得待识别的仪表倾斜图中仪表读数R。Step 104: Obtain the instrument reading R in the tilt diagram of the instrument to be identified based on the basic parameters of the instrument when the instrument is in front view and the angle value θpoint when the instrument pointer is in the front view.
本实施例是针对仪表倾斜的问题，对现有的深度卷积神经网络进行改进，利用改进后的深度卷积神经网络进行指针提取，然后再利用选择至少三个特征点进行旋转识别方式进行三维图像还原，计算出三维下的旋转角度，实现三维图像的还原，解决在倾斜状态下的仪表识别问题。This embodiment aims to improve the existing deep convolutional neural network to solve the problem of instrument tilt. The improved deep convolutional neural network is used to extract pointers, and then the rotation recognition method is used to select at least three feature points for three-dimensional recognition. Image restoration, calculates the rotation angle in three dimensions, realizes the restoration of three-dimensional images, and solves the problem of instrument identification in tilted state.
实施例二Embodiment 2
结合图2至图4对本发明一实施例提供的一种倾斜指针式仪表自动读数方法进行详细说明。An automatic reading method for a tilted pointer type meter provided by an embodiment of the present invention will be described in detail with reference to FIGS. 2 to 4 .
201、对仪表正面图即正面图像(简称仪表正面图)和当前待识别仪表的倾斜图(简称仪表倾斜图)分别标注对应的三个特征点。201. Mark the corresponding three characteristic points respectively on the front view of the instrument, that is, the front image (referred to as the front view of the instrument) and the tilt image of the current instrument to be identified (referred to as the instrument tilt image).
具体地，预先在变电站用摄像头对仪表进行拍摄时，通过调整摄像头的预置位可以拍摄到仪表处于正面时的视频，在视频中提取图像，进而获得仪表的正面图。对仪表正面图以及仪表倾斜图分别进行三个特征点标注，这三个特征点是一一对应的关系，特征点的选取是任意的。如图2所示，图2a为仪表正面图，图2b为仪表倾斜图。Specifically, when the instrument is photographed with a camera in the substation in advance, the video of the instrument when it is in the front can be captured by adjusting the preset position of the camera, and the image is extracted from the video to obtain the front view of the instrument. Three feature points are marked on the front view of the instrument and the tilt view of the instrument. These three feature points are in a one-to-one correspondence, and the selection of the feature points is arbitrary. As shown in Figure 2, Figure 2a is the front view of the instrument, and Figure 2b is the tilt view of the instrument.
举例来说，本实施例的特征点的标注可通过labelme的工具进行的，该工具标注完之后会形成一个json文件，在这个文件中是这三个点的二维坐标(x,y)即像素坐标，也就是三个点的像素坐标，每一个特征点都会对应一个(x,y)。For example, the feature points in this embodiment can be annotated through the labelme tool. After the tool is annotated, a json file will be formed. In this file are the two-dimensional coordinates (x, y) of the three points, that is Pixel coordinates, that is, the pixel coordinates of three points, each feature point will correspond to an (x, y).
202、针对正面图像中特征点的二维坐标值和仪表倾斜图中特征点的坐标，获取各自的第一角度值。202. For the two-dimensional coordinate values of the feature points in the frontal image and the coordinates of the feature points in the instrument tilt diagram, obtain their respective first angle values.
在本实施例中，通过公式2对每个特征点求出其第一角度值ang1,ang2,ang3,ang4,ang5,ang6；如图3a,图3b所示，将仪表正面图的三个特征点标为a,b,c,对应的第一角度值为ang1,ang2,ang3，仪表倾斜图的三个特征点标为d,e,f,对应的第一角度值为ang4,ang5,ang6；In this embodiment, the first angle value ang1, ang2, ang3, ang4, ang5, ang6 is calculated for each feature point through Formula 2; as shown in Figure 3a and Figure 3b, the three features of the front view of the instrument are The points are marked a, b, c, and the corresponding first angle values are ang1, ang2, ang3. The three characteristic points of the instrument tilt diagram are marked d, e, f, and the corresponding first angle values are ang4, ang5, ang6. ;
仪表正面图的第一特征点a对应的第一角度值为ang1；The first angle value corresponding to the first feature point a in the front view of the instrument is ang1;
仪表正面图的第二特征点b对应的第一角度值为ang2；The first angle value corresponding to the second feature point b in the front view of the instrument is ang2;
仪表正面图的第三特征点c对应的第一角度值为ang3；The first angle value corresponding to the third characteristic point c in the front view of the instrument is ang3;
仪表倾斜图的第一特征点d对应的第一角度值为ang4；The first angle value corresponding to the first characteristic point d of the instrument tilt diagram is ang4;
仪表倾斜图的第二特征点e对应的第一角度值为ang5；The first angle value corresponding to the second characteristic point e of the instrument tilt diagram is ang5;
仪表倾斜图的第三特征点f对应的第一角度值为ang6.The first angle value corresponding to the third characteristic point f of the instrument tilt diagram is ang6.
thr为预设的误差。thr is the preset error.
ang1至ang6的计算公式均使用公式2的方式计算：The calculation formulas from ang1 to ang6 are calculated using Formula 2:
其中，ang是特征点的角度值，x，y是特征点的坐标值。此时会得到仪表正面图中三个特征点a,b,c的第一角度值以及仪表倾斜图中三个特征点d,e,f的第一角度值。Among them, ang is the angle value of the feature point, x, y are the coordinate values of the feature point. At this time, the first angle values of the three characteristic points a, b, and c in the front view of the instrument and the first angle values of the three characteristic points d, e, and f in the instrument tilt diagram will be obtained.
203、根据各自的第一角度值，获取仪表正面图和仪表倾斜图中每一特征点的三维坐标。203. Obtain the three-dimensional coordinates of each feature point in the instrument front view and instrument tilt diagram based on their respective first angle values.
实际应用中，这里可不需要求解倾斜仪表图的三个特征点的三维坐标，后续比较的时候都统一成角度进行比较了，故只需要求解倾斜仪表图的三个特征点的角度即可。In practical applications, there is no need to solve the three-dimensional coordinates of the three characteristic points of the tilted instrument diagram. In subsequent comparisons, they are all unified into angles for comparison. Therefore, it is only necessary to solve the angles of the three characteristic points of the tilted instrument diagram.
举例来说，通过以下公式3，求出仪表正面图中三个特征点a,b,c所对应的的三维坐标(x1,y1,z1)，(x2,y2,z2)，(x3,y3,z3).其中(x1,y1,z1)对应ang1,(x2,y2,z2)对应ang2,(x3,y3,z3)对应ang3,公式3是通用公式:For example, use the following formula 3 to find the three-dimensional coordinates (x1, y1, z1), (x2, y2, z2), (x3, y3) corresponding to the three feature points a, b, c in the front view of the instrument. , z3). Among them (x1, y1, z1) corresponds to ang1, (x2, y2, z2) corresponds to ang2, (x3, y3, z3) corresponds to ang3, and formula 3 is a general formula:
z＝0 (公式3)。z=0 (Formula 3).
204、基于仪表正面图中每一特征点的三维坐标和仪表倾斜图中每一特征点的三维坐标，获取仪表正面图至仪表倾斜图的旋转矩阵。204. Based on the three-dimensional coordinates of each feature point in the instrument front view and the three-dimensional coordinates of each feature point in the instrument tilt chart, obtain the rotation matrix from the instrument front view to the instrument tilt chart.
具体地，基于(x1,y1,z1)，(x2,y2,z2)，(x3,y3,z3)值，以z轴、y轴、x轴的顺序旋转三个特征点(旋转的是仪表正面图的三个特征点，通过旋转仪表正面图的三个特征点去逼近仪表倾斜图的特征点)，旋转矩阵如下所示：Specifically, based on the values of (x1, y1, z1), (x2, y2, z2), (x3, y3, z3), three feature points are rotated in the order of z-axis, y-axis, and x-axis (the instrument is rotated The three feature points of the front view are approximated by rotating the three feature points of the front view of the instrument to approximate the feature points of the instrument tilt view). The rotation matrix is as follows:
其中：θx，θy，θz是从(0～180°)进行遍历取值的。Among them: θx , θy , θz are traversed from (0 to 180°).
具体地，θx，θy，θz的遍历公式是通过下述方式实现的。Specifically, the traversal formulas of θx , θy , and θz are implemented in the following manner.
根据通用公式5可以求出仪表正面图中三个特征点旋转后的坐标(x_R、y_R、z_R)，把这三个点都投影到一个z＝0的平面上，通过公式6计算出这三个特征点在这个平面上的角度rotAng1，rotAng2，rotAng3，其中rotAng1对应(x_R1,y_R1)，rotAng2对应(x_R2,y_R2)，rotAng3对应(x_R3,y_R3)；如果旋转之后的三个特征点的角度都接近仪表倾斜图上的三个特征点的角度(公式2求解出了仪表正面图的三个特征点角度和仪表倾斜图的三个特征点的角度)，则就找到了需要的三个旋转角。According to the general formula 5, the rotated coordinates (x_R, y_R, z_R) of the three characteristic points in the front view of the instrument can be obtained. Project these three points onto a plane with z=0, and calculate these three points through formula 6. The angles of the feature points on this plane are rotAng1, rotAng2, and rotAng3, where rotAng1 corresponds to (x_R1, y_R1), rotAng2 corresponds to (x_R2, y_R2), and rotAng3 corresponds to (x_R3, y_R3); if the angles of the three feature points after rotation are all close to the angles of the three characteristic points on the instrument tilt diagram (Formula 2 solves the angles of the three characteristic points of the instrument front view and the angles of the three characteristic points of the instrument tilt diagram), then the three required rotations are found horn.
即：Right now:
(x1,y1,z1)-》(x_R1,y_R1,z_R1)，(x1,y1,z1)-》(x_R1,y_R1,z_R1),
(x2,y2,z2)-》(x_R2,y_R2,z_R2)，(x2,y2,z2)-》(x_R2,y_R2,z_R2),
(x3,y3,z3)-》(x_R3,y_R3,z_R3)，(x3,y3,z3)-》(x_R3,y_R3,z_R3),
上述式中，R1,R2，R3中R代表旋转，1，2，3是用来标记是第几个特征点，x_R1，y_R1,z_R1代表的是仪表正面图的三个特征点分别旋转后的坐标x轴坐标，y轴坐标和z轴坐标，1代表第一个特征点，2代表第二个特征点，3代表第三个特征点)In the above formula, R in R1, R2, and R3 represents rotation, 1, 2, and 3 are used to mark which feature point it is. x_R1, y_R1, z_R1 represent the three feature points of the front view of the instrument after rotation. Coordinates x-axis coordinate, y-axis coordinate and z-axis coordinate, 1 represents the first feature point, 2 represents the second feature point, 3 represents the third feature point)
x_R＝x*rotMz*rotMY*rotMXx_R＝x*rotMz *rotMY *rotMX
y_R＝y*rotMz*rotMY*rotMX (公式5)y_R＝y*rotMz *rotMY *rotMX (Formula 5)
z_R＝z*rotMz*rotMY*rotMXz_R＝z*rotMz *rotMY *rotMX
x_R,y_R,z_R分别代表仪表正面图的特征点通过旋转后的x坐标，y坐标，z坐标，x,y,z分别代表仪表正面图的特征点的x坐标，y坐标，z坐标，rotMz,rotMy,rotMx分别代表绕z轴的旋转矩阵，绕y轴的旋转矩阵，绕x轴的旋转矩阵。x_R, y_R, z_R respectively represent the x-coordinate, y-coordinate, and z-coordinate of the characteristic point of the front view of the instrument after rotation. ,rotMy,rotMx respectively represent the rotation matrix around the z-axis, the rotation matrix around the y-axis, and the rotation matrix around the x-axis.
把这三个点都投影到一个z＝0的平面上，通过公式6计算出这三个特征点在这个平面上的角度rotAng1，rotAng2，rotAng3，其中rotAng1对应(x_R1,y_R1)，rotAng2对应(x_R2,y_R2)，rotAng3对应(x_R3,y_R3)。如果旋转之后的三个特征点的角度都接近仪表倾斜图上的三个特征点的角度，则就找到了需要的三个旋转角:Project these three points onto a plane with z=0, and calculate the angles rotAng1, rotAng2, rotAng3 of these three feature points on this plane through formula 6, where rotAng1 corresponds to (x_R1, y_R1), and rotAng2 corresponds to ( x_R2,y_R2), rotAng3 corresponds to (x_R3,y_R3). If the angles of the three feature points after rotation are close to the angles of the three feature points on the instrument tilt diagram, then the three required rotation angles have been found:
θ-thr＜rotAng＜θ+thr (公式7)θ-thr＜rotAng＜θ+thr (Formula 7)
如果ang4-thr<rotAng1<ang4+thr,ang5-thr<rotAng2<ang5+thr,ang6-thr<rotAng3<ang6+thr，则此时的θx，θy，θz就是最终得到的三个旋转角。If ang4-thr<rotAng1<ang4+thr, ang5-thr<rotAng2<ang5+thr, ang6-thr<rotAng3<ang6+thr, then θx , θy , θz at this time are the three final rotations horn.
205、通过改进的深度卷积神经网络识别出倾斜仪表中指针的角度值后，设神经网络识别出的角度为ang_Net,通过公式8计算出其对应的三维坐标x_Net，y_Net,其中z_Net＝-1。205. After identifying the angle value of the pointer in the tilt meter through the improved deep convolutional neural network, let the angle recognized by the neural network be ang_Net, and calculate its corresponding three-dimensional coordinates x_Net, y_Net through Formula 8, where z_Net=-1 .
也就是说，将仪表倾斜图送到神经网络进行识别，识别结果是仪表倾斜图中指针的角度，根据这个角度可以计算出仪表倾斜图的指针的三维坐标。That is to say, the instrument tilt diagram is sent to the neural network for identification. The recognition result is the angle of the pointer in the instrument tilt diagram. Based on this angle, the three-dimensional coordinates of the pointer of the instrument tilt diagram can be calculated.
z_Net＝-1z_Net＝-1
本实施例的仪表倾斜图是因为摄像机拍摄时并不是正对仪表，导致最终拍摄得到的仪表图时倾斜仪表图，所以这个倾斜仪表图本质上是有一个对应的仪表正面图的，该部分提及的旋转后的投影指的就是这个倾斜仪表图所对应的仪表正面图通过旋转后会得到倾斜仪表图，当仪表正面图的时候，这个仪表z轴方向的坐标对应的是z＝0,因为它没有发生任何的倾斜(旋转)，而倾斜仪表图是仪表正面图发生旋转之后得到的，那么此时倾斜仪表图对应的z轴方向的坐标就一定不是z＝0了，所以这里假设等于-1，本质上是做了一种归一化。The instrument tilt diagram in this embodiment is because the camera is not facing the instrument when shooting, so the instrument diagram finally captured is tilted. Therefore, this tilted instrument diagram essentially has a corresponding instrument front view. This section provides The projection after rotation refers to the front view of the instrument corresponding to the tilted instrument diagram. After rotation, the tilted instrument diagram will be obtained. When the instrument is viewed from the front, the z-axis coordinate of the instrument corresponds to z=0, because It does not have any tilt (rotation), and the tilted instrument diagram is obtained after the front view of the instrument is rotated. Then the z-axis coordinate corresponding to the tilted instrument diagram at this time must not be z=0, so it is assumed here to be equal to - 1. It is essentially a kind of normalization.
仪表的正面图位于z＝0平面上，假设仪表是通过旋转后的投影生成的，则其生成在z＝-1平面上。因此，将其z_Net坐标设置为-1。通过之前推导出的三个旋转角度θx，θy，θz，可以逆向推导出旋转平面的法向量，根据旋转矩阵的性质，可以使用以下公式9来计算旋转面的法向量：The front view of the instrument is located on the z=0 plane. Assuming that the instrument is generated through a rotated projection, it is generated on the z=-1 plane. Therefore, set its z_Net coordinate to -1. Through the previously derived three rotation angles θx , θy , θz , the normal vector of the rotation plane can be deduced inversely. According to the properties of the rotation matrix, the following formula 9 can be used to calculate the normal vector of the rotation plane:
up＝[0 1 0]up＝[0 1 0]
r＝[1 0 0]r＝[1 0 0]
up_R＝rotMX×rotMY×rotMZ×upup_R＝rotMX ×rotMY ×rotMZ ×up
r_R＝rotMX×rotMY×rotMZ×rr_R＝rotMX ×rotMY ×rotMZ ×r
nl＝-(r_R×up_R) (公式9)nl=-(r_R×up_R) (Formula 9)
其中，up和r分别代表y轴上端一个单位的点，x轴上右端一个单位的点，up_R和r_R分别是旋转后的坐标，nl是旋转面的法线。Among them, up and r respectively represent the point one unit at the upper end of the y-axis, and the point one unit at the right end of the x-axis. up_R and r_R are the coordinates after rotation, and nl is the normal of the rotation surface.
206、倾斜仪表中指针的三维坐标以及旋转面的法线，求出指针在旋转斜面上的坐标，公式10如下所示：206. Using the three-dimensional coordinates of the pointer in the tilt instrument and the normal of the rotating surface, find the coordinates of the pointer on the rotating inclined surface. Formula 10 is as follows:
其中[x_Net y_Net z_Net]是倾斜仪表中指针的三维坐标，[xn yn zn]是指针在旋转斜面上面的坐标。Where [x_Net y_Net z_Net] is the three-dimensional coordinates of the pointer in the tilt meter, [xn yn zn ] is the coordinates of the pointer on the rotating inclined plane.
([x_Net y_Net z_Net]这个坐标是根据神经网络识别出的指针角度值计算出来的，旋转斜面是三个旋转角形成的平面，需要将[x_Net y_Net z_Net]这个坐标映射到旋转斜面上，之后再进行逆运算，得到处于正面时指针的坐标。([x_Net y_Net z_Net] This coordinate is calculated based on the pointer angle value recognized by the neural network. The rotation slope is a plane formed by three rotation angles. The coordinates [x_Net y_Net z_Net] need to be mapped to the rotation slope, and then Perform the inverse operation to obtain the coordinates of the pointer when it is on the front.
207、基于指针在旋转斜面上的坐标，将旋转矩阵以及旋转角度进行逆向计算得到倾斜仪表中的指针处于正面时的坐标207. Based on the coordinates of the pointer on the rotating inclined plane, reversely calculate the rotation matrix and rotation angle to obtain the coordinates of the pointer in the tilted instrument when it is facing forward.
front_x,front_y,front_z，front_x,front_y,front_z,
如公式11所示，其中旋转矩阵rotMX，rotMY，rotMZ中的角度就是上面公式7得到的θx，θy，θz：Asshown in Formula 11,the angles inthe rotation matricesrotM
(rotMX，rotMY，rotMZ时旋转矩阵，θx，θy，θz时旋转角度，旋转矩阵需要用到旋转角度进行计算)(rotM____
front_x＝xn*rotMx*rotMY*rotMZfront_x＝xn *rotMx *rotMY *rotMZ
front_y＝yn*rotMx*rotMY*rotMZfront_y＝yn *rotMx *rotMY *rotMZ
front_z＝0 (公式11)front_z=0 (Formula 11)
因为是正面时的坐标，所以front_z＝0,根据公式12可以求出指针处于正面时的角度值。Because it is the coordinate when the pointer is facing forward, front_z=0. According to Formula 12, the angle value when the pointer is facing forward can be calculated.
208、通过以上计算，就可以得到倾斜仪表中指针在正面时的坐标和对应的角度值。最后，通过角度法以公式13计算出仪表读数。208. Through the above calculation, you can get the coordinates and corresponding angle value of the pointer in the tilt meter when it is on the front. Finally, the meter reading is calculated using the angle method using Equation 13.
其中，R是读数结果，Vmin,Vmax分别代表仪表的最小读数和最大读数，θmin,θmax分别为仪表最小读数和水平线夹角角度，仪表最大读数和水平线夹角角度。Among them, R is the reading result, Vmin and Vmax respectively represent the minimum reading and the maximum reading of the instrument, θmin and θmax are respectively the angle between the minimum reading of the instrument and the horizontal line, and the angle between the maximum reading of the instrument and the horizontal line.
这里需要说明的是，旋转斜面指的是三个旋转角θx，θy，θz所构成的平面，通过旋转正面图像的三个特征点去逼近倾斜图像的特征点的结果是产生三个旋转角度θx，θy，θz，这三个角度可以构成一个平面，称之为旋转斜面，需要先将倾斜仪表指针的三维坐标转换成在旋转斜面上的坐标之后，然后再进行旋转矩阵和旋转角度的逆向计算得到这个倾斜仪表的指针在正面时的坐标，这个坐标值指的是顶点的坐标，本实施例中提到的所有指针的坐标都是指的顶点。What needs tobe explained here is that the rotation slope refers to the plane formed bythe three rotation anglesθ Rotation angles θx , θy , θz , these three angles can form a plane, called a rotation slope. It is necessary to first convert the three-dimensional coordinates of the tilt instrument pointer into coordinates on the rotation slope, and then perform the rotation matrix The inverse calculation of the angle of rotation obtains the coordinates of the pointer of the tilt meter when it is facing forward. This coordinate value refers to the coordinates of the vertex. The coordinates of all pointers mentioned in this embodiment refer to the vertex.
上述给出的是利用倾斜矫正算法然后进行三维图像还原，计算出三维下的旋转角度，实现三维图像的还原，解决在倾斜状态下的仪表识别问题，提高了识别精度。What is given above is to use the tilt correction algorithm and then restore the three-dimensional image, calculate the rotation angle in three dimensions, realize the restoration of the three-dimensional image, solve the problem of instrument identification in the tilted state, and improve the identification accuracy.
实施例三Embodiment 3
本实施例对实施例一和实施例二中提及的深度卷积神经网络进行训练的过程进行说明，本实施的深度卷积神经网络可为改进的一种深度卷积神经网络。This embodiment describes the training process of the deep convolutional neural network mentioned in Embodiment 1 and 2. The deep convolutional neural network in this implementation may be an improved deep convolutional neural network.
本实施例主要包含两个部分，一是对使用改进的深度卷积神经网络仪表识别网络进行训练的改进，二是使用网络训练出的模型进行基于三维空间的倾斜矫正与指针读数计算。This embodiment mainly includes two parts. One is the improvement of training using the improved deep convolutional neural network instrument recognition network, and the other is using the model trained by the network to perform tilt correction and pointer reading calculation based on three-dimensional space.
训练阶段：Training phase:
1.获取训练数据集。使用变电站的摄像头可以对仪表图像进行视频采集，将采集到的视频转换为图像，转换图像时以视频帧为单位，一个视频帧就是一张图像，最终得到N张图像。对这N张图像使用labelme标注软件进行标注，标注样例如图1所示，最终将这些标注后的N张图像作为数据集。1. Obtain the training data set. The camera of the substation can be used to collect video of the instrument image, and convert the collected video into an image. When converting the image, the video frame is used as the unit. One video frame is one image, and N images are finally obtained. Use labelme annotation software to annotate these N images. The annotation example is shown in Figure 1. Finally, these annotated N images are used as a data set.
2.划分训练数据集。将这N个被检测对象样本进行数据集划分，0.8N个样本划分为训练集，用于训练阶段；0.2N个样本划分为测试集，用于预测阶段。2. Divide the training data set. The N detected object samples are divided into data sets, 0.8N samples are divided into training sets, used in the training phase; 0.2N samples are divided into test sets, used in the prediction phase.
3.将这0.8N个样本送入到改进的深度卷积神经网络进行训练。3. Send these 0.8N samples to the improved deep convolutional neural network for training.
具体过程为，将预处理后的640*640*3的仪表图像经过1次卷积，得到尺寸为320*320*16的特征图；将该320*320*16的特征图再进行一次卷积，变成160*160*32的特征图；将该160*160*32的特征图在进行一次卷积，变成80*80*64的特征图；将该80*80*64的特征图在进行一次卷积，变成40*40*64的特征图；再对40*40*64的特征图进行下采样，变成20*20*64的特征图，再对其进行深度可分离卷积处理。The specific process is as follows: Convolve the preprocessed 640*640*3 instrument image once to obtain a feature map with a size of 320*320*16; perform another convolution on the 320*320*16 feature map. , becomes a feature map of 160*160*32; perform a convolution on the 160*160*32 feature map, and become a feature map of 80*80*64; convert the 80*80*64 feature map into Perform a convolution to become a 40*40*64 feature map; then downsample the 40*40*64 feature map to become a 20*20*64 feature map, and then perform depth separable convolution on it deal with.
深度可分离卷积分为两个过程，分别为深度卷积和逐点卷积。深度卷积的过程为首先对输入的20*20*64的特征图，使用64个大小为3*3*1的卷积核，在深度卷积中，每个卷积核只对输入的一个通道进行卷积操作，生成一个深度为1的特征图。因此，对于每个输入通道，会有一个对应的深度为1的特征图。Depthwise separable convolution is divided into two processes, namely depthwise convolution and point-wise convolution. The process of depth convolution is to first use 64 convolution kernels with a size of 3*3*1 for the input 20*20*64 feature map. In depth convolution, each convolution kernel only applies one of the input The channel performs a convolution operation to generate a feature map with a depth of 1. Therefore, for each input channel, there will be a corresponding feature map with depth 1.
在深度卷积操作后，输出特征图的尺寸将变为20*20*64。接下来，对深度卷积的结果进行逐点卷积操作。逐点卷积是指使用1x1的卷积核对深度卷积的结果进行卷积操作。这样可以将每个深度为1的特征图转换为一个深度为64的特征图。在逐点卷积操作后，得到的仍然是20*20*64的特征图；对20*20*64的特征图进行下采样，变成10*10*64的特征图，再对其进行深度可分离卷积处理，深度可分离卷积处理的过程仍然是在深度卷积的过程中首先对输入的10*10*64的特征图，使用64个大小为3*3*1的卷积核，在深度卷积中，每个卷积核只对输入的一个通道进行卷积操作，生成一个深度为1的特征图。因此，对于每个输入通道，会有一个对应的深度为1的特征图。在深度卷积操作后，输出特征图的尺寸将变为10*10*64。接下来，对深度卷积的结果进行逐点卷积操作。逐点卷积是指使用1x1的卷积核对深度卷积的结果进行卷积操作。这样可以将每个深度为1的特征图转换为一个深度为64的特征图。在逐点卷积操作后，得到的仍然是10*10*64的特征图；对10*10*64的特征图进行下采样，变成5*5*64的特征图，再对其进行深度可分离卷积处理，处理的方法为在深度卷积的过程中首先对输入的5*5*64的特征图，使用64个大小为3*3*1的卷积核，在深度卷积中，每个卷积核只对输入的一个通道进行卷积操作，生成一个深度为1的特征图。因此，对于每个输入通道，会有一个对应的深度为1的特征图。在深度卷积操作后，输出特征图的尺寸将变为5*5*64。接下来，对深度卷积的结果进行逐点卷积操作。逐点卷积是指使用1x1的卷积核对深度卷积的结果进行卷积操作。这样可以将每个深度为1的特征图转换为一个深度为64的特征图。在逐点卷积操作后，得到的仍然是5*5*64的特征图；对5*5*64的特征图进行下采样，变成3*3*64的特征图，再对其进行深度可分离卷积处理，处理的过程为在深度卷积的过程中首先对输入的3*3*64的特征图，使用64个大小为3*3*1的卷积核，在深度卷积中，每个卷积核只对输入的一个通道进行卷积操作，生成一个深度为1的特征图。因此，对于每个输入通道，会有一个对应的深度为1的特征图。在深度卷积操作后，输出特征图的尺寸将变为3*3*64。接下来，对深度卷积的结果进行逐点卷积操作。逐点卷积是指使用1x1的卷积核对深度卷积的结果进行卷积操作。这样可以将每个深度为1的特征图转换为一个深度为64的特征图。在逐点卷积操作后，得到的仍然是3*3*64的特征图；对3*3*64的特征图进行下采样，变成2*2*64的特征图，再对其进行深度可分离卷积处理，处理的过程为在深度卷积的过程中首先对输入的2*2*64的特征图，使用64个大小为3*3*1的卷积核，在深度卷积中，每个卷积核只对输入的一个通道进行卷积操作，生成一个深度为1的特征图。因此，对于每个输入通道，会有一个对应的深度为1的特征图。在深度卷积操作后，输出特征图的尺寸将变为2*2*64。接下来，对深度卷积的结果进行逐点卷积操作。逐点卷积是指使用1x1的卷积核对深度卷积的结果进行卷积操作。这样可以将每个深度为1的特征图转换为一个深度为64的特征图。在逐点卷积操作后，得到的仍然是2*2*64的特征图；对2*2*64的特征图进行上采样，从而得到4*4*64的特征图；这里由于需要提取浅层网络的特征，因此将该4*4*64与上面得到的3*3*64特征图进行路由连接；经过上采样生成5*5大小的特征图之后，与浅层的5*5*64的特征图进行路由连接；此后再次进行上采样生成10*10的特征图，并与浅层的10*10*64的特征图进行路由连接；此后再次进行上采样生成20*20的特征图，并与浅层的20*20*64的特征图进行路由连接；此后再次进行上采样生成40*40的特征图，并与浅层的40*40*64的特征图进行路由连接；最终得到的是40*40*64大小的特征图。After the depth convolution operation, the size of the output feature map will become 20*20*64. Next, a point-wise convolution operation is performed on the results of depth convolution. Pointwise convolution refers to using a 1x1 convolution kernel to perform a convolution operation on the results of depth convolution. This converts each depth-1 feature map into a depth-64 feature map. After the point-by-point convolution operation, the obtained feature map is still 20*20*64; the 20*20*64 feature map is downsampled to become a 10*10*64 feature map, and then the depth is Separable convolution processing, the process of depth separable convolution processing is still in the depth convolution process, first use 64 convolution kernels of size 3*3*1 for the input 10*10*64 feature map , In depth convolution, each convolution kernel only performs a convolution operation on one channel of the input to generate a feature map with a depth of 1. Therefore, for each input channel, there will be a corresponding feature map with depth 1. After the depth convolution operation, the size of the output feature map will become 10*10*64. Next, a point-wise convolution operation is performed on the results of depth convolution. Pointwise convolution refers to using a 1x1 convolution kernel to perform a convolution operation on the results of depth convolution. This converts each depth-1 feature map into a depth-64 feature map. After the point-by-point convolution operation, the obtained feature map is still 10*10*64; the 10*10*64 feature map is down-sampled into a 5*5*64 feature map, and then the depth is Separable convolution processing, the processing method is to first use 64 convolution kernels of size 3*3*1 for the input 5*5*64 feature map in the depth convolution process. , each convolution kernel only performs a convolution operation on one channel of the input, generating a feature map with a depth of 1. Therefore, for each input channel, there will be a corresponding feature map with depth 1. After the depth convolution operation, the size of the output feature map will become 5*5*64. Next, a point-wise convolution operation is performed on the results of depth convolution. Pointwise convolution refers to using a 1x1 convolution kernel to perform a convolution operation on the results of depth convolution. This converts each depth-1 feature map into a depth-64 feature map. After the point-by-point convolution operation, the obtained feature map is still 5*5*64; the 5*5*64 feature map is down-sampled into a 3*3*64 feature map, and then the depth is Separable convolution processing, the processing process is to first use 64 convolution kernels of size 3*3*1 for the input 3*3*64 feature map in the depth convolution process. , each convolution kernel only performs a convolution operation on one channel of the input, generating a feature map with a depth of 1. Therefore, for each input channel, there will be a corresponding feature map with depth 1. After the depth convolution operation, the size of the output feature map will become 3*3*64. Next, a point-wise convolution operation is performed on the results of depth convolution. Pointwise convolution refers to using a 1x1 convolution kernel to perform a convolution operation on the results of depth convolution. This converts each depth-1 feature map into a depth-64 feature map. After the point-by-point convolution operation, the obtained feature map is still a 3*3*64 feature map; the 3*3*64 feature map is downsampled into a 2*2*64 feature map, and then the depth is Separable convolution processing, the processing process is to first use 64 convolution kernels of size 3*3*1 for the input 2*2*64 feature map in the depth convolution process. , each convolution kernel only performs a convolution operation on one channel of the input, generating a feature map with a depth of 1. Therefore, for each input channel, there will be a corresponding feature map with depth 1. After the depth convolution operation, the size of the output feature map will become 2*2*64. Next, a point-wise convolution operation is performed on the results of the depth convolution. Pointwise convolution refers to using a 1x1 convolution kernel to perform a convolution operation on the results of depth convolution. This converts each depth-1 feature map into a depth-64 feature map. After the point-by-point convolution operation, the feature map obtained is still 2*2*64; the 2*2*64 feature map is upsampled to obtain a 4*4*64 feature map; here, due to the need to extract shallow The characteristics of the layer network, so the 4*4*64 is routed and connected with the 3*3*64 feature map obtained above; after upsampling to generate a 5*5 size feature map, it is connected with the shallow 5*5*64 The feature map is routed and connected; thereafter, upsampling is performed again to generate a 10*10 feature map, and routed and connected with the shallow 10*10*64 feature map; thereafter, upsampling is performed again to generate a 20*20 feature map. And routing connection with the shallow 20*20*64 feature map; then upsampling again to generate a 40*40 feature map, and routing connection with the shallow 40*40*64 feature map; finally obtained It is a feature map of 40*40*64 size.
5.对40*40*64的特征图执行Softmax操作，得到特征图中每个像素点分别属于各个类别的概率，再经过argmax运算将概率转换成每个像素具体所属的类别，最后将40*40*64尺寸的特征图还原为输入图像的640*640*3的尺寸大小，方法如下：对40*40*64的特征图采用双线性插值的方式进行上采样，将尺寸扩大到640*640，此时的输出为640*640*64的特征图，接着使用1x1的卷积核来减少通道数，将640*640*64的特征图转换为640*640*3大小的特征图。5. Perform the Softmax operation on the 40*40*64 feature map to obtain the probability that each pixel in the feature map belongs to each category, and then convert the probability into the specific category of each pixel through the argmax operation, and finally convert 40* The 40*64 size feature map is restored to the 640*640*3 size of the input image. The method is as follows: use bilinear interpolation to upsample the 40*40*64 feature map and expand the size to 640* 640, the output at this time is a feature map of 640*640*64, and then a 1x1 convolution kernel is used to reduce the number of channels, and the 640*640*64 feature map is converted into a feature map of 640*640*3 size.
由于样本标注的是三个点，则网络最终的预测输出是三个点的像素坐标，根据预测出的三个点的像素坐标，计算出指针基于图像底边的斜率，根据斜率求出角度，这个角度就是指针角度。Since the sample is marked with three points, the final prediction output of the network is the pixel coordinates of the three points. Based on the predicted pixel coordinates of the three points, the slope of the pointer based on the bottom edge of the image is calculated, and the angle is calculated based on the slope. This angle is the pointer angle.
为了评估算法的拟合能力，需要将仪表指针的角度作为惩罚项加入到损失函数中，以强制模型更好地学习角度信息。综合角度信息和lbox损失，最终得到的损失函数如下所示。In order to evaluate the fitting ability of the algorithm, the angle of the instrument pointer needs to be added to the loss function as a penalty term to force the model to better learn angle information. Combining the angle information and lbox loss, the final loss function is as follows.
Loss＝λ1*Angle Loss+λ2*lbox Loss+λ3*lobj Loss+λ4*lcls Loss (公式1)Loss＝λ1*Angle Loss+λ2*lbox Loss+λ3*lobj Loss+λ4*lcls Loss (Formula 1)
其中，Angle Loss用于衡量模型预测的指针角度与实际指针角度之间的差异。这个损失函数是基于余弦相似度定义的，因为余弦相似度能够捕捉角度之间的相似性，同时也对模型预测的精度进行了惩罚。lbox Loss则用于测量模型的位置误差，它可以通过计算模型预测框的中心点和实际框的中心点之间的距离来计算。lobj Loss用于度量模型的置信度误差，它可以通过计算模型预测的置信度值和实际置信度值之间的误差来计算。最后，lcls Loss用于衡量模型的类别误差，它可以通过计算模型预测的类别标签和实际标签之间的差异来计算。为了平衡这些不同的损失函数的影响，需要使用四个权重参数λ1，λ2，λ3，λ4。这些参数将根据实际的数据和任务进行调整，以获得最佳的结果。例如，在某些情况下，类别误差可能比位置误差更重要，因此可以增加λ4的值以放大类别误差的影响。训练过程中，模型将不断地调整自己的参数，以尽可能地减小损失函数的值。Among them, Angle Loss is used to measure the difference between the pointer angle predicted by the model and the actual pointer angle. This loss function is defined based on cosine similarity, which captures the similarity between angles while also penalizing the accuracy of model predictions. lbox Loss is used to measure the position error of the model. It can be calculated by calculating the distance between the center point of the model predicted box and the center point of the actual box. lobj Loss is used to measure the confidence error of the model, which can be calculated by calculating the error between the confidence value predicted by the model and the actual confidence value. Finally, lcls Loss is used to measure the class error of the model, which can be calculated by calculating the difference between the class labels predicted by the model and the actual labels. In order to balance the influence of these different loss functions, four weight parameters λ1, λ2, λ3, λ4 need to be used. These parameters will be adjusted based on the actual data and task to obtain the best results. For example, in some cases, the category error may be more important than the position error, so the value of λ4 can be increased to amplify the impact of the category error. During the training process, the model will continuously adjust its parameters to reduce the value of the loss function as much as possible.
使用阶段Use stage
通过改进的深度卷积神经网络识别出仪表倾斜图中倾斜仪表的指针的角度值ang_Net。The angle value of the pointer of the tilted instrument in the instrument tilt diagram is identified through an improved deep convolutional neural network ang_Net.
实施例四Embodiment 4
本发明实施例还提供一种电子设备，所述电子设备可连接仪表的监控设备，所述电子设备包括：存储器和处理器，所述存储器中存储指令，所述处理器执行所述存储器中存储的指令，并具体执行上述任意实施例所述的一种倾斜指针式仪表自动读数方法的步骤。An embodiment of the present invention also provides an electronic device that can be connected to a monitoring device of an instrument. The electronic device includes: a memory and a processor. The memory stores instructions, and the processor executes the instructions stored in the memory. instructions, and specifically execute the steps of an automatic reading method for a tilted pointer instrument described in any of the above embodiments.
为了更好的理解上述技术方案，下面将参照附图更详细地描述本发明的示例性实施例。虽然附图中显示了本发明的示例性实施例，然而应当理解，可以以各种形式实现本发明而不应被这里阐述的实施例所限制。相反，提供这些实施例是为了能够更清楚、透彻地理解本发明，并且能够将本发明的范围完整的传达给本领域的技术人员。In order to better understand the above technical solutions, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that the present invention may be understood more clearly and thoroughly, and the scope of the present invention may be fully conveyed to those skilled in the art.
本领域内的技术人员应明白，本发明的实施例可提供为方法、系统或计算机程序产品。因此，本发明可采用完全硬件实施例、完全软件实施例，或结合软件和硬件方面的实施例的形式。而且，本发明可采用在一个或多个其中包含有计算机可用程序代码的计算机可用存储介质(包括但不限于磁盘存储器、CD-ROM、光学存储器等)上实施的计算机程序产品的形式。Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems or computer program products. Thus, the invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.
本发明是参照根据本发明实施例的方法、设备(系统)和计算机程序产品的流程图和/或方框图来描述的。应理解可由计算机程序指令实现流程图和/或方框图中的每一流程和/或方框、以及流程图和/或方框图中的流程和/或方框的结合。The invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each process and/or block in the flowchart illustrations and/or block diagrams, and combinations of processes and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions.
应当注意的是，在权利要求中，不应将位于括号之间的任何附图标记理解成对权利要求的限制。词语“包含”不排除存在未列在权利要求中的部件或步骤。位于部件之前的词语“一”或“一个”不排除存在多个这样的部件。本发明可以借助于包括有若干不同部件的硬件以及借助于适当编程的计算机来实现。在列举了若干装置的权利要求中，这些装置中的若干个可以是通过同一个硬件来具体体现。词语第一、第二、第三等的使用，仅是为了表述方便，而不表示任何顺序。可将这些词语理解为部件名称的一部分。It should be noted that in the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps other than those listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention may be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In the claim enumerating several means, several of these means may be embodied by the same hardware. The use of the words first, second, third, etc. is only for convenience of expression and does not indicate any order. These words can be understood as part of the component name.
此外，需要说明的是，在本说明书的描述中，术语“一个实施例”、“一些实施例”、“实施例”、“示例”、“具体示例”或“一些示例”等的描述，是指结合该实施例或示例描述的具体特征、结构、材料或者特点包含于本发明的至少一个实施例或示例中。在本说明书中，对上述术语的示意性表述不必须针对的是相同的实施例或示例。而且，描述的具体特征、结构、材料或者特点可以在任一个或多个实施例或示例中以合适的方式结合。此外，在不相互矛盾的情况下，本领域的技术人员可以将本说明书中描述的不同实施例或示例以及不同实施例或示例的特征进行结合和组合。In addition, it should be noted that in the description of this specification, the terms "one embodiment", "some embodiments", "embodiments", "examples", "specific examples" or "some examples" are used to describe It means that a specific feature, structure, material or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention. In this specification, the schematic expressions of the above terms are not necessarily directed to the same embodiment or example. Furthermore, the specific features, structures, materials or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, those skilled in the art may combine and combine different embodiments or examples and features of different embodiments or examples described in this specification unless they are inconsistent with each other.
尽管已描述了本发明的优选实施例，但本领域的技术人员在得知了基本创造性概念后，则可对这些实施例作出另外的变更和修改。所以，权利要求应该解释为包括优选实施例以及落入本发明范围的所有变更和修改。显然，本领域的技术人员可以对本发明进行各种修改和变型而不脱离本发明的精神和范围。这样，倘若本发明的这些修改和变型属于本发明权利要求及其等同技术的范围之内，则本发明也应该包含这些修改和变型在内。Although the preferred embodiments of the present invention have been described, those skilled in the art can make additional changes and modifications to these embodiments after learning the basic inventive concepts. Therefore, the claims should be construed to include the preferred embodiments and all changes and modifications that fall within the scope of the invention. Obviously, those skilled in the art can make various modifications and variations to the present invention without departing from the spirit and scope of the invention. In this way, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and equivalent technologies, the present invention should also include these modifications and variations.
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311181600.8ACN117315635A (en) | 2023-09-13 | 2023-09-13 | Automatic reading method for inclined pointer type instrument |
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311181600.8ACN117315635A (en) | 2023-09-13 | 2023-09-13 | Automatic reading method for inclined pointer type instrument |
| Publication Number | Publication Date |
|---|---|
| CN117315635Atrue CN117315635A (en) | 2023-12-29 |
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311181600.8APendingCN117315635A (en) | 2023-09-13 | 2023-09-13 | Automatic reading method for inclined pointer type instrument |
| Country | Link |
|---|---|
| CN (1) | CN117315635A (en) |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN118736583A (en)* | 2024-06-26 | 2024-10-01 | 北京智盟信通科技有限公司 | A pointer instrument reading recognition method based on deep learning |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN118736583A (en)* | 2024-06-26 | 2024-10-01 | 北京智盟信通科技有限公司 | A pointer instrument reading recognition method based on deep learning |
| Publication | Publication Date | Title |
|---|---|---|
| CN113269237B (en) | Assembly change detection method, device and medium based on attention mechanism | |
| US11222471B2 (en) | Implementing three-dimensional augmented reality in smart glasses based on two-dimensional data | |
| CN107953329B (en) | Object recognition and attitude estimation method, device and robotic arm grasping system | |
| CN108648194B (en) | Method and device for 3D target recognition, segmentation and pose measurement based on CAD model | |
| CN101394573B (en) | A method and system for generating panoramas based on feature matching | |
| CN114863007B (en) | Image rendering method, device and electronic equipment for three-dimensional object | |
| US12380594B2 (en) | Pose estimation method and apparatus | |
| JP2020507850A (en) | Method, apparatus, equipment, and storage medium for determining the shape of an object in an image | |
| US12175691B1 (en) | Method and device for mapping three-dimensional (3D) point cloud model based on deep learning | |
| CN114119987B (en) | Feature extraction and descriptor generation method and system based on convolutional neural network | |
| CN115690542A (en) | A method for directional recognition of aerial insulators based on improved yolov5 | |
| CN109525786B (en) | Video processing method and device, terminal equipment and storage medium | |
| CN114766039B (en) | Object detection method, object detection device, terminal device, and medium | |
| CN111127556A (en) | Target object identification and pose estimation method and device based on 3D vision | |
| CN116524160B (en) | Product Consistency Auxiliary Verification System and Method Based on AR Recognition | |
| CN112053441A (en) | Full-automatic layout recovery method for indoor fisheye image | |
| CN108805056A (en) | A kind of monitoring camera-shooting face sample extending method based on 3D faceforms | |
| CN111399634A (en) | Gesture-guided object recognition method and device | |
| CN117173232A (en) | Depth image acquisition method, device and equipment | |
| CN115222810A (en) | Target pose estimation method and device, computing equipment and storage medium | |
| CN117315635A (en) | Automatic reading method for inclined pointer type instrument | |
| CN119672766A (en) | Method, storage medium, electronic device and product for detecting hand joints in eyewear equipment | |
| KR20120020711A (en) | Object recognition system and method the same | |
| CN119515980A (en) | A six-degree-of-freedom pose estimation method for objects based on deep learning | |
| CN114882106A (en) | Pose determination method and device, equipment and medium |
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |