技术领域Technical Field
本公开实施例涉及分布式被动传感器多目标关联匹配技术领域，尤其涉及一种分布式双红外传感器时序孪生网络多目标同一性判定方法。The disclosed embodiments relate to the technical field of distributed passive sensor multi-target association matching, and in particular to a method for determining multi-target identity of a distributed dual-infrared sensor time-series twin network.
背景技术Background Art
红外传感器是一种无源探测传感器，其工作原理是通过捕捉被探测物体的红外与热辐射强度对物体进行探测，在此过程中不主动发射电磁波，因此具有隐蔽性好，不易被探测目标察觉的优点。同时相较于雷达等主动传感器，红外传感器具有测角精度高(测角误差约0.001rad)，数据刷新率高(大于100Hz)，探测距离远(大于100km)的优点。随着无人机集群与协同作战技术以及红外小目标探测技术的发展，通过无人机搭载红外传感器实现对空中目标进行高精度探测跟踪成为研究热点。Infrared sensor is a passive detection sensor. Its working principle is to detect objects by capturing the infrared and thermal radiation intensity of the detected object. In this process, it does not actively emit electromagnetic waves, so it has the advantages of good concealment and is not easily detected by the detected target. At the same time, compared with active sensors such as radar, infrared sensors have the advantages of high angle measurement accuracy (angle measurement error is about 0.001rad), high data refresh rate (greater than 100Hz), and long detection distance (greater than 100km). With the development of drone clusters and collaborative combat technology and infrared small target detection technology, high-precision detection and tracking of aerial targets through drones equipped with infrared sensors has become a research hotspot.
然而，在使用单个无人机搭载单台红外传感器对目标进行探测时，仅能测得目标的红外辐射强度信息，以及被探测目标相对于传感器的观测方位角与俯仰角度值，无法精确获得传感器与目标之间的距离信息，因此单个无人机搭载红外传感器无法完成对空中机动目标的实时定位与跟踪任务。而通过两架采用分布式站位的搭载红外传感器的无人机从不同角度对同一个目标进行角度测量，再结合两架无人机自身位置与传感器姿态信息，通过三角定位方法就能够实时的解算被探测目标的位置信息，继而实现单目标的定位与跟踪任务，上述任务构成分布式双红外传感器对单目标的探测场景。对单目标的探测场景无需考虑两个传感器成像平面中唯一目标的关联匹配问题，只需应用定位算法即能实现对单个目标距离的解算，然而当双红外传感器的重叠探测视场内出现不少于一个目标时，双视图中多目标的关联错配将会导致交叉伪点的组合爆炸问题。所谓交叉伪点的组合爆炸问题是指在双红外传感器所探测的重叠视场内，若出现m个目标，将会有m2种关联关系，而其中只有一种关联关系是正确关联，错误的关联结果将会为后续多目标的定位与跟踪造成困难。However, when a single drone equipped with a single infrared sensor is used to detect a target, only the infrared radiation intensity information of the target and the observed azimuth and pitch angle values of the detected target relative to the sensor can be measured, and the distance information between the sensor and the target cannot be accurately obtained. Therefore, a single drone equipped with an infrared sensor cannot complete the real-time positioning and tracking tasks of aerial maneuvering targets. However, by using two drones equipped with infrared sensors in distributed positions to measure the angle of the same target from different angles, and then combining the positions of the two drones themselves and the sensor attitude information, the position information of the detected target can be solved in real time through the triangulation positioning method, and then the positioning and tracking tasks of a single target are realized. The above tasks constitute the detection scenario of a distributed dual infrared sensor for a single target. For the detection scenario of a single target, there is no need to consider the association matching problem of the only target in the imaging plane of the two sensors. The distance of a single target can be solved by applying the positioning algorithm. However, when there is no less than one target in the overlapping detection field of view of the dual infrared sensors, the association mismatch of multiple targets in the dual views will lead to the combinatorial explosion problem of cross pseudo points. The so-called combinatorial explosion problem of cross-pseudo-points means that if there are m targets in the overlapping field of view detected by dual infrared sensors, there will bem2 kinds of associations, and only one of them is correct. The wrong association result will cause difficulties for the subsequent positioning and tracking of multiple targets.
发明内容Summary of the invention
为了避免现有技术的不足之处，本申请提供一种分布式双红外传感器时序孪生网络多目标同一性判定方法，用以解决现有技术中存在当双红外传感器的重叠探测视场内出现不少于一个目标时，双视图中多目标的关联错配将会导致交叉伪点的组合爆炸的问题。In order to avoid the shortcomings of the prior art, the present application provides a distributed dual-infrared sensor time-series twin network multi-target identity determination method to solve the problem in the prior art that when there is at least one target in the overlapping detection field of view of the dual infrared sensors, the association mismatch of multiple targets in the dual views will lead to a combinatorial explosion of cross-pseudo-points.
根据本公开实施例，提供一种分布式双红外传感器时序孪生网络多目标同一性判定方法，该方法包括：According to an embodiment of the present disclosure, a method for determining the identity of multiple targets in a distributed dual-infrared sensor time-series twin network is provided, the method comprising:
确定分布式双红外传感器中的基准传感器和非基准传感器，选取并固定基准坐标系；Determine the reference sensor and non-reference sensor in the distributed dual infrared sensor, and select and fix the reference coordinate system;
基于基准坐标系，根据预设时间段内同一目标在分布式双红外传感器的成像平面中所成像的时序点迹中所包含的像素坐标信息，利用插值法拟合同一目标形成的二维时序航迹，且将基准传感器的时间戳同步至非基准传感器，以实现基准传感器和非基准传感器的时间软同步；Based on the reference coordinate system, according to the pixel coordinate information contained in the time series point traces of the same target imaged in the imaging plane of the distributed dual infrared sensors within a preset time period, the two-dimensional time series track formed by the same target is fitted by the interpolation method, and the timestamp of the reference sensor is synchronized to the non-reference sensor to achieve time soft synchronization between the reference sensor and the non-reference sensor;
确定分布式红外传感器的成像平面中时序点迹与二维时序航迹的ID编号赋予原则，同时对分布式双红外传感器多目标同一性判定问题进行数学描述，建立并定义标准同一性评分矩阵和标准同一性判定矩阵；Determine the ID number assignment principle for the time series point traces and two-dimensional time series tracks in the imaging plane of the distributed infrared sensor, and mathematically describe the problem of multi-target identity determination of distributed dual infrared sensors, establish and define the standard identity scoring matrix and standard identity determination matrix;
基于标准同一性判定矩阵，构建时序孪生网络多目标同一性判定模型；其中，时序孪生网络多目标同一性判定模型包括特征提取网络和同一性判定度量网络，特征提取网络包括第一特征提取模块和第二特征提取模块，第一特征提取模块和第二特征提取模块的结构相同；Based on the standard identity determination matrix, a temporal twin network multi-objective identity determination model is constructed; wherein the temporal twin network multi-objective identity determination model includes a feature extraction network and an identity determination metric network, the feature extraction network includes a first feature extraction module and a second feature extraction module, and the first feature extraction module and the second feature extraction module have the same structure;
分别对分布式双红外传感器中的基准传感器的时序数据信息和非基准传感器获取的时序数据信息进行分析，以得到第一多维特征信息和第二多维特征信息；Respectively analyzing the time series data information of the reference sensor and the time series data information acquired by the non-reference sensor in the distributed dual infrared sensor to obtain first multi-dimensional feature information and second multi-dimensional feature information;
利用第一特征提取模块对第一多维特征信息进行特征提取，以得到第一多维特征，利用第二特征提取模块对第二多维特征信息进行特征提取，以得到第二多维特征；Using a first feature extraction module to perform feature extraction on the first multidimensional feature information to obtain a first multidimensional feature, and using a second feature extraction module to perform feature extraction on the second multidimensional feature information to obtain a second multidimensional feature;
将第一多维特征和第二多维特征输入至同一性判定度量网络中进行处理，以得到第一多维特征和第二多维特征的同一性判定概率，根据同一性判定概率构建目标同一性评分矩阵，并根据目标同一性评分矩阵构建目标同一性判定矩阵，得到同一性判定结果。The first multidimensional feature and the second multidimensional feature are input into the identity determination metric network for processing to obtain the identity determination probability of the first multidimensional feature and the second multidimensional feature, a target identity scoring matrix is constructed according to the identity determination probability, and a target identity determination matrix is constructed according to the target identity scoring matrix to obtain the identity determination result.
进一步的，基于基准坐标系，根据预设时间段内同一目标在分布式双红外传感器的成像平面中所成像的时序点迹中所包含的像素坐标信息，利用插值法拟合同一目标形成的二维时序航迹的步骤中，包括：Furthermore, based on the reference coordinate system, according to the pixel coordinate information contained in the time series point traces of the same target imaged in the imaging plane of the distributed dual infrared sensor within a preset time period, the step of fitting the two-dimensional time series track formed by the same target by using the interpolation method includes:
对于分布式双红外传感器成像平面中的时序点迹所表示的目标，将最初观测到目标的时刻记为0时刻，记录该时刻后连续个时刻目标的点迹坐标，通过三次样条插值法拟合得到目标在分布式双红外传感器中的二维时序航迹；For the target represented by the time series points in the imaging plane of the distributed dual infrared sensor , the target is initially observed The moment is recorded as time 0, and the continuous Moment Target The point coordinates of the target are obtained by fitting the cubic spline interpolation method. Two-dimensional time-series trajectory in distributed dual infrared sensors;
重复上述步骤，分别以分布式双红外传感器中不同时序点迹被观测到的时刻为0时刻并开始记录其后连续个时刻自身的像素坐标值，以拟合出预设时间段内不同成像点迹在分布式双红外传感器中形成的二维时序航迹。Repeat the above steps, taking the time when the distributed dual infrared sensors are observed at different time sequence points as time 0 and starting to record the subsequent continuous The pixel coordinate value of each moment is used to fit the two-dimensional time series track formed by different imaging points in the distributed dual infrared sensors within a preset time period.
进一步的，确定分布式红外传感器的成像平面中时序点迹与二维时序航迹的ID编号赋予原则，同时对分布式双红外传感器多目标同一性判定问题进行数学描述，建立并定义标准同一性评分矩阵和标准同一性判定矩阵的步骤中，包括：Furthermore, the ID number assignment principle of the time series point traces and the two-dimensional time series track in the imaging plane of the distributed infrared sensor is determined, and the problem of multi-target identity determination of the distributed dual infrared sensor is mathematically described. The steps of establishing and defining the standard identity scoring matrix and the standard identity determination matrix include:
针对分布式双红外传感器在连续一段时间段内分别观测到个和个二维时序航迹数据的情况，对多目标同一性判定问题进行数学描述，以构建多目标的标准同一性评分矩阵，标准同一性评分矩阵内各位置数值的大小表征此位置行号对应ID编号代表的二维时序航迹与列号对应ID编号代表的二维时序航迹被判定为同一目标的概率值，概率值大小位于0~1之间；The distributed dual infrared sensors observed and Based on the situation of two-dimensional time series track data, the problem of multi-target identity determination is mathematically described to construct a standard identity scoring matrix for multiple targets. The value of each position in the standard identity scoring matrix represents the probability value that the two-dimensional time series track represented by the ID number corresponding to the row number of this position and the two-dimensional time series track represented by the ID number corresponding to the column number are determined to be the same target, and the probability value is between 0 and 1.
结合标准同一性评分矩阵中数值大小定义标准同一性判定矩阵，标准同一性判定矩阵内元素由0和1组成，将标准同一评分矩阵中同维度概率值最大的元素所在的位置定义为1，其余位置定义为0，标准同一性判定矩阵中每个行或列有且仅有一个1，表明此位置行号对应ID编号代表的二维时序航迹与列号对应ID编号代表的二维时序航迹被判定为同一目标；其中，标准同一性判定矩阵中等于1的元素个数即分布式双红外传感器所探测目标的总个数。The standard identity determination matrix is defined in combination with the numerical values in the standard identity scoring matrix. The elements in the standard identity determination matrix are composed of 0 and 1. The position of the element with the largest probability value of the same dimension in the standard identity scoring matrix is defined as 1, and the remaining positions are defined as 0. Each row or column in the standard identity determination matrix has only one 1, indicating that the two-dimensional time series track represented by the ID number corresponding to the row number at this position and the two-dimensional time series track represented by the ID number corresponding to the column number are judged to be the same target; among them, the number of elements equal to 1 in the standard identity determination matrix is the total number of targets detected by the distributed dual infrared sensors.
进一步的，确定分布式红外传感器的成像平面中时序点迹与二维时序航迹的ID编号赋予原则，同时对分布式双红外传感器多目标同一性判定问题进行数学描述，建立并定义标准同一性评分矩阵和标准同一性判定矩阵的步骤中，还包括：Furthermore, the ID number assignment principle of the time series point traces and the two-dimensional time series track in the imaging plane of the distributed infrared sensor is determined, and the problem of multi-target identity determination of the distributed dual infrared sensor is mathematically described. The steps of establishing and defining the standard identity scoring matrix and the standard identity determination matrix also include:
对于同一个目标在基准传感器或非基准传感器的成像平面中形成的二维时序航迹，记录开始探测到此目标的时刻及该时刻单帧图像中表征该目标的时序点迹数据的像素坐标大小，以作为ID编号依据；For the two-dimensional time-series track formed by the same target in the imaging plane of the reference sensor or the non-reference sensor, the time when the target is detected and the pixel coordinate size of the time-series point track data representing the target in the single-frame image at that time are recorded as the basis for the ID number;
比较所有被记录的时序点迹的像素横坐标大小，以#1开始编号，从小到大依次递增1，若像素横坐标相同，则比较像素纵坐标大小并从小开始编号；其中，对于已有ID编号的二维时序航迹，其ID编号固定，对于新探测到的目标，记录其进入成像平面的时刻，并计算此时刻之前的目标个数，在此个数上加1给新探测到的目标赋予ID编号。Compare the pixel horizontal coordinates of all recorded time-series points, start numbering with #1, and increase by 1 from small to large. If the pixel horizontal coordinates are the same, compare the pixel vertical coordinates and start numbering from small. Among them, for the two-dimensional time-series tracks with existing ID numbers, their ID numbers are fixed. For newly detected targets, record the time when they enter the imaging plane, calculate the number of targets before this time, and add 1 to this number to assign an ID number to the newly detected target.
进一步的，将第一多维特征和第二多维特征输入至同一性判定度量网络中进行处理，以得到第一多维特征和第二多维特征的同一性判定概率，根据同一性判定概率构建目标同一性评分矩阵，并根据目标同一性评分矩阵构建目标同一性判定矩阵，得到同一性判定结果的步骤中，包括：Further, the first multidimensional feature and the second multidimensional feature are input into the identity determination metric network for processing to obtain the identity determination probability of the first multidimensional feature and the second multidimensional feature, a target identity scoring matrix is constructed according to the identity determination probability, and a target identity determination matrix is constructed according to the target identity scoring matrix to obtain the step of obtaining the identity determination result, including:
通过同一性判定度量网络中的卷积层对特征提取网络提取的第一多维特征和第二多维特征进行处理；Processing the first multidimensional features and the second multidimensional features extracted by the feature extraction network through a convolutional layer in the identity determination metric network;
采用同一性判定度量网络中的全局均值池化对处理后的第一多维特征和第二多维特征进行二次处理，以减少模型参数，缓解模型的过拟合问题；The global mean pooling in the identity determination metric network is used to perform secondary processing on the processed first multidimensional features and the second multidimensional features to reduce model parameters and alleviate the overfitting problem of the model;
利用同一性判定度量网络中的sigmoid激活函数将同一性判定概率映射至0~1之间；The sigmoid activation function in the identity determination metric network is used to map the identity determination probability to between 0 and 1;
基于二次处理后的第一多维特征和第二多维特征，及不同编号的二维时序航迹间的同一性判定概率构建目标同一性评分矩阵；A target identity scoring matrix is constructed based on the first multidimensional features and the second multidimensional features after secondary processing and the identity judgment probability between two-dimensional time-series tracks with different numbers;
通过比较目标同一性评分矩阵同维度元素大小定义目标同一性判定矩阵，并输出目标同一性判定矩阵。The target identity judgment matrix is defined by comparing the sizes of elements of the same dimension of the target identity score matrix, and the target identity judgment matrix is output.
进一步的，该方法还包括：Furthermore, the method also includes:
通过仿真环境或实际场景获取分布式双红外传感器观测多目标的训练集图像，对训练集图像内的二维时序航迹赋予ID编号，同时对训练集图像进行处理和标注，以得到训练集图像的输出真值；Obtain training set images of distributed dual infrared sensors observing multiple targets through simulation environments or actual scenes, assign ID numbers to the two-dimensional time-series tracks in the training set images, and process and annotate the training set images to obtain the output true values of the training set images;
将训练集图像与所标注的输出真值作为训练样本训练时序孪生网络多目标同一性判定模型，直至时序孪生网络多目标同一性判定模型收敛。The training set images and the annotated output true values are used as training samples to train the temporal twin network multi-target identity determination model until the temporal twin network multi-target identity determination model converges.
进一步的，特征提取网络中的第一特征提取模块和第二特征提取模块共享权值和参数。Furthermore, the first feature extraction module and the second feature extraction module in the feature extraction network share weights and parameters.
进一步的，基准传感器的时序数据信息中至少包括：Furthermore, the time series data information of the reference sensor includes at least:
时序点迹的中心像素坐标信息、分布式双红外传感器自身位置信息、传感器相对于时序目标的观测角信息、分布式双红外传感器之间的相对位置信息、时序点迹在各个成像平面中的红外辐射强度信息、时间信息、各个时序点迹的编号ID信息、同一成像目标的连续运动信息、分布式双红外传感器在基准坐标系下的连续运动信息、红外传感器相对于同一目标的观测角连续变化信息、目标的红外辐射强度变化信息以及多个目标形成的二维时序航迹的编号ID信息。The central pixel coordinate information of the time series point traces, the position information of the distributed dual infrared sensors themselves, the observation angle information of the sensors relative to the time series targets, the relative position information between the distributed dual infrared sensors, the infrared radiation intensity information of the time series point traces in each imaging plane, the time information, the numbering ID information of each time series point trace, the continuous motion information of the same imaging target, the continuous motion information of the distributed dual infrared sensors in the reference coordinate system, the continuous change information of the observation angle of the infrared sensor relative to the same target, the change information of the infrared radiation intensity of the target and the numbering ID information of the two-dimensional time series track formed by multiple targets.
本公开的实施例提供的技术方案可以包括以下有益效果：The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:
本公开的实施例中，通过上述分布式双红外传感器时序孪生网络多目标同一性判定方法，一方面，充分挖掘到红外传感器成像平面内多目标成像二维时序航迹信息中所隐含的表征同一性判定特性的多维特征，并通过同一性判定度量网络能够准确的对多目标实现关联匹配，解决了仅考虑几何特征时由于目标数量增多所造成的交叉伪点组合爆炸问题，尤其适用于分布式双红外传感器探测视场内具有多个目标的应用场景。另一方面，设计的时序孪生网络多目标同一性判定模型中的特征提取网络由两个结构和参数完全相同的第一特征提取模块和第二特征提取模块组成，第一特征提取模块和第二特征提取模块基于具有较快前向推断速度的时序神经处理网络构建，将分布式双红外传感器实测的一对时序数据信息分别输入至训练好的模型的第一特征提取模块和第二特征提取模块中，整个时序孪生网络同一性判定模型能够快速推断出输入的一对实测的时序数据信息的同一性判定结果，具有较好的实用性。In the embodiment of the present disclosure, through the above-mentioned distributed dual infrared sensor time series twin network multi-target identity determination method, on the one hand, the multi-dimensional features characterizing the identity determination characteristics implicit in the two-dimensional time series track information of multi-target imaging in the infrared sensor imaging plane are fully excavated, and the multi-target can be accurately matched through the identity determination metric network, which solves the problem of cross-pseudo-point combination explosion caused by the increase in the number of targets when only geometric features are considered, and is particularly suitable for distributed dual infrared sensors to detect application scenarios with multiple targets in the field of view. On the other hand, the feature extraction network in the designed time series twin network multi-target identity determination model consists of two first feature extraction modules and second feature extraction modules with exactly the same structure and parameters. The first feature extraction module and the second feature extraction module are constructed based on a time series neural processing network with a faster forward inference speed. A pair of time series data information measured by the distributed dual infrared sensor is respectively input into the first feature extraction module and the second feature extraction module of the trained model. The entire time series twin network identity determination model can quickly infer the identity determination result of the input pair of measured time series data information, and has good practicality.
附图说明BRIEF DESCRIPTION OF THE DRAWINGS
此处的附图被并入说明书中并构成本说明书的一部分，示出了符合本公开的实施例，并与说明书一起用于解释本公开的原理。显而易见地，下面描述中的附图仅仅是本公开的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.
图1示出本公开示例性实施例中分布式双红外传感器时序孪生网络多目标同一性判定方法的步骤图；FIG1 is a step diagram showing a method for determining the identity of multiple targets in a distributed dual-infrared sensor temporal twin network in an exemplary embodiment of the present disclosure;
图2示出本公开示例性实施例中两台机载平台分别搭载红外传感器构成分布式站位并探测远距离的多个时序目标的场景示意图；FIG2 is a schematic diagram showing a scenario in which two airborne platforms are respectively equipped with infrared sensors to form a distributed station and detect multiple time-sequential targets at a long distance in an exemplary embodiment of the present disclosure;
图3示出本公开示例性实施例中选择的基准坐标系与基准传感器示意图；FIG3 is a schematic diagram showing a reference coordinate system and a reference sensor selected in an exemplary embodiment of the present disclosure;
图4示出本公开示例性实施例中红外传感器成像平面内对单帧图像中多目标成像时序数据信息的ID编号赋予原则示意图；FIG4 is a schematic diagram showing the principle of assigning ID numbers to multi-target imaging time-series data information in a single frame image within an infrared sensor imaging plane in an exemplary embodiment of the present disclosure;
图5示出本公开示例性实施例中红外传感器成像平面内对连续帧图像中多目标成像时序数据信息的ID编号赋予原则示意图；FIG5 is a schematic diagram showing the principle of assigning ID numbers to multi-target imaging time-series data information in continuous frame images within an infrared sensor imaging plane in an exemplary embodiment of the present disclosure;
图6示出本公开示例性实施例中分布式双红外传感器多目标同一性判定问题描述示意图；FIG6 is a schematic diagram showing a description of the problem of multi-target identity determination using a distributed dual infrared sensor in an exemplary embodiment of the present disclosure;
图7示出本公开示例性实施例中第一特征提取模块和第二特征提取模块的具体结构原理示意图；FIG7 is a schematic diagram showing the specific structural principles of the first feature extraction module and the second feature extraction module in an exemplary embodiment of the present disclosure;
图8示出本公开示例性实施例中以N-BEATS作为特征提取网络和同一性判定度量网络构建的时序孪生网络多目标同一性判定模型完成分布式双红外传感器多目标同一性判定任务的整体流程图。Figure 8 shows the overall flow chart of the distributed dual infrared sensor multi-target identity determination task using a temporal twin network multi-target identity determination model constructed using N-BEATS as the feature extraction network and the identity determination metric network in an exemplary embodiment of the present disclosure.
具体实施方式DETAILED DESCRIPTION
现在将参考附图更全面地描述示例实施方式。然而，示例实施方式能够以多种形式实施，且不应被理解为限于在此阐述的范例；相反，提供这些实施方式使得本公开将更加全面和完整，并将示例实施方式的构思全面地传达给本领域的技术人员。所描述的特征、结构或特性可以以任何合适的方式结合在一个或更多实施方式中。Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.
此外，附图仅为本公开实施例的示意性图解，并非一定是按比例绘制。图中相同的附图标记表示相同或类似的部分，因而将省略对它们的重复描述。附图中所示的一些方框图是功能实体，不一定必须与物理或逻辑上独立的实体相对应。In addition, the accompanying drawings are only schematic illustrations of the embodiments of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated descriptions will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities.
本示例实施方式中提供了一种分布式双红外传感器时序孪生网络多目标同一性判定方法。参考图1中所示，该分布式双红外传感器时序孪生网络多目标同一性判定方法可以包括：步骤S101~步骤S107。In this example implementation, a method for determining the identity of multiple targets in a distributed dual-infrared sensor sequential twin network is provided. Referring to FIG1 , the method for determining the identity of multiple targets in a distributed dual-infrared sensor sequential twin network may include steps S101 to S107.
步骤S101：确定分布式双红外传感器中的基准传感器和非基准传感器，选取并固定基准坐标系；Step S101: determining a reference sensor and a non-reference sensor in a distributed dual infrared sensor, and selecting and fixing a reference coordinate system;
步骤S102：基于基准坐标系，根据预设时间段内同一目标在分布式双红外传感器的成像平面中所成像的时序点迹中所包含的像素坐标信息，利用插值法拟合同一目标形成的二维时序航迹，且将基准传感器的时间戳同步至非基准传感器，以实现基准传感器和非基准传感器的时间软同步；Step S102: Based on the reference coordinate system, according to the pixel coordinate information contained in the time series point traces of the same target imaged in the imaging plane of the distributed dual infrared sensors within a preset time period, the two-dimensional time series track formed by the same target is fitted by using the interpolation method, and the timestamp of the reference sensor is synchronized to the non-reference sensor, so as to realize the time soft synchronization of the reference sensor and the non-reference sensor;
步骤S103：确定分布式红外传感器的成像平面中时序点迹与二维时序航迹的ID编号赋予原则，同时对分布式双红外传感器多目标同一性判定问题进行数学描述，建立并定义标准同一性评分矩阵和标准同一性判定矩阵；Step S103: determining the ID number assignment principle for the time series point traces and the two-dimensional time series track in the imaging plane of the distributed infrared sensor, and mathematically describing the multi-target identity determination problem of the distributed dual infrared sensor, and establishing and defining the standard identity scoring matrix and the standard identity determination matrix;
步骤S104：基于标准同一性判定矩阵，构建时序孪生网络多目标同一性判定模型；其中，时序孪生网络多目标同一性判定模型包括特征提取网络和同一性判定度量网络，特征提取网络包括第一特征提取模块和第二特征提取模块，第一特征提取模块和第二特征提取模块的结构相同；Step S104: constructing a temporal twin network multi-objective identity determination model based on a standard identity determination matrix; wherein the temporal twin network multi-objective identity determination model includes a feature extraction network and an identity determination metric network, the feature extraction network includes a first feature extraction module and a second feature extraction module, and the first feature extraction module and the second feature extraction module have the same structure;
步骤S105：分别对分布式双红外传感器中的基准传感器的时序数据信息和非基准传感器获取的时序数据信息进行分析，以得到第一多维特征信息和第二多维特征信息；Step S105: analyzing the time series data information of the reference sensor and the time series data information acquired by the non-reference sensor in the distributed dual infrared sensor respectively to obtain first multi-dimensional feature information and second multi-dimensional feature information;
步骤S106：利用第一特征提取模块对第一多维特征信息进行特征提取，以得到第一多维特征，利用第二特征提取模块对第二多维特征信息进行特征提取，以得到第二多维特征；Step S106: using a first feature extraction module to perform feature extraction on the first multidimensional feature information to obtain a first multidimensional feature, and using a second feature extraction module to perform feature extraction on the second multidimensional feature information to obtain a second multidimensional feature;
步骤S107：将第一多维特征和第二多维特征输入至同一性判定度量网络中进行处理，以得到第一多维特征和第二多维特征的同一性判定概率，根据同一性判定概率构建目标同一性评分矩阵，并根据目标同一性评分矩阵构建目标同一性判定矩阵，得到同一性判定结果。Step S107: Input the first multidimensional feature and the second multidimensional feature into the identity determination metric network for processing to obtain the identity determination probability of the first multidimensional feature and the second multidimensional feature, construct a target identity scoring matrix according to the identity determination probability, and construct a target identity determination matrix according to the target identity scoring matrix to obtain the identity determination result.
通过上述分布式双红外传感器时序孪生网络多目标同一性判定方法，一方面，充分挖掘到红外传感器成像平面内多目标成像二维时序航迹信息中所隐含的表征同一性判定特性的多维特征，并通过同一性判定度量网络能够准确的对多目标实现关联匹配，解决了仅考虑几何特征时由于目标数量增多所造成的交叉伪点组合爆炸问题，尤其适用于分布式双红外传感器探测视场内具有多个目标的应用场景。另一方面，设计的时序孪生网络多目标同一性判定模型中的特征提取网络由两个结构和参数完全相同的第一特征提取模块和第二特征提取模块组成，第一特征提取模块和第二特征提取模块基于具有较快前向推断速度的时序神经处理网络构建，将分布式双红外传感器实测的一对时序数据信息分别输入至训练好的模型的第一特征提取模块和第二特征提取模块中，整个时序孪生网络同一性判定模型能够快速推断出输入的一对实测的时序数据信息的同一性判定结果，具有较好的实用性。Through the above-mentioned distributed dual infrared sensor time series twin network multi-target identity determination method, on the one hand, the multi-dimensional features that characterize the identity determination characteristics implicit in the two-dimensional time series track information of multi-target imaging in the infrared sensor imaging plane are fully excavated, and the multi-target can be accurately matched through the identity determination metric network, which solves the problem of cross-pseudo-point combination explosion caused by the increase in the number of targets when only geometric features are considered. It is particularly suitable for application scenarios where distributed dual infrared sensors have multiple targets in the detection field of view. On the other hand, the feature extraction network in the designed time series twin network multi-target identity determination model consists of two first feature extraction modules and second feature extraction modules with exactly the same structure and parameters. The first feature extraction module and the second feature extraction module are constructed based on a time series neural processing network with a faster forward inference speed. A pair of time series data information measured by the distributed dual infrared sensor is respectively input into the first feature extraction module and the second feature extraction module of the trained model. The entire time series twin network identity determination model can quickly infer the identity determination result of the input pair of measured time series data information, and has good practicality.
下面，将参考图1至图8对本示例实施方式中的上述分布式双红外传感器时序孪生网络多目标同一性判定方法的各个步骤进行更详细的说明。Below, the various steps of the above-mentioned distributed dual infrared sensor temporal twin network multi-target identity determination method in this example implementation will be described in more detail with reference to Figures 1 to 8.
在步骤S101中，选取分布式双红外传感器中某一个传感器作为基准传感器，选取并固定地心地固坐标系为基准坐标系；其中分布式双红外传感器为两台机载平台搭载的两台不同红外传感器，两台机载平台以分布式站位；基准传感器用于确定基准时间戳对双红外传感器进行时间同步，基准坐标系用于传感器位置与运动信息的计算。In step S101, one of the distributed dual infrared sensors is selected as the reference sensor, and the Earth-centered Earth-fixed coordinate system is selected and fixed as the reference coordinate system; the distributed dual infrared sensors are two different infrared sensors carried by two airborne platforms, and the two airborne platforms are in distributed positions; the reference sensor is used to determine the reference timestamp to synchronize the dual infrared sensors, and the reference coordinate system is used to calculate the sensor position and motion information.
具体的，选取地心地固坐标系作为基准坐标系，将惯性导航系统获取的机载平台所在位置的经纬度坐标信息通过坐标变换转换为基准坐标系坐标；Specifically, the Earth-centered Earth-fixed coordinate system is selected as the reference coordinate system, and the longitude and latitude coordinate information of the position of the airborne platform obtained by the inertial navigation system is converted into the coordinates of the reference coordinate system through coordinate transformation;
选取分布式双红外传感器中的传感器1作为基准传感器，以获得基准传感器时间戳，用于双红外传感器的时间同步，消除双红外传感器由于时间不同步所造成的观测误差对同一性结果的影响。Sensor 1 of the distributed dual infrared sensors is selected as the reference sensor to obtain the reference sensor timestamp for time synchronization of the dual infrared sensors, thereby eliminating the influence of the observation error caused by the time asynchrony of the dual infrared sensors on the identity result.
在步骤S102中，根据连续一段时间同一目标在双红外传感器成像平面中所成像的时序点迹中所包含的像素坐标信息，通过插值方法拟合同一目标形成的二维时序航迹，同时将基准传感器时间戳同步至非基准传感器，实现双红外传感器时间软同步；其中二维时序航迹由连续一段时间内的多个时序点迹数据拟合而成。In step S102, based on the pixel coordinate information contained in the time series point traces of the same target imaged in the imaging plane of the dual infrared sensors over a continuous period of time, a two-dimensional time series track formed by the same target is fitted by an interpolation method, and at the same time, the timestamp of the reference sensor is synchronized to the non-reference sensor to achieve soft time synchronization of the dual infrared sensors; wherein the two-dimensional time series track is fitted by multiple time series point trace data over a continuous period of time.
具体的，对于红外传感器成像平面中的时序点迹所表示的目标，将最初观测到该目标的时刻记为0时刻，记录该时刻后连续个时刻此目标的点迹坐标，通过三次样条插值方法拟合得到该点目标在双红外传感器中的二维时序航迹。Specifically, for the target represented by the time series points in the infrared sensor imaging plane , record the time when the target is first observed as time 0, and record the time after that. The point coordinates of the target at each moment are fitted by the cubic spline interpolation method to obtain the two-dimensional time series track of the point target in the dual infrared sensors.
重复上述步骤，分别以双红外传感器中不同时序点迹被观测到的时刻为0时刻并开始记录其后连续个时刻自身的像素坐标值，能够拟合出其后一段连续时间不同成像点迹在双红外传感器中形成的二维时序航迹。Repeat the above steps, taking the time when the different time sequence points in the dual infrared sensors are observed as time 0 and starting to record the subsequent continuous The pixel coordinate value of each moment can be used to fit the two-dimensional time series track formed by different imaging points in the dual infrared sensors in the subsequent continuous period of time.
在步骤S103中，确定红外传感器成像平面中时序点迹与二维时序航迹的ID编号赋予原则，同时对分布式双红外传感器多目标同一性判定问题进行数学描述，在此基础上建立双红外传感器多目标同一性评分矩阵与同一性判定矩阵表达式。In step S103, the ID number assignment principle for the time series point traces and the two-dimensional time series tracks in the infrared sensor imaging plane is determined, and the problem of multi-target identity determination of distributed dual infrared sensors is mathematically described. On this basis, the dual infrared sensor multi-target identity scoring matrix and identity determination matrix expression are established.
具体的，对于同一个目标在某一传感器成像平面中形成的二维时序航迹数据，记录开始探测到此目标的时刻及该时刻单帧图像中表征该目标的时序点迹数据的像素坐标大小，以作为ID编号依据。首先比较所有被记录的时序点迹数据的像素横坐标大小，从坐标较小的以#1开始编号，从小到大依次递增1，若遇到像素横坐标相同的情况，比较像素纵坐标大小并从较小的开始编号，对于已经有ID的时序航迹，其编号不再随后续因为运动造成的像素坐标的变化而变化，此后对于新探测到的目标，记录其进入传感器成像平面的时刻，并计算此时刻之前的目标个数，在此个数上加1给新探测到的目标赋予ID编号。Specifically, for the two-dimensional time-series track data formed by the same target in a certain sensor imaging plane, the moment when the target is detected and the pixel coordinate size of the time-series point track data representing the target in the single-frame image at that moment are recorded as the basis for the ID number. First, compare the horizontal coordinate sizes of the pixels of all the recorded time-series point track data, start numbering from the smaller coordinates with #1, and increase by 1 from small to large. If the horizontal coordinates of the pixels are the same, compare the vertical coordinate sizes of the pixels and start numbering from the smaller ones. For the time-series track that already has an ID, its number will no longer change with the subsequent changes in the pixel coordinates caused by movement. After that, for the newly detected target, record the moment when it enters the sensor imaging plane, and calculate the number of targets before this moment, and add 1 to this number to assign an ID number to the newly detected target.
针对分布式双红外传感器在连续一段时间段内分别观测到个和个二维时序航迹数据的情况，对此情景的多目标同一性判定问题进行数学描述，基于此数学描述构建多目标同一性评分矩阵，评分矩阵内各位置数值的大小表征此位置行号对应ID编号代表的二维时序航迹与列号对应ID编号代表的二维时序航迹被判定为同一目标的概率值，概率值大小位于0~1之间。The distributed dual infrared sensors observed and The problem of multi-target identity determination in this scenario is mathematically described based on the mathematical description. A multi-target identity scoring matrix is constructed based on the mathematical description. The value of each position in the scoring matrix represents the probability that the two-dimensional time series track represented by the ID number corresponding to the row number at this position and the two-dimensional time series track represented by the ID number corresponding to the column number are judged to be the same target, and the probability value is between 0 and 1.
结合同一性评分矩阵中数值大小定义同一性判定矩阵表达式，同一性判定矩阵内元素由0和1组成，将评分矩阵中同维度概率值最大的元素所在的位置定义为1，其余位置定义为0，同一性判定矩阵中每个行或列有且仅有一个1，表明此位置行号对应ID编号代表的二维时序航迹与列号对应ID编号代表的二维时序航迹被判定为同一目标。同一性判定矩阵中等于1的元素个数即双红外传感器所探测目标的总个数。The expression of the identity determination matrix is defined in combination with the numerical values in the identity scoring matrix. The elements in the identity determination matrix are composed of 0 and 1. The position of the element with the largest probability value of the same dimension in the scoring matrix is defined as 1, and the remaining positions are defined as 0. Each row or column in the identity determination matrix has and only has one 1, indicating that the two-dimensional time series track represented by the ID number corresponding to the row number at this position and the two-dimensional time series track represented by the ID number corresponding to the column number are judged to be the same target. The number of elements equal to 1 in the identity determination matrix is the total number of targets detected by the dual infrared sensors.
在步骤S104中，分析分布式双红外传感器获取的能够表征多目标同一性判定特征的时序数据信息，基于时序数据信息特点与实际应用场景构建时序孪生网络多目标同一性判定模型的特征提取网络，实现提取分布式双红外传感器获取的时序数据信息中所隐含的表征目标同一性判定特性的多维特征。构建时序孪生网络多目标同一性判定模型的同一性判定度量网络，以神经网络度量方式求得特征提取网络所提取的两个多维特征的同一性判定概率并构建目标同一性评分矩阵，在此基础上构建目标同一性判定矩阵作为模型的输出。In step S104, the time series data information acquired by the distributed dual infrared sensors that can characterize the characteristics of multi-target identity determination is analyzed, and a feature extraction network of the time series twin network multi-target identity determination model is constructed based on the characteristics of the time series data information and the actual application scenario, so as to extract the multi-dimensional features that characterize the characteristics of target identity determination implicit in the time series data information acquired by the distributed dual infrared sensors. The identity determination metric network of the time series twin network multi-target identity determination model is constructed, and the identity determination probability of the two multi-dimensional features extracted by the feature extraction network is obtained by the neural network metric method and the target identity scoring matrix is constructed. On this basis, the target identity determination matrix is constructed as the output of the model.
具体的，对红外传感器获取的时序数据信息进行分析，所获得的单帧图像中能够表征目标同一性判定特性的多维特征信息包括：时序点迹的中心像素坐标信息、分布式双红外传感器自身位置信息、传感器相对于时序目标的观测角信息、分布式双红外传感器之间的相对位置信息、时序点迹在各个成像平面中的红外辐射强度信息、时间信息以及各个时序点迹的编号ID信息；连续多帧图像中能够表征目标同一性判定特性的多维特征信息包括：同一成像目标的连续运动信息、分布式双红外传感器在基准坐标系下的连续运动信息、红外传感器相对于同一目标的观测角连续变化信息、目标的红外辐射强度变化信息以及多个目标形成的二维时序航迹的编号ID信息。Specifically, the time series data information obtained by the infrared sensor is analyzed, and the multi-dimensional feature information in the obtained single-frame image that can characterize the target identity determination characteristics includes: the central pixel coordinate information of the time series point trace, the distributed dual infrared sensor's own position information, the sensor's observation angle information relative to the time series target, the relative position information between the distributed dual infrared sensors, the infrared radiation intensity information of the time series point trace in each imaging plane, the time information and the numbering ID information of each time series point trace; the multi-dimensional feature information in continuous multi-frame images that can characterize the target identity determination characteristics includes: the continuous motion information of the same imaging target, the continuous motion information of the distributed dual infrared sensors in the reference coordinate system, the continuous change information of the infrared sensor's observation angle relative to the same target, the target's infrared radiation intensity change information and the numbering ID information of the two-dimensional time series track formed by multiple targets.
考虑分布式双红外传感器实际场景中时序数据信息通常成对存在，以及考虑实际场景的数据集较少并且需要模型快速前向推断实现准确快速提取输入的时序数据信息的多维特征，构建时序孪生网络作为多目标同一性判定模型的特征提取网络，时序孪生网络的两个子网络模块以具有较快前向推断速度的时序处理神经网络构建，并共享权值与参数。特征提取网络的输入为分布式双红外传感器采集得到的一对时序数据信息，将两个传感器采集到的时序数据分别输入至时序孪生网络的两个子网络模块，经过特征提取网络输出提取到两个时序数据信息的多维特征。Considering that time series data information in actual scenes of distributed dual infrared sensors usually exists in pairs, and considering that the data set of actual scenes is small and the model needs to be quickly forward inferred to accurately and quickly extract the multi-dimensional features of the input time series data information, a time series twin network is constructed as the feature extraction network of the multi-target identity judgment model. The two sub-network modules of the time series twin network are constructed with time series processing neural networks with faster forward inference speed, and share weights and parameters. The input of the feature extraction network is a pair of time series data information collected by the distributed dual infrared sensors. The time series data collected by the two sensors are respectively input into the two sub-network modules of the time series twin network, and the multi-dimensional features of the two time series data information are extracted through the output of the feature extraction network.
在步骤S105至S107中，构建时序孪生网络多目标同一性判定模型的同一性判定度量网络，以神经网络度量方式求得特征提取网络所提取的两个多维特征的同一性判定概率并构建同一性评分矩阵，在此基础上构建同一性判定矩阵作为模型输出。In steps S105 to S107, an identity determination metric network of the time series twin network multi-objective identity determination model is constructed, the identity determination probability of the two multidimensional features extracted by the feature extraction network is obtained by a neural network metric method, and an identity scoring matrix is constructed. On this basis, an identity determination matrix is constructed as a model output.
具体的，同一性判定度量网络首先通过卷积层对特征提取网络提取的多维特征进行处理，其次采用全局均值池化对特征继续处理，目的是减少模型参数，缓解模型的过拟合问题，然后利用sigmoid激活函数将同一性判定概率映射至0~1之间；基于不同编号的时序航迹数据间的同一性判定概率构建同一性评分矩阵，此矩阵共有两维，分别表示两个红外传感器所采集的时序数据的ID编号。通过比较目标同一性评分矩阵同维度元素大小定义目标同一性判定矩阵作为同一性判定度量网络的最终输出。Specifically, the identity determination metric network first processes the multi-dimensional features extracted by the feature extraction network through a convolutional layer, and then uses global mean pooling to continue processing the features in order to reduce model parameters and alleviate the overfitting problem of the model. Then, the sigmoid activation function is used to map the identity determination probability to between 0 and 1. The identity scoring matrix is constructed based on the identity determination probability between time series track data with different numbers. This matrix has two dimensions, which represent the ID numbers of the time series data collected by the two infrared sensors. The target identity determination matrix is defined as the final output of the identity determination metric network by comparing the size of the same-dimensional elements of the target identity scoring matrix.
特征提取网络与同一性判定度量网络组成了时序孪生网络多目标同一性判定模型，模型的输入为分布式双红外传感器采集的时序数据信息，模型的输出为同一性判定矩阵。The feature extraction network and the identity determination measurement network constitute a time series twin network multi-target identity determination model. The input of the model is the time series data information collected by distributed dual infrared sensors, and the output of the model is the identity determination matrix.
另外，通过仿真环境或实际场景大量获取分布式双红外传感器观测多目标的数据集图像，对图像内时序数据赋予ID编号同时通过人工辅助现有的同一性判定方法对图像进行处理和标注，将训练集图像与所标注的期望同一性判定矩阵输出作为训练样本离线训练时序孪生网络多目标同一性判定模型，直至网络模型收敛；其中时序孪生网络多目标同一性判定模型由特征提取网络和同一性判定度量网络构成。In addition, a large number of data set images of distributed dual infrared sensors observing multiple targets are obtained through simulation environments or actual scenes, and the time series data in the images are assigned ID numbers. At the same time, the images are processed and labeled with the help of existing identity determination methods with the assistance of humans. The training set images and the annotated expected identity determination matrix outputs are used as training samples to offline train the time series twin network multi-target identity determination model until the network model converges; the time series twin network multi-target identity determination model is composed of a feature extraction network and an identity determination measurement network.
具体的，尽可能多的通过仿真环境或实际场景采集分布式双红外传感器观测多目标的红外图像，将两个传感器在同一时间获得的时序数据信息看作一对，通过插值方法拟合连续一段时间内红外传感器中多个目标形成的二维时序航迹，同时按照ID编号赋予原则为图像中的多个二维时序航迹分别赋予不同的ID编号。Specifically, as many infrared images of multiple targets observed by distributed dual infrared sensors are collected through simulation environments or actual scenes as possible, the time series data information obtained by the two sensors at the same time is regarded as a pair, and the two-dimensional time series tracks formed by multiple targets in the infrared sensors over a continuous period of time are fit through the interpolation method. At the same time, different ID numbers are assigned to the multiple two-dimensional time series tracks in the image according to the ID number assignment principle.
对于仿真环境获取的数据，自动对仿真结果进行标注，而对于实际场景采集的图像，通过人工辅助现有的航迹关联方法处理并标注这对时序数据信息，基于同一性判定矩阵表达方式得到此对数据的期望输出真值，并进行标注。For the data obtained in the simulation environment, the simulation results are automatically annotated. For the images collected in the actual scene, the existing track association method is used to process and annotate the time series data information. The expected output true value of this pair of data is obtained based on the expression of the identity judgment matrix and annotated.
对不同时间段内双红外传感器观测得到的时序信息数据完成上述处理，形成训练数据集。将训练数据集内不同对时序数据作为时序孪生网络多目标同一性判定模型中孪生网络的输入，将人工标注的该对数据的同一性判定矩阵作为时序孪生网络多目标同一性判定模型的输出真值。以完成模型的训练。The above processing is completed for the time series information data observed by the dual infrared sensors in different time periods to form a training data set. Different pairs of time series data in the training data set are used as the input of the twin network in the time series twin network multi-target identity judgment model, and the manually labeled identity judgment matrix of the pair of data is used as the output truth value of the time series twin network multi-target identity judgment model to complete the training of the model.
模型训练时，计算实际输出矩阵与期望输出矩阵的偏差计算模型的准确率，以交叉熵函数作为损失函数对网络进行测试训练，寻找最优的网络参数，直至模型收敛，生成最终的时序孪生网络多目标同一性判定模型。During model training, the deviation between the actual output matrix and the expected output matrix is calculated to calculate the accuracy of the model. The network is tested and trained using the cross entropy function as the loss function to find the optimal network parameters until the model converges to generate the final time series twin network multi-target identity determination model.
将模型部署至实际分布式双红外传感器探测多目标场景中，将两台传感器实测的一对时序数据输入至训练好的模型中，得到此场景下的多目标同一性判定矩阵，矩阵中每行或每列有且仅有一个位置为1，此位置行号对应ID编号代表的二维时序航迹与列号对应ID编号代表的二维时序航迹被判定为同一目标。The model is deployed in an actual distributed dual-infrared sensor detection multi-target scenario, and a pair of time series data measured by the two sensors is input into the trained model to obtain the multi-target identity judgment matrix in this scenario. Each row or column in the matrix has only one position of 1. The two-dimensional time series track represented by the ID number corresponding to the row number and the two-dimensional time series track represented by the ID number corresponding to the column number are judged to be the same target.
具体的，将模型嵌入至实际应用场景中，将分布式双红外传感器实时测量的数据按照既定的ID编号赋予原则编号后，将实际工作中的两台红外传感器分别采集到的时序数据信息分别作为多目标同一性判定模型中时序孪生网络两个子网络模块的输入，输入到训练好的时序孪生网络多目标同一性判定模型中，得到实测数据的同一性判定结果输出矩阵。Specifically, the model is embedded into the actual application scenario, and the data measured in real time by the distributed dual infrared sensors are numbered according to the established ID numbering principles. The time series data information collected by the two infrared sensors in actual work are used as the input of the two sub-network modules of the time series twin network in the multi-target identity determination model, and then input into the trained time series twin network multi-target identity determination model to obtain the output matrix of the identity determination results of the measured data.
判定结果输出矩阵中每行或者每列有且仅有一个位置为1，此位置行号对应ID编号代表的二维时序航迹与列号对应ID编号代表的二维时序航迹被判定为由同一目标在分布式双红外传感器内形成，完成基于时序孪生网络的分布式双红外传感器多目标的同一性判定。In the judgment result output matrix, there is only one position that is 1 in each row or column. The two-dimensional time-series track represented by the ID number corresponding to the row number of this position and the two-dimensional time-series track represented by the ID number corresponding to the column number are judged to be formed by the same target in the distributed dual infrared sensor, completing the identity judgment of multiple targets of the distributed dual infrared sensor based on the time twin network.
在一个实施例中，本申请的实施例描述了典型场景即两台采用分布式站位的机载平台分别搭载红外传感器构成分布式双红外传感器（传感器1、传感器2）探测远距离多个目标的场景，具体场景如图2所示。所述分布式双红外传感器均处于匀速运动或静止状态，且均处于对目标的搜索跟踪阶段，同时由于双红外传感器自身位置、视场大小、干扰与遮挡等原因，双红外传感器所观测到的目标个数可能不同，但处于任一传感器可探测视场内的目标均能在此传感器的红外成像平面中投影出时序点迹数据和时间上具有连续特征的二维时序航迹数据，其中二维时序航迹数据是由连续多个时刻的时序点迹数据拟合而成。本申请不考虑双红外传感器相对于机载平台的安装位置，即假设机载平台本身质心坐标位置近似看为传感器在基准坐标系下的空间坐标位置。同时不考虑目标检测问题，即双红外传感器能够实现对远距离的点目标连续的检测，并实现跟踪，点目标在两个传感器的二维成像平面中能够形成一段具有连续运动特征的二维时序航迹，以考虑此场景下的多目标同一性判定问题。图1为该场景下本实施例的具体流程步骤，结合图1对本申请进行详细的描述，具体包括如下步骤：In one embodiment, the embodiment of the present application describes a typical scenario, that is, two airborne platforms using distributed stations are respectively equipped with infrared sensors to form a distributed dual infrared sensor (sensor 1, sensor 2) to detect multiple targets at a long distance, and the specific scenario is shown in Figure 2. The distributed dual infrared sensors are both in a uniform motion or static state, and are both in the stage of searching and tracking the target. At the same time, due to the position of the dual infrared sensors themselves, the size of the field of view, interference and occlusion, the number of targets observed by the dual infrared sensors may be different, but the targets in the detectable field of view of any sensor can project time series point data and two-dimensional time series track data with continuous characteristics in time in the infrared imaging plane of this sensor, wherein the two-dimensional time series track data is fitted by the time series point data at multiple consecutive moments. This application does not consider the installation position of the dual infrared sensor relative to the airborne platform, that is, it is assumed that the coordinate position of the center of mass of the airborne platform itself is approximately regarded as the spatial coordinate position of the sensor in the reference coordinate system. At the same time, the target detection problem is not considered, that is, the dual infrared sensors can realize continuous detection and tracking of distant point targets. The point targets can form a two-dimensional time-series track with continuous motion characteristics in the two-dimensional imaging planes of the two sensors to consider the problem of multi-target identity determination in this scenario. Figure 1 shows the specific process steps of this embodiment in this scenario. Combined with Figure 1, this application is described in detail, which specifically includes the following steps:
针对两个机载平台分别搭载的红外传感器，选择红外传感器1为基准传感器，目的是以基准传感器的时间戳为基准完成双红外传感器的时间软同步；则红外传感器2为非基准传感器；同时选择基准坐标系，目的是在基准坐标系下计算双红外传感器的三维空间坐标位置。For the infrared sensors carried by the two airborne platforms, infrared sensor 1 is selected as the reference sensor, with the purpose of completing the time soft synchronization of the dual infrared sensors based on the timestamp of the reference sensor; infrared sensor 2 is the non-reference sensor; at the same time, the reference coordinate system is selected, with the purpose of calculating the three-dimensional spatial coordinate positions of the dual infrared sensors under the reference coordinate system.
由于双红外传感器的红外焦平面阵列扫描标称速率与积分时间的不同，双红外传感器在某一时刻所采集的表示同一目标的时序点迹很难在严格的同一时间一一对应，此问题常被称为由于时间不同步所造成的时移观测偏差问题，为解决这种问题，采用时间软同步的方法对分布式双红外传感器完成时间同步，即以基准传感器的时间为基准时间戳，利用基准传感器在某一时间段内形成的二维时序航迹信息，通过插值的方法，得到基准时间戳下非基准传感器成像平面中同一目标所形成的时序点迹的像素坐标信息。本实例中所描述的双红外传感器成像平面中的时序点迹数据与时序航迹数据，均是在完成双红外传感器时间软同步后所获得的，以尽量减小由于双红外传感器时间不同步所造成的观测偏差问题，继而提高最终的双红外传感器多目标关联正确率。Due to the difference in the nominal scanning rate and integration time of the infrared focal plane array of the dual infrared sensors, the time series points representing the same target collected by the dual infrared sensors at a certain moment are difficult to correspond one by one at the same time. This problem is often called the time-shift observation deviation problem caused by time asynchrony. To solve this problem, the time soft synchronization method is used to complete the time synchronization of the distributed dual infrared sensors, that is, the time of the reference sensor is used as the reference timestamp, and the two-dimensional time series track information formed by the reference sensor in a certain period of time is used to obtain the pixel coordinate information of the time series points formed by the same target in the imaging plane of the non-reference sensor under the reference timestamp through interpolation. The time series point data and time series track data in the imaging plane of the dual infrared sensors described in this example are obtained after completing the time soft synchronization of the dual infrared sensors, so as to minimize the observation deviation problem caused by the time asynchrony of the dual infrared sensors, and then improve the final dual infrared sensor multi-target association accuracy.
通常机载平台的坐标位置数据由机载惯性导航系统获得，所获得的坐标数据以机载平台所在位置的经度，维度和高度数据表示，此形式的表示方法也被称为大地坐标系（W系）表示方法。本申请选取地心地固坐标系（E系）作为基准坐标系，此坐标系以地心为坐标原点，X轴指向本初子午线与赤道的交点，Z轴指向北极，Y轴服从右手定则。通过坐标变换能够将获得的机载平台经纬高数据转换为地心地固坐标系下的表达方式，具体坐标变换公式如下：Usually, the coordinate position data of the airborne platform is obtained by the airborne inertial navigation system. The obtained coordinate data is expressed in the longitude, latitude and altitude data of the location of the airborne platform. This form of representation is also called the geodetic coordinate system (W system) representation method. This application selects the geocentric earth-fixed coordinate system (E system) as the reference coordinate system. This coordinate system uses the center of the earth as the coordinate origin, the X-axis points to the intersection of the prime meridian and the equator, the Z-axis points to the North Pole, and the Y-axis obeys the right-hand rule. The obtained latitude, longitude and altitude data of the airborne platform can be converted into the expression in the geocentric earth-fixed coordinate system through coordinate transformation. The specific coordinate transformation formula is as follows:
式中为机载平台质心所在位置经度，纬度，高度数据。是机载平台所在位置的子午椭圆平面的椭圆表达式系数，分别表示长轴半径和短轴半径。为机载平台质心在基准坐标系下的坐标，N为第一参数，e为第二参数。后续本实施例中所有场景均在基准坐标系（地心地固直角坐标系）下完成，所选取的基准坐标系与基准传感器如图3所示。In the formula It is the longitude, latitude and altitude data of the center of mass of the airborne platform. are the coefficients of the ellipse expression of the meridian ellipse plane at the location of the airborne platform, representing the major axis radius and the minor axis radius respectively. is the coordinate of the center of mass of the airborne platform in the reference coordinate system, N is the first parameter, and e is the second parameter. All subsequent scenes in this embodiment are completed in the reference coordinate system (earth-centered earth-fixed rectangular coordinate system), and the selected reference coordinate system and reference sensor are shown in Figure 3.
根据连续一段时间同一目标在双红外传感器成像平面中所成像的时序点迹数据中所包含的像素坐标信息，通过三次样条插值方法，拟合出双红外传感器成像平面中同一目标形成的二维时序航迹数据，同时将基准传感器时间戳信息同步至非基准传感器，完成双红外传感器时间软同步。According to the pixel coordinate information contained in the time series point data of the same target imaged in the dual infrared sensor imaging plane for a continuous period of time, the two-dimensional time series track data formed by the same target in the dual infrared sensor imaging plane is fitted through the cubic spline interpolation method. At the same time, the timestamp information of the reference sensor is synchronized to the non-reference sensor to complete the dual infrared sensor time soft synchronization.
对于同一目标，在没有遮挡的情况下，红外传感器所截取的单帧图像中会包含此目标在图像中的成像时序点迹数据，由于此目标在三维空间中的运动具有连续特征，因此其在双红外传感器中的二维成像平面中也应具有一种连续的运动特征，这一运动特征信息我们称之为二维时序航迹，二维时序航迹包含多个时序点迹数据。本申请结合时序点迹数据在连续一段时间内的多个像素坐标信息通过三次样条插值的方式拟合出红外传感器中同一目标的二维时序航迹。从观测到目标的第时刻开始记录同一目标所成像时序点迹的像素坐标信息，根据其后连续多个时刻的时序点迹像素坐标信息，实时拟合出该目标在某一红外传感器成像平面中的二维时序航迹。具体方法为：For the same target, in the absence of occlusion, the single-frame image captured by the infrared sensor will contain the imaging time-series point track data of the target in the image. Since the movement of the target in three-dimensional space has a continuous feature, it should also have a continuous motion feature in the two-dimensional imaging plane of the dual infrared sensors. We call this motion feature information a two-dimensional time-series track, which contains multiple time-series point track data. The present application combines multiple pixel coordinate information of the time-series point track data within a continuous period of time through cubic spline interpolation to fit the two-dimensional time-series track of the same target in the infrared sensor. From the observation to the target’s first The pixel coordinate information of the time series point traces of the same target imaged starts from the moment, and the two-dimensional time series track of the target in a certain infrared sensor imaging plane is fitted in real time based on the pixel coordinate information of the time series point traces at multiple consecutive moments. The specific method is:
对于在某个红外传感器成像平面中的时序点迹所表示的时序目标，最初观测到该目标的时间记为基准传感器时间戳下0时刻，此时刻的时序点迹像素坐标记为，其后个时刻观测得到的此目标的时序点迹像素坐标分别记为，将上述像素坐标看作自变量时间的因变量，该时间处于基准传感器的时间戳下，For the time series target represented by the time series trace in a certain infrared sensor imaging plane The time when the target is first observed is recorded as time 0 under the timestamp of the reference sensor, and the pixel coordinates of the time series point trace at this time are marked as , thereafter The pixel coordinates of the time series points of this target observed at each moment are recorded as , considering the above pixel coordinates as the independent variable time The dependent variable is at the time stamp of the reference sensor.
个时刻包含个区间，然后使用三次样条分段函数来拟合每个区间的数据，三次样条分段函数表示为： The moment contains intervals, and then use the cubic spline piecewise function to fit the data in each interval. The cubic spline piecewise function It is expressed as:
式中表示第个区间内第时刻的时序点迹像素坐标，分别表示第个区间三次样条插值函数的未知系数，表示区间总个数，每个区间均有4个未知系数，对于个区间，需求解个未知系数。同时，得到的各样条区间内的分段插值函数在各已知取值点上连续且光滑，满足：In the formula Indicates In the interval The pixel coordinates of the time series points at the moment, Respectively represent The unknown coefficients of the cubic spline interpolation function on an interval, Represents the total number of intervals, each interval has 4 unknown coefficients. interval, requiring solution unknown coefficients. At the same time, the obtained piecewise interpolation function in each spline interval It is continuous and smooth at each known value point, satisfying:
式中，分别表示第时刻表示的端点两侧像素坐标大小求解函数表达式，函数值相等表示函数连续，分别表示第时刻表示的端点两侧的像素坐标求解函数表达式的一阶导数，分别表示第时刻表示的端点两侧的像素坐标求解函数表达式的二阶导数，其函数表达式的一阶导数与二阶导数存在且相等表示任意时刻分段插值函数连续且光滑。In the formula, Respectively represent Solve the function expression by the pixel coordinates on both sides of the endpoint represented by the moment. If the function value is equal, it means the function is continuous. Respectively represent The pixel coordinates on both sides of the endpoints represented by the time instant are used to solve the first-order derivative of the function expression. Respectively represent The pixel coordinates on both sides of the endpoints represented by the time are used to solve the second-order derivative of the function expression. The first-order derivative and the second-order derivative of the function expression exist and are equal, indicating the piecewise interpolation function at any time. Continuous and smooth.
结合三次样条插值方程得到目标在红外传感器成像平面中二维时序航迹，二维时序航迹方程为：Combined with the cubic spline interpolation equation, the target Two-dimensional time-series trajectory in the infrared sensor imaging plane , the two-dimensional time series track equation is:
式中为该二维时序航迹在第时刻的像素坐标，为每个样条区间的分段插值函数，为每个样条区间的样条系数，表示区间总个数，表示第个任意区间。In the formula The two-dimensional time series trajectory is The pixel coordinates at the time, is the piecewise interpolation function for each spline interval, are the spline coefficients for each spline interval, Represents the total number of intervals, Indicates An arbitrary interval.
确定分布式红外传感器所获得的成像平面中多目标时序点迹数据与二维时序航迹数据的ID编号赋予原则,并根据编号原则分别对双红外传感器成像平面中的二维时序航迹数据赋予ID编号，同时对分布式双红外传感器多目标同一性判定问题进行数学描述，并在此基础上建立目标数未知条件下的双红外传感器多目标同一性评分矩阵与同一性判定矩阵表达式。The ID numbering principle for the multi-target time-series point data and the two-dimensional time-series track data in the imaging plane obtained by the distributed infrared sensor is determined, and the two-dimensional time-series track data in the imaging plane of the dual infrared sensors are assigned ID numbers according to the numbering principle. At the same time, the problem of identity determination of multi-targets of distributed dual infrared sensors is mathematically described, and on this basis, the identity scoring matrix and identity determination matrix expressions of multi-targets of dual infrared sensors under the condition of unknown target number are established.
对于单帧图像中获取的时序点迹数据，以其所在成像平面中的像素坐标大小作为ID编号赋予原则，获得所有时序点迹数据的像素坐标值，其中表示成像平面中的时序点迹个数。首先比较所有时序点迹的横坐标大小，从坐标较小的以#1开始编号，从小到大依次递增1，若遇到横坐标相同的情况，比较两时序点迹的像素纵坐标大小，从较小的开始编号。For the time series point trace data obtained in a single frame image, the pixel coordinate size in the imaging plane is used as the ID number assignment principle to obtain the pixel coordinate value of all time series point trace data ,in Indicates the number of time series traces in the imaging plane. First, compare the horizontal coordinates of all time series traces The size is numbered starting from #1 with the smaller coordinate and increasing by 1 from small to large. In the same situation, compare the pixel ordinates of the two time series traces Size, numbered starting with the smallest.
对于同一个目标在同一传感器成像平面中连续多帧形成的二维时序航迹数据，选取并记录开始探测到此目标的单帧图像中该目标形成的时序点迹的像素坐标作为ID编号依据，对于已经有ID号的目标，其编号不再随后续像素坐标的变化而变化，此后对于新探测到的目标，记录其进入传感器成像平面的时刻，并计算此时刻之前的目标个数，在此个数上加1给新探测到的目标赋予ID编号。For the two-dimensional time-series track data formed by multiple consecutive frames of the same target in the same sensor imaging plane, the pixel coordinates of the time-series point track formed by the target in the single-frame image where the target is detected are selected and recorded as the basis for the ID number. For targets that already have ID numbers, their numbers will no longer change with the subsequent changes in pixel coordinates. After that, for newly detected targets, the moment when they enter the sensor imaging plane is recorded, and the number of targets before this moment is calculated. The number is added by 1 to assign an ID number to the newly detected target.
上述ID编号赋予原则以具体传感器的成像平面为例，示于图4和图5中，图4为对单帧图像中的时序点迹数据进行编号，成像平面中共计五个目标形成的时序点迹，依据各个时序点迹数据在当前时刻的像素坐标大小赋予编号#1~#5，其中#1与#2像素横坐标相同，而#1像素纵坐标小于#2像素纵坐标，因此以此编号赋予各个时序点迹；图5为对二维时序航迹数据编号的示意，图中已有三条时序航迹，其ID编号均是以其初始被探测时刻单帧图像中所在像素坐标位置大小原则赋予的，对于被赋予#4的时序航迹，是当前时刻刚被探测到的时序数据，尽管此时刻#4的点迹数据像素横坐标小于其余航迹初始被探测到的时刻所记录的像素横坐标，但不对原有编号进行更新。The above-mentioned ID number assignment principle is shown in Figures 4 and 5, taking the imaging plane of a specific sensor as an example. Figure 4 shows the numbering of the time series point data in a single frame image. The time series point data formed by a total of five targets in the imaging plane are numbered #1~#5 according to the pixel coordinate size of each time series point data at the current moment, where #1 and #2 have the same pixel horizontal coordinates, and #1 pixel vertical coordinate is smaller than #2 pixel vertical coordinate, so each time series point data is assigned with this number; Figure 5 is a schematic diagram of the numbering of two-dimensional time series track data. There are three time series tracks in the figure, and their ID numbers are all assigned based on the principle of the size of the pixel coordinate position in the single frame image at the time of initial detection. For the time series track assigned #4, it is the time series data just detected at the current moment. Although the pixel horizontal coordinate of the point track data of #4 at this moment is smaller than the pixel horizontal coordinate recorded at the time when the other tracks were initially detected, the original number is not updated.
对于分布式双红外传感器观测多个目标的场景，在未知目标数条件下，假设在连续一段时间内，基准与非基准红外传感器成像平面中分别获得个和个二维时序航迹数据，并分别表示为集合和，针对双红外传感器成像平面中时序航迹数据的同一性判定问题可以描述为图6所示，该问题的数学描述表示为：For the scenario where the distributed dual infrared sensors observe multiple targets, under the condition of unknown target number, it is assumed that in a continuous period of time, the reference and non-reference infrared sensor imaging planes are respectively obtained and Two-dimensional time series track data, and are represented as sets and , the problem of determining the identity of the time series track data in the imaging plane of the dual infrared sensors can be described as shown in Figure 6. The mathematical description of the problem is expressed as:
其中，表示目标同一性判定估计度值，表示约束条件。为同一性判定结果，为同一性评分，表示两条不同传感器中的二维时序航迹数据匹配概率，大小在0~1之间，根据此评分的大小判断两条时序航迹数据是否匹配，并得到，表示双红外传感器成像平面中的两条二维时序航迹是否判定为同一目标在不同红外传感器中的成像，0表示不是同一目标，1表示判定为同一目标。in, represents the target identity judgment estimate value, Represents a constraint. The result of the identity determination is is the identity score, which indicates the matching probability of two-dimensional time series track data from two different sensors. Its value is between 0 and 1. The size of this score is used to determine whether the two time series track data match, and the matching probability is obtained. , Indicates whether the two 2D time-series tracks in the dual infrared sensor imaging plane are judged to be the images of the same target in different infrared sensors. 0 means they are not the same target, and 1 means they are the same target.
根据上述概念,显然两个红外传感器探测到的目标总数的范围是：。针对同一性判定的数学问题描述给出可行同一性评分矩阵与可行同一性判定矩阵的定义。According to the above concepts, it is clear that the range of the total number of targets detected by the two infrared sensors is: According to the mathematical problem description of identity determination, the definitions of feasible identity scoring matrix and feasible identity determination matrix are given.
对于双红外传感器分别观测到的个时序航迹数据，定义同一性评分矩阵：For the dual infrared sensors respectively observed Time series track data, define the identity score matrix:
上述矩阵内的各元素取值在0到1之间，数值越接近于1表明同一性判定概率越大。对于双红外传感器的多目标同一性判定问题，比较矩阵中每一维度元素的大小，数值最大的元素表明基准传感器中与行号大小相同的二维时序航迹编号代表的目标与非基准传感器中与列号相同的二维时序航迹编号代表的目标被判定为同一目标。以矩阵第一行为例，若矩阵中为同维度中最接近1的元素，则判定基准传感器中第#1号二维时序航迹与非基准传感器中第#2号二维时序航迹为同一目标形成。为方便表述，根据同一性评分矩阵定义可行同一性判定矩阵，根据同一性评分矩阵将每个维度最接近1的元素所在位置修改为1，其余元素修改为0，定义如下同一性判定矩阵：Each element in the above matrix takes a value between 0 and 1. The closer the value is to 1, the greater the probability of identity determination. For the problem of multi-target identity determination of dual infrared sensors, the size of the elements in each dimension of the matrix is compared. The element with the largest value indicates that the target represented by the two-dimensional time series track number with the same row number in the reference sensor and the target represented by the two-dimensional time series track number with the same column number in the non-reference sensor are determined to be the same target. Taking the first row of the matrix as an example, if the matrix If it is the element closest to 1 in the same dimension, then it is determined that the 2D time series track No. 1 in the reference sensor and the 2D time series track No. 2 in the non-reference sensor are formed by the same target. For the convenience of expression, a feasible identity determination matrix is defined according to the identity score matrix. According to the identity score matrix, the position of the element closest to 1 in each dimension is modified to 1, and the remaining elements are modified to 0. The following identity determination matrix is defined:
同一性判定矩阵满足如下性质：The identity decision matrix satisfies the following properties:
①矩阵内元素取值为0或者1。①The elements in the matrix have values of 0 or 1.
②,表示每行有且仅有一个1。② , indicating that there is only one 1 in each row.
③,表示每列有且仅有一个1。③ , indicating that there is only one 1 in each column.
④若,且，表明基准传感器中第#号二维时序航迹与非基准传感器中第#号二维时序航迹为同一目标形成。④If ,and , indicating that the # Two-dimensional time series track and non-reference sensor No. The two-dimensional time-series tracks are formed for the same target.
⑤当时，且，表示非基准传感器中的第个目标为基准传感器独立观测。⑤When When, and , which means the first The targets are observed independently by the reference sensors.
⑥当时，且，表示基准传感器中的第个目标为基准传感器独立观测。⑥ When When, and , which indicates the first The targets are observed independently by the reference sensors.
上述同一性判定矩阵确定了基准传感器与非基准传感器中所探测目标之间的关联关系，其表示两个传感器测量信息之间的一个可关联解。同一性判定矩阵中等于1的元素的个数即双红外传感器中所探测目标的总个数。The above identity determination matrix determines the association relationship between the targets detected by the reference sensor and the non-reference sensor, which represents a relatable solution between the measurement information of the two sensors. The number of elements equal to 1 in the identity determination matrix is the total number of targets detected by the dual infrared sensors.
分析分布式双红外传感器获取的能够表征多目标同一性判定特征的时序信息，基于时序信息特点，搭建时序孪生网络多目标同一性判定模型的特征提取网络，将分布式双红外传感器采集得到的同一时间段内对自身探测视场内目标的观测时序数据信息分别输入特征提取网络的两个子网络以提取时序数据信息的多维特征。The time series information obtained by the distributed dual infrared sensors that can characterize the characteristics of multi-target identity determination is analyzed. Based on the characteristics of the time series information, a feature extraction network of the time series twin network multi-target identity determination model is built. The observation time series data information of the targets in its own detection field of view within the same time period collected by the distributed dual infrared sensors is input into the two sub-networks of the feature extraction network respectively to extract the multi-dimensional characteristics of the time series data information.
红外传感器在探测远距离的目标时，单帧图像下的目标成像通常为占几个像素的点坐标，此时分布式双红外传感器获得的能够表征目标同一性判定特征的时序数据信息包括：时序点迹的中心像素坐标信息、红外传感器自身位置与速度信息、传感器相对于时序点迹的观测角信息、时序点迹在各个成像平面中的红外辐射强度信息、时间信息以及各个时序点迹的编号ID信息；连续多帧图像中能够表征目标同一性判定特征的信息是单帧图像中信息的连续变化，将其表征为：When infrared sensors detect targets at long distances, the target image in a single frame is usually a point coordinate that occupies several pixels. At this time, the time series data information obtained by the distributed dual infrared sensors that can characterize the target identity determination features includes: the center pixel coordinate information of the time series point trace 、The infrared sensor's own position With speed information , the sensor's observation angle information relative to the time series trace , infrared radiation intensity information of time series points in each imaging plane , time information And the ID information of each timing point The information that can characterize the target identity determination features in the continuous multi-frame images is the continuous change of the information in the single-frame image, which can be characterized as:
其中表示同一性判定特征信息，分别红外传感器基准坐标系下的三维坐标位置连续变化信息，分别表征红外传感器在基准坐标系下的三维速度变化信息，分别表征红外传感器相对于同一目标的观测方位角与俯仰角连续变化信息，分别表征同一成像目标的成像像素横纵坐标连续变化信息，表示同一目标的红外辐射强度的连续变化信息，表示二维时序航迹的编号ID信息。in Indicates identity determination feature information, The continuous change information of the three-dimensional coordinate position under the infrared sensor reference coordinate system, Respectively represent the three-dimensional velocity change information of the infrared sensor in the reference coordinate system, Respectively represent the continuous change information of the observation azimuth and elevation angle of the infrared sensor relative to the same target, Respectively represent the continuous change information of the horizontal and vertical coordinates of the imaging pixels of the same imaging target, Indicates the continuous change information of the infrared radiation intensity of the same target. Indicates the ID information of the two-dimensional time series track.
所设计的神经网络架构能否提取出上述特征决定着最终的同一性判定效果，而又由于需要同时处理在同一时间段内不同目标在两台红外传感器成像平面在生成的时序数据，并评判这些时序数据是否具有相似性，故选用一种特殊的神经网络架构——孪生网络作为特征提取网络。孪生网络架构由两个子网络模块构成，在处理图像任务时，其可以同时接收两个图片作为输入并且共享权值，并输出所输入的两张图像的相似度度量。此外孪生网络结构还具有自由拓展训练集数据量的优点，这尤其适用于多目标同一性判定此类训练数据集较少的应用场景，能够充分利用有限的数据集进行网络训练。同时，由于多目标的同一性判定问题需要在目标检出过程中实时的推断同一性判定结果，并在嵌入式设备中实时快速的处理时序信息，因此需要选取一种具有较高的前向推断速度的网络结构作为孪生网络的子网络结构，本实施例选取N-BEATS网络结构作为时序孪生网络的子网络进行描述，同时也可以选取其他具有较快前向推断速度的时序处理神经网络。Whether the designed neural network architecture can extract the above features determines the final identity determination effect. Since it is necessary to simultaneously process the time series data generated by different targets in the same time period on the imaging planes of two infrared sensors and judge whether these time series data are similar, a special neural network architecture, the twin network, is selected as the feature extraction network. The twin network architecture consists of two sub-network modules. When processing image tasks, it can simultaneously receive two pictures as input and share weights, and output the similarity measurement of the two input images. In addition, the twin network structure also has the advantage of freely expanding the amount of training set data, which is particularly suitable for application scenarios such as multi-target identity determination where there are fewer training data sets, and can make full use of limited data sets for network training. At the same time, since the multi-target identity determination problem requires real-time inference of the identity determination results during the target detection process, and real-time and fast processing of time series information in embedded devices, it is necessary to select a network structure with a higher forward inference speed as the sub-network structure of the twin network. In this embodiment, the N-BEATS network structure is selected as the sub-network of the time series twin network for description, and other time series processing neural networks with faster forward inference speeds can also be selected.
其中单个子网络模块的输入是过去一段时间某个红外传感器所采集的时序数据信息，将此时序数据输入如图7所示的N-BEATS神经网络模型中，模型包含个栈，每个栈都会输出自身的预报结果，而每个栈又由个块组成，每个块都会输出一个预报输出与回溯输出。其中全连接层由四个relu函数组成，表达式如下：The input of a single sub-network module is the time series data information collected by a certain infrared sensor over a period of time. , input this time series data into the N-BEATS neural network model shown in Figure 7. The model contains stacks, each of which will output its own prediction results , and each stack consists of Each block will output a forecast output With traceback output The fully connected layer consists of four relu functions, and the expression is as follows:
式中，，，，分别表示为第个模块内第层全连接层的输入，权重，偏置和输出。其中线性层的数学表达式为：In the formula, , , , Respectively expressed as In the module The input, weight, bias and output of the fully connected layer. The mathematical expression of the linear layer is:
式中，，分别为用于预测第个模块预报输出的系数和权重，，分别为用于预测第个模块回溯输出的参数和权重，分别将，作为参数，对输入序列进行估计得到和，计算公式如下：In the formula, , They are used to predict the Module forecast output The coefficients and weights of , They are used to predict the Module traceback output The parameters and weights of , As a parameter, the input sequence is estimated to obtain and , the calculation formula is as follows:
式中和分别表示历史数据组成的向量。通过上述方法能够预测得到第个模块预报输出与回溯输出，其中回溯输出用于计算其后一个块的输入，方法是当前块的输入减去回溯输出：In the formula and Respectively represent the vectors composed of historical data. Through the above method, it can be predicted that Module forecast output With traceback output , where the backtracking output is used to calculate the input of the next block by subtracting the backtracking output from the input of the current block:
而将每个块的预报输出进行累加求和得到每个栈的输出，同样，将每个栈的输出进行累加得到模型的最终输出。The predicted output of each block is accumulated and summed to obtain the output of each stack. Similarly, the output of each stack is accumulated to obtain the final output of the model.
式中表示第个栈的输出，表示个栈的总输出，即模型的总输出。此输出表示时序孪生网络中单个子网络对某个红外传感器所采集的时序数据提取的多维特征。采用同样的结构搭建时序孪生网络中另一个子网络，并将另外一个红外传感器所采集的传感器成像平面内多个成像目标的连续运动信息、自身的三维空间连续运动信息、红外传感器相对于多个成像目标的观测角连续变化信息、多个成像目标的红外辐射强度变化信息等时序信息输入至该网络。并对两个子网络共享权重，提取此红外传感器所采集时序信息的多维特征。In the formula Indicates The output of the stack, express The total output of the stack is the total output of the model. This output represents the multi-dimensional features extracted by a single sub-network in the time series twin network from the time series data collected by a certain infrared sensor. The same structure is used to build another sub-network in the time series twin network, and the continuous motion information of multiple imaging targets in the sensor imaging plane collected by another infrared sensor, its own three-dimensional space continuous motion information, the continuous change information of the observation angle of the infrared sensor relative to multiple imaging targets, and the change information of the infrared radiation intensity of multiple imaging targets are input into the network. The weights of the two sub-networks are shared to extract the multi-dimensional features of the time series information collected by this infrared sensor.
至此，通过将分布式双红外传感器在同一时间段内所采集的时序信息分别输入孪生时序网络特征提取网络的两个子网络，提取出了双红外传感器中时序信息中表征目标同一性判定特性的多维特征。At this point, by inputting the timing information collected by the distributed dual infrared sensors in the same time period into the two sub-networks of the twin timing network feature extraction network, the multi-dimensional features representing the target identity judgment characteristics in the timing information of the dual infrared sensors are extracted.
构建时序孪生网络多目标同一性判定模型的同一性判定度量网络，以神经网络度量方式求得提取的两个多维特征的同一性判定概率并构建同一性评分矩阵，通过比较评分矩阵同维度元素的大小构建分布式双红外传感器多目标同一性判定矩阵，并将此矩阵作为时序孪生网络多目标同一性判定模型的输出。An identity determination measurement network of the time series twin network multi-target identity determination model is constructed. The identity determination probability of the two extracted multidimensional features is obtained by using a neural network measurement method and an identity scoring matrix is constructed. The distributed dual-infrared sensor multi-target identity determination matrix is constructed by comparing the sizes of the same-dimensional elements of the scoring matrix, and this matrix is used as the output of the time series twin network multi-target identity determination model.
同一性判定度量网络的作用是将特征提取网络提取出的多维特征映射为同一性判定概率并基于此概率构建同一性评分矩阵，继而通过此矩阵获得时序孪生网络多目标同一性判定模型最终的同一性判定结果输出矩阵。The role of the identity determination measurement network is to map the multidimensional features extracted by the feature extraction network into identity determination probabilities and construct an identity scoring matrix based on this probability, and then obtain the final identity determination result output matrix of the temporal twin network multi-objective identity determination model through this matrix.
如图8所示，展示了以N-BEATS作为子网络模块构建的时序孪生网络多目标同一性判定模型处理分布式双红外传感器多目标同一性判定任务的整体流程框图。其中的同一性判定度量网络首先通过卷积层对特征提取网络提取的多维特征进行处理，其次采用全局均值池化对特征继续处理，目的是减少模型参数，缓解模型的过拟合问题，然后利用sigmoid激活函数将同一性判定概率映射至0~1之间。计算同一性判定概率的数学表达式如下：As shown in Figure 8, the overall flow chart of the multi-target identity determination model of the time series twin network built with N-BEATS as the sub-network module to process the multi-target identity determination task of the distributed dual infrared sensor is shown. The identity determination measurement network first processes the multi-dimensional features extracted by the feature extraction network through the convolution layer, and then uses the global mean pooling to continue processing the features, in order to reduce the model parameters and alleviate the overfitting problem of the model, and then uses the sigmoid activation function to map the identity determination probability to between 0 and 1. The mathematical expression for calculating the identity determination probability is as follows:
式中，表示第#编号代表的时序数据与#编号代表的时序数据之间的同一性判定概率，表示将特征向量映射为同一性判定概率的关系函数，表示特征提取网络的输出，表示第#编号代表的时序数据与#编号代表的时序数据，为激活函数。In the formula, Indicates the # The number represents the time series data and # The probability of identity determination between the time series data represented by the numbers, represents the relationship function that maps feature vectors to identity determination probabilities, represents the output of the feature extraction network, Indicates the # The number represents the time series data and # The numbers represent the time series data. is the activation function.
基于不同编号的时序航迹数据间的同一性判定概率构建同一性评分矩阵，此矩阵共有两维，分别表示两个红外传感器所采集的时序数据的ID编号，其相应元素位置表示行号对应ID编号代表的时序数据与列号对应ID编号代表的时序数据被判定为同一目标的概率，此概率在0~1之间，越趋近于1说明被判定为同一目标的可能性越大。Based on the probability of identity judgment between time series track data with different numbers, an identity scoring matrix is constructed. This matrix has two dimensions, which represent the ID numbers of the time series data collected by the two infrared sensors. The corresponding element position represents the probability that the time series data represented by the row number corresponding to the ID number and the time series data represented by the column number corresponding to the ID number are judged to be the same target. This probability is between 0 and 1. The closer it is to 1, the greater the possibility of being judged to be the same target.
最后通过比较评分矩阵中同维度数值的大小修改对应位置元素，将同一维度中评分概率最大的位置定义为1，表示此位置行号对应ID编号代表的时序数据与列号对应ID编号代表的时序数据被判定为同一目标，其余位置定义为0，表示此位置行号对应ID编号代表的时序数据与列号对应ID编号代表的时序数据被判定不是同一目标。Finally, by comparing the values of the same dimension in the scoring matrix, the corresponding position elements are modified. The position with the largest scoring probability in the same dimension is defined as 1, indicating that the time series data represented by the ID number corresponding to the row number of this position and the time series data represented by the ID number corresponding to the column number are judged to be the same target. The remaining positions are defined as 0, indicating that the time series data represented by the ID number corresponding to the row number of this position and the time series data represented by the ID number corresponding to the column number are judged not to be the same target.
上述构建的特征提取网络与上述构建的同一性判定度量网络组成了时序孪生网络多目标同一性判定模型，模型的输入为分布式双红外传感器采集的一对时序数据信息，模型的输出为同一性判定矩阵。The feature extraction network constructed above and the identity determination measurement network constructed above constitute a time series twin network multi-target identity determination model. The input of the model is a pair of time series data information collected by distributed dual infrared sensors, and the output of the model is an identity determination matrix.
获取分布式双红外传感器观测多目标实际场景的红外图像，以形成数据训练集与测试集，按照ID编号赋予原则为图像内时序数据信息赋予编号，通过人工辅助现有的同一性判定方法处理训练集图像，以得到分布式双红外传感器内多目标的同一性判定结果，并根据所述标准同一性判定矩阵表示方法，得到不同训练集图像的标注输出真值。将训练集图像与其对应的标注输出作为训练样本离线训练时序孪生网络多目标同一性判定模型，直至网络模型收敛。Obtain infrared images of actual scenes of multiple targets observed by distributed dual infrared sensors to form data training sets and test sets, assign numbers to the time series data information in the images according to the ID number assignment principle, process the training set images by manually assisting the existing identity determination method to obtain the identity determination results of multiple targets in the distributed dual infrared sensors, and obtain the true values of the labeled outputs of different training set images according to the standard identity determination matrix representation method. Use the training set images and their corresponding labeled outputs as training samples to train the time series twin network multi-target identity determination model offline until the network model converges.
尽可能多的采集分布式双红外传感器观测多目标实际场景的红外图像，记录在一段时间内两个红外传感器观测得到的多个目标形成的时序信息，同一时间段两个传感器分别获得的时序数据看为一对数据，并采用二维平面航迹插值方法得到同一目标的二维时序航迹，同时通过ID编号赋予原则为多个目标形成的二维时序航迹分别赋予不同的ID编号，然后通过人工辅助现有的航迹关联方法处理结果，得到此实际场景下的分布式双红外传感器获取的一对数据的多个目标形成的二维时序航迹的同一性判定结果，根据同一性判定矩阵表达方式得到此对数据的期望输出真值。对不同时间段内双红外传感器观测得到的时序信息数据完成上述处理，形成训练数据集。将训练数据集内不同对数据内的两个时序信息分别作为时序孪生网络多目标同一性判定模型中孪生网络两个子网络模块的输入，将人工标注的不同对数据的同一性判定矩阵作为时序孪生网络多目标同一性判定模型的输出真值，以完成模型的训练。利用上述训练数据集，通过计算实际输出矩阵与期望输出矩阵的偏差计算模型的准确率，以交叉熵函数作为损失函数对网络进行训练，寻找最优的网络参数，直至模型收敛，生成最终的时序孪生网络多目标同一性判定模型。Collect as many infrared images of actual scenes of multiple targets observed by distributed dual infrared sensors as possible, record the time series information formed by multiple targets observed by two infrared sensors within a period of time, regard the time series data obtained by two sensors in the same time period as a pair of data, and use the two-dimensional plane track interpolation method to obtain the two-dimensional time series track of the same target. At the same time, different ID numbers are assigned to the two-dimensional time series tracks formed by multiple targets according to the ID number assignment principle, and then the results are processed by artificially assisting the existing track association method to obtain the identity judgment result of the two-dimensional time series track formed by multiple targets of a pair of data obtained by the distributed dual infrared sensors in this actual scene, and the expected output true value of this pair of data is obtained according to the expression of the identity judgment matrix. The above processing is completed for the time series information data observed by the dual infrared sensors in different time periods to form a training data set. The two time series information in different pairs of data in the training data set are respectively used as the input of the two sub-network modules of the twin network in the time series twin network multi-target identity judgment model, and the identity judgment matrix of different pairs of data manually labeled is used as the output true value of the time series twin network multi-target identity judgment model to complete the model training. Using the above training data set, the accuracy of the model is calculated by calculating the deviation between the actual output matrix and the expected output matrix. The network is trained with the cross entropy function as the loss function to find the optimal network parameters until the model converges to generate the final time series twin network multi-target identity determination model.
将模型嵌入至实际应用场景的平台中，将分布式双红外传感器实时测量的一对时序数据按照既定的ID编号赋予原则编号后，并作为模型中时序孪生网络的输入，输入到训练好的时序孪生网络多目标同一性判定模型中，得到实测数据的同一性判定结果输出矩阵。最终的判定结果输出矩阵中每行或者每列有且仅有一个位置为1，此位置行号对应ID编号代表的二维时序航迹与列号对应ID编号代表的二维时序航迹被判定由同一目标在分布式双红外传感器内形成，至此完成基于时序孪生网络的分布式双红外传感器多目标的同一性判定。The model is embedded in the platform of the actual application scenario. A pair of time series data measured by the distributed dual infrared sensor in real time is assigned a number according to the established ID number principle, and is used as the input of the time series twin network in the model, and is input into the trained time series twin network multi-target identity judgment model to obtain the output matrix of the identity judgment result of the measured data. In the final judgment result output matrix, there is only one position of 1 in each row or column. The two-dimensional time series track represented by the row number corresponding to the ID number and the two-dimensional time series track represented by the column number corresponding to the ID number are judged to be formed by the same target in the distributed dual infrared sensor. At this point, the identity judgment of the distributed dual infrared sensor multi-target based on the time series twin network is completed.
通过上述分布式双红外传感器时序孪生网络多目标同一性判定方法，一方面，充分挖掘到红外传感器成像平面内多目标成像二维时序航迹信息中所隐含的表征同一性判定特性的多维特征，并通过同一性判定度量网络能够准确的对多目标实现关联匹配，解决了仅考虑几何特征时由于目标数量增多所造成的交叉伪点组合爆炸问题，尤其适用于分布式双红外传感器探测视场内具有多个目标的应用场景。另一方面，设计的时序孪生网络多目标同一性判定模型中的特征提取网络由两个结构和参数完全相同的第一特征提取模块和第二特征提取模块组成，第一特征提取模块和第二特征提取模块基于具有较快前向推断速度的时序神经处理网络构建，将分布式双红外传感器实测的一对时序数据信息分别输入至训练好的模型的第一特征提取模块和第二特征提取模块中，时序孪生网络多目标同一性判定模型能够快速推断出输入的一对实测的时序数据信息的同一性判定结果，具有较好的实用性。Through the above-mentioned distributed dual infrared sensor time series twin network multi-target identity determination method, on the one hand, the multi-dimensional features that characterize the identity determination characteristics implicit in the two-dimensional time series track information of multi-target imaging in the infrared sensor imaging plane are fully excavated, and the multi-target can be accurately matched through the identity determination metric network, which solves the problem of cross-pseudo-point combination explosion caused by the increase in the number of targets when only geometric features are considered. It is particularly suitable for application scenarios where distributed dual infrared sensors have multiple targets in the field of view. On the other hand, the feature extraction network in the designed time series twin network multi-target identity determination model consists of two first feature extraction modules and second feature extraction modules with exactly the same structure and parameters. The first feature extraction module and the second feature extraction module are constructed based on a time series neural processing network with a faster forward inference speed. A pair of time series data information measured by the distributed dual infrared sensor is respectively input into the first feature extraction module and the second feature extraction module of the trained model. The time series twin network multi-target identity determination model can quickly infer the identity determination result of the input pair of measured time series data information, and has good practicality.
此外，术语“第一”、“第二”仅用于描述目的，而不能理解为指示或暗示相对重要性或者隐含指明所指示的技术特征的数量。由此，限定有“第一”、“第二”的特征可以明示或者隐含地包括一个或者更多个该特征。在本公开实施例的描述中，“多个”的含义是两个或两个以上，除非另有明确具体的限定。In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present disclosure, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.
在本说明书的描述中，参考术语“一个实施例”、“一些实施例”、“示例”、“具体示例”、或“一些示例”等的描述意指结合该实施例或示例描述的具体特征、结构、材料或者特点包含于本公开的至少一个实施例或示例中。在本说明书中，对上述术语的示意性表述不必须针对的是相同的实施例或示例。而且，描述的具体特征、结构、材料或者特点可以在任何的一个或多个实施例或示例中以合适的方式结合。此外，本领域的技术人员可以将本说明书中描述的不同实施例或示例进行结合和组合。In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification.
本领域技术人员在考虑说明书及实践这里公开的发明后，将容易想到本公开的其它实施方案。本申请旨在涵盖本公开的任何变型、用途或者适应性变化，这些变型、用途或者适应性变化遵循本公开的一般性原理并包括本公开未公开的本技术领域中的公知常识或惯用技术手段。说明书和实施例仅被视为示例性的，本公开的真正范围和精神由所附的权利要求指出。Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any modification, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the appended claims.
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410693990.5ACN118279568B (en) | 2024-05-31 | 2024-05-31 | Multi-target identity judging method for distributed double-infrared sensor time sequence twin network |
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410693990.5ACN118279568B (en) | 2024-05-31 | 2024-05-31 | Multi-target identity judging method for distributed double-infrared sensor time sequence twin network |
| Publication Number | Publication Date |
|---|---|
| CN118279568A CN118279568A (en) | 2024-07-02 |
| CN118279568Btrue CN118279568B (en) | 2024-09-17 |
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410693990.5AActiveCN118279568B (en) | 2024-05-31 | 2024-05-31 | Multi-target identity judging method for distributed double-infrared sensor time sequence twin network |
| Country | Link |
|---|---|
| CN (1) | CN118279568B (en) |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111612820A (en)* | 2020-05-15 | 2020-09-01 | 北京百度网讯科技有限公司 | Multi-target tracking method, training method and device for feature extraction model |
| WO2023124133A1 (en)* | 2021-12-29 | 2023-07-06 | 上海商汤智能科技有限公司 | Traffic behavior detection method and apparatus, electronic device, storage medium, and computer program product |
| CN117974766A (en)* | 2024-03-28 | 2024-05-03 | 西北工业大学 | Multi-target identity determination method of distributed dual infrared sensors based on time and space |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115311329B (en)* | 2019-10-11 | 2023-05-23 | 杭州云栖智慧视通科技有限公司 | Video multi-target tracking method based on double-link constraint |
| CN113920159B (en)* | 2021-09-15 | 2024-05-10 | 河南科技大学 | Infrared air small and medium target tracking method based on full convolution twin network |
| CN114970321B (en)* | 2022-04-28 | 2025-03-07 | 长安大学 | A scene flow digital twin method and system based on dynamic trajectory flow |
| CN114596338B (en)* | 2022-05-09 | 2022-08-16 | 四川大学 | A Siamese Network Object Tracking Method Considering Temporal Relationship |
| CN115330839B (en)* | 2022-08-22 | 2025-09-05 | 西安电子科技大学 | Anchor-free Siamese neural network-based integrated multi-target detection and tracking method |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111612820A (en)* | 2020-05-15 | 2020-09-01 | 北京百度网讯科技有限公司 | Multi-target tracking method, training method and device for feature extraction model |
| WO2023124133A1 (en)* | 2021-12-29 | 2023-07-06 | 上海商汤智能科技有限公司 | Traffic behavior detection method and apparatus, electronic device, storage medium, and computer program product |
| CN117974766A (en)* | 2024-03-28 | 2024-05-03 | 西北工业大学 | Multi-target identity determination method of distributed dual infrared sensors based on time and space |
| Publication number | Publication date |
|---|---|
| CN118279568A (en) | 2024-07-02 |
| Publication | Publication Date | Title |
|---|---|---|
| Zhang et al. | A lightweight and accurate localization algorithm using multiple inertial measurement units | |
| CN112785702A (en) | SLAM method based on tight coupling of 2D laser radar and binocular camera | |
| Raza et al. | Comparing and evaluating indoor positioning techniques | |
| CN110334701A (en) | Data acquisition method based on deep learning and multi-eye vision in digital twin environment | |
| CN110992487B (en) | Fast 3D map reconstruction device and reconstruction method for handheld aircraft fuel tank | |
| CN108828643B (en) | An indoor and outdoor seamless positioning system and method based on grey prediction model | |
| CN112268564B (en) | Unmanned aerial vehicle landing space position and attitude end-to-end estimation method | |
| Batstone et al. | Towards real-time time-of-arrival self-calibration using ultra-wideband anchors | |
| CN114329724A (en) | Building information model BIM updating method and system based on vision | |
| Wang et al. | Communication efficient, distributed relative state estimation in UAV networks | |
| CN115112123A (en) | Multi-mobile-robot cooperative positioning method and system based on vision-IMU fusion | |
| CN115022964A (en) | Indoor positioning radio map reconstruction method and system based on map signals | |
| CN118982678B (en) | Intelligent humanoid robot scene understanding method based on laser point cloud | |
| CN120356099A (en) | Image library construction and identification algorithm under large-scale incomplete multi-view and multi-mode scene | |
| CN119904765A (en) | A cross-view geolocation method based on image matching | |
| CN115052245A (en) | Unmanned aerial vehicle-assisted wireless sensor network node positioning method based on deep learning | |
| CN118279568B (en) | Multi-target identity judging method for distributed double-infrared sensor time sequence twin network | |
| CN117826141B (en) | Collaborative positioning method for distributed unmanned aerial vehicle group in complex environment | |
| CN117906609A (en) | A UAV positioning method integrating visual features and multi-sensor information | |
| CN114429515B (en) | A method, device and equipment for constructing point cloud map | |
| Wang et al. | SLAM-based cooperative calibration for optical sensors array with GPS/IMU aided | |
| Badalkhani et al. | An improved simultaneous localization and mapping for dynamic environments | |
| CN114972493B (en) | Method, device, cloud service and storage medium for predicting visual positioning reliability | |
| CN117649619B (en) | UAV visual navigation positioning recovery method, system, device and readable storage medium | |
| CN118521764B (en) | Unmanned aerial vehicle to ground target combined positioning method, device and system under refusing environment |
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |