CN112998697B

Movatterモバイル変換

Info

Publication number: CN112998697B
Application number: CN202110198589.0A
Authority: CN
Inventors: 刘晞
Original assignee: University of Electronic Science and Technology of China
Current assignee: University of Electronic Science and Technology of China
Priority date: 2021-02-22
Filing date: 2021-02-22
Publication date: 2022-06-14
Anticipated expiration: 2041-02-22
Also published as: CN112998697A

Abstract

The application discloses a falling injury degree prediction method, a system and a terminal based on skeleton data, wherein the skeleton data in the falling process are collected, and each part of a skeleton of a body is subjected to vector representation; establishing a fall-to-injury evaluation model, wherein a first ST-LSTM network is used for detecting an injured part when the user falls, and a second ST-LSTM network is used for evaluating injury degree; and inputting the processed data into the falling injury evaluation model to obtain the injury key part and the injury degree evaluation result thereof. The injury evaluation model effectively extracts the space-time characteristics of joints and body parts, and implements an attention mechanism through each layer, so that the influence of different falling modes on the body injury parts and the influence of different parts on the falling injury degree are better distinguished, and the falling injury degree evaluation precision is effectively improved. The method can be suitable for evaluating the injury degree of each high-risk part of the body during falling, so that the simultaneous evaluation of the injury degrees of a plurality of parts becomes possible.

Description

Translated fromChinese

一种基于骨架数据的跌倒伤害程度预测方法、系统及终端A fall injury degree prediction method, system and terminal based on skeleton data

技术领域technical field

本申请涉及跌倒伤害评估技术领域，具体涉及一种基于骨架数据的跌倒伤害程度预测方法、系统及终端。The present application relates to the technical field of fall injury assessment, and in particular to a method, system and terminal for predicting the degree of fall injury based on skeleton data.

背景技术Background technique

人体跌倒事件可能造成身体伤害，严重者甚至导致死亡。相关研究表明，跌倒事件造成伤害最多的前三个部位分别是头部、髋部和膝盖。跌倒损伤的严重程度与坠落高度、速度、动能和撞击位置的加速度等因素都有关。对跌倒造成的伤害程度进行评估，不仅有助于急救过程采取有效措施，而且对后续制定合理的护理措施也可提供参考。A human fall event can cause bodily injury, and even death in severe cases. Related research shows that the top three areas of injury caused by falls are the head, hip and knee. The severity of a fall injury is related to factors such as the height of the fall, velocity, kinetic energy, and acceleration at the impact location. Assessing the degree of injury caused by falls can not only help to take effective measures in the first aid process, but also provide a reference for the subsequent formulation of reasonable nursing measures.

目前临床上主要采用损伤严重程度量表进行跌倒损伤严重程度评定，如颅脑损伤标准、颅脑损伤模型和简易损伤量表，但主要是针对车祸、运动或行人事故的损伤程度评定。有研究采用仿真模拟实验，基于人体电子计算机断层扫描数据建立有限元模型对易受伤部位进行受力分析实验，评估在相应跌倒场景下可能会造成的伤害程度，例如Majumder使用骨盆-股骨-软组织复合体三维有限元模型评估向后跌倒对骨盆损伤的影响。At present, the injury severity scale is mainly used in clinical practice to assess the severity of fall injury, such as craniocerebral injury standard, craniocerebral injury model and simple injury scale, but it is mainly aimed at the assessment of the injury degree of traffic accident, sports or pedestrian accident. Some studies have used simulation experiments to establish a finite element model based on human computer tomography data to conduct force analysis experiments on vulnerable parts to evaluate the degree of injury that may be caused in corresponding fall scenarios. For example, Majumder uses pelvis-femur-soft tissue composite. A volumetric three-dimensional finite element model to assess the impact of backward falls on pelvic injury.

但是，传统量表评定方式基于主观判断，评定结果受评定师主观影响大，存在主观差异和通用性问题。而针对特定部位的有限元仿真方法，则只考虑单个部位受伤程度评定。However, the traditional scale assessment method is based on subjective judgment, and the assessment results are greatly influenced by the assessors, and there are subjective differences and general problems. The finite element simulation method for a specific part only considers the assessment of the injury degree of a single part.

发明内容SUMMARY OF THE INVENTION

本申请为了解决上述技术问题，提出了如下技术方案：In order to solve the above-mentioned technical problems, the present application proposes the following technical solutions:

第一方面，本申请实施例提供了一种基于骨架数据的跌倒伤害程度预测方法，所述方法包括：对跌倒过程中的骨架数据进行采集，其中采集时间根据跌倒持续时间进行设定；对身体骨骼的各个部位进行向量表示；建立跌倒伤害评估模型，所述伤害评估模型包括第一ST-LSTM网络和第二ST-LSTM网络，所述第一ST-LSTM网络用于对跌倒时受伤部位进行检测，所述第二ST-LSTM网络用于对伤害程度进行评估；将处理后的数据输入所述跌倒伤害评估模型，获取伤害关键部位及其伤害程度评估结果。In a first aspect, an embodiment of the present application provides a method for predicting a fall injury degree based on skeleton data, the method comprising: collecting skeleton data during a fall, wherein the collection time is set according to the fall duration; Each part of the skeleton is represented by a vector; a fall injury assessment model is established, and the injury assessment model includes a first ST-LSTM network and a second ST-LSTM network, and the first ST-LSTM network is used for the injured part during a fall. The second ST-LSTM network is used to evaluate the degree of injury; the processed data is input into the fall injury evaluation model, and the evaluation results of the key parts of the injury and the degree of injury are obtained.

采用上述实现方式，伤害评估模型有效提取关节和身体部位的时空特征，并通过各层实施注意力机制，更好地区分不同跌倒方式对身体伤害部位的影响，以及不同部位对跌倒伤害程度的影响，有效提高跌倒伤害程度评估精度。可适用于跌倒时身体各高危部位的伤害程度评估，使得同时评估多个部位伤害程度成为可能。Using the above implementation methods, the injury assessment model effectively extracts the spatiotemporal features of joints and body parts, and implements an attention mechanism through each layer to better distinguish the impact of different falling methods on the injured parts of the body and the impact of different parts on the degree of fall injury , which can effectively improve the accuracy of fall injury assessment. It can be applied to the assessment of the injury degree of various high-risk parts of the body during a fall, making it possible to assess the injury degree of multiple parts at the same time.

结合第一方面，在第一方面第一种可能的实现方式中，所述对跌倒过程中的骨架数据进行采集包括：根据预设频率在所述采集时间内采集人体骨骼节点序列数据；将所述人体骨骼节点序列数据进行帧采样，将输入序列分成预设数量的等长段，从每个等长段中随机选择一帧，获取训练样本。With reference to the first aspect, in a first possible implementation manner of the first aspect, the collecting skeleton data during a fall includes: collecting human skeleton node sequence data within the collection time according to a preset frequency; Frame sampling is performed on the human skeleton node sequence data, the input sequence is divided into a preset number of equal-length segments, and a frame is randomly selected from each equal-length segment to obtain a training sample.

结合第一方面，在第一方面第二种可能的实现方式中，第一ST-LSTM网络对跌倒时受伤部位进行检测包括：将所有节点信息进行聚合；对信息量进行归一化，得到不同关节点在不同时间点的隐藏状态信息的注意力权重；利用注意力权重，对每列进行加权求和，得到不同关节点的加权特征表示；将加权后的输出状态输入至全连接层，获得特征向量；将全连接层输出的分类数值计算网络的预测值，即受伤部位的概率分布。In combination with the first aspect, in the second possible implementation manner of the first aspect, the first ST-LSTM network detects the injured part during a fall, including: aggregating all node information; normalizing the amount of information to obtain different The attention weights of the hidden state information of the joint points at different time points; using the attention weights, each column is weighted and summed to obtain the weighted feature representation of different joint points; the weighted output state is input to the fully connected layer to obtain Feature vector; the classification value output by the fully connected layer is used to calculate the predicted value of the network, that is, the probability distribution of the injured part.

结合第一方面第二种可能的实现方式，在第一方面第三种可能的实现方式中，第二ST-LSTM网络对伤害程度进行评估包括：将第一ST-LSTM网络得到的隐藏表示及输出标签概率分布作为第二ST-LSTM网络的输入；对第二ST-LSTM网络实施注意力，通过上下文信息评估第二ST-LSTM网络在每个时空步骤的输入信息量；获得注意力权重概率向量及其加权输出表示；将第二ST-LSTM网络的加权输出通过全连接网络映射到类标签向量上获得特征向量，通过softmax分类器预测跌倒伤害程度；输出概率最大的作为伤害等级评估结果。In combination with the second possible implementation manner of the first aspect, in the third possible implementation manner of the first aspect, evaluating the degree of damage by the second ST-LSTM network includes: combining the hidden representation obtained by the first ST-LSTM network and the The output label probability distribution is used as the input of the second ST-LSTM network; attention is applied to the second ST-LSTM network, and the amount of input information of the second ST-LSTM network at each spatiotemporal step is evaluated through context information; the attention weight probability is obtained The vector and its weighted output representation; the weighted output of the second ST-LSTM network is mapped to the class label vector through the fully connected network to obtain the feature vector, and the degree of fall injury is predicted by the softmax classifier; the output probability is the largest as the injury level evaluation result.

结合第一方面，在第一方面第四种可能的实现方式中，还包括对所述跌倒伤害评估模型进行训练优化。With reference to the first aspect, in a fourth possible implementation manner of the first aspect, the method further includes training and optimizing the fall injury assessment model.

结合第一方面第四种可能的实现方式，在第一方面第五种可能的实现方式中，对所述跌倒伤害评估模型进行训练优化包括：将两层分类器联合进行训练，使用训练样本训练模型，采用负对数似然函数作为损失函数表示模型预测值与样本真实值的误差值；通过反向传播最小化损失函数，完成模型训练优化。With reference to the fourth possible implementation manner of the first aspect, in the fifth possible implementation manner of the first aspect, the training and optimization of the fall injury assessment model includes: jointly training two layers of classifiers, using training samples for training The model uses the negative log-likelihood function as the loss function to represent the error value between the predicted value of the model and the real value of the sample; the loss function is minimized through back propagation to complete the model training optimization.

第二方面，本申请实施例提供了一种基于骨架数据的跌倒伤害程度预测系统，所述系统包括：数据采集模块，用于对跌倒过程中的骨架数据进行采集，其中采集时间根据跌倒持续时间进行设定；数据处理模块，用于对身体骨骼的各个部位进行向量表示；模型建立模块，用于建立跌倒伤害评估模型，所述伤害评估模型包括第一ST-LSTM网络和第二ST-LSTM网络，所述第一ST-LSTM网络用于对跌倒时受伤部位进行检测，所述第二ST-LSTM网络用于对伤害程度进行评估；评估模块，用于将处理后的数据输入所述跌倒伤害评估模型，获取伤害关键部位及其伤害程度评估结果。In a second aspect, an embodiment of the present application provides a fall injury degree prediction system based on skeleton data. The system includes: a data collection module for collecting skeleton data during a fall, wherein the collection time is based on the fall duration. setting; a data processing module for vector representation of various parts of the body skeleton; a model building module for establishing a fall injury assessment model, the injury assessment model comprising a first ST-LSTM network and a second ST-LSTM network, the first ST-LSTM network is used to detect the injured part during a fall, and the second ST-LSTM network is used to evaluate the degree of injury; an evaluation module is used to input the processed data into the fall Injury evaluation model to obtain the evaluation results of the key parts of the injury and the degree of injury.

结合第二方面，在第二方面第一种可能的实现方式中，所述数据采集模块包括：采集单元，用于根据预设频率在所述采集时间内采集人体骨骼节点序列数据；第一获取单元，用于将所述人体骨骼节点序列数据进行帧采样，将输入序列分成预设数量的等长段，从每个等长段中随机选择一帧，获取训练样本。With reference to the second aspect, in a first possible implementation manner of the second aspect, the data collection module includes: a collection unit configured to collect human skeleton node sequence data within the collection time according to a preset frequency; The unit is configured to perform frame sampling on the human skeleton node sequence data, divide the input sequence into a preset number of equal-length segments, randomly select a frame from each equal-length segment, and obtain a training sample.

结合第二方面，在第二方面第二种可能的实现方式中，所述第一ST-LSTM网络包括：聚合单元，用于将所有节点信息进行聚合；第二获取单元，用于对信息量进行归一化，得到不同关节点在不同时间点的隐藏状态信息的注意力权重；第三获取单元，用于利用注意力权重，对每列进行加权求和，得到不同关节点的加权特征表示；第四获取单元，用于将加权后的输出状态输入至全连接层，获得特征向量；计算单元，用于将全连接层输出的分类数值计算网络的预测值，即受伤部位的概率分布。With reference to the second aspect, in a second possible implementation manner of the second aspect, the first ST-LSTM network includes: an aggregation unit for aggregating all node information; a second acquisition unit for Perform normalization to obtain the attention weight of the hidden state information of different joint points at different time points; the third acquisition unit is used to use the attention weight to perform weighted summation of each column to obtain the weighted feature representation of different joint points The fourth acquisition unit is used to input the weighted output state to the fully connected layer to obtain the feature vector; the calculation unit is used to calculate the predicted value of the network from the classification value output by the fully connected layer, that is, the probability distribution of the injured part.

结合第二方面第二种可能的实现方式，在第二方面第三种可能的实现方式中，所述第二ST-LSTM网络包括：第五获取单元，用于将第一ST-LSTM网络得到的隐藏表示及输出标签概率分布作为第二ST-LSTM网络的输入；权重评估单元，用于对第二ST-LSTM网络实施注意力，通过上下文信息评估第二ST-LSTM网络在每个时空步骤的输入信息量；第六获取单元，用于获得注意力权重概率向量及其加权输出表示；预测单元，用于将第二ST-LSTM网络的加权输出通过全连接网络映射到类标签向量上获得特征向量，通过softmax分类器预测跌倒伤害程度；输出单元，用于输出概率最大的作为伤害等级评估结果。In combination with the second possible implementation manner of the second aspect, in the third possible implementation manner of the second aspect, the second ST-LSTM network includes: a fifth obtaining unit, configured to obtain the first ST-LSTM network The hidden representation and output label probability distribution are used as the input of the second ST-LSTM network; the weight evaluation unit is used to implement attention to the second ST-LSTM network, and evaluate the second ST-LSTM network through context information at each spatio-temporal step The sixth acquisition unit is used to obtain the attention weight probability vector and its weighted output representation; the prediction unit is used to map the weighted output of the second ST-LSTM network to the class label vector through the fully connected network. The feature vector is used to predict the degree of fall injury through the softmax classifier; the output unit is used to output the highest probability as the injury level evaluation result.

第三方面，本申请实施例提供了一种终端，包括：处理器；用于存储所述处理器处理可执行指令的存储器；所述处理器执行第一方面或第一方面任一可能实现方式所述的基于骨架数据的跌倒伤害程度预测方法，对用户跌倒进行伤害等级评估。In a third aspect, an embodiment of the present application provides a terminal, including: a processor; a memory for storing executable instructions for processing by the processor; the processor executing the first aspect or any possible implementation manner of the first aspect The skeleton data-based fall injury degree prediction method evaluates the user's fall injury level.

附图说明Description of drawings

图1为本申请实施例提供的一种基于骨架数据的跌倒伤害程度预测方法的流程示意图；1 is a schematic flowchart of a method for predicting a fall injury degree based on skeleton data provided by an embodiment of the present application;

图2为本申请实施例提供的三维骨架示意图及三轴坐标系方向示意图；2 is a schematic diagram of a three-dimensional skeleton and a schematic diagram of the direction of a three-axis coordinate system provided by an embodiment of the present application;

图3为本申请实施例提供的伤害评估模型结构示意图；3 is a schematic structural diagram of an injury assessment model provided by an embodiment of the present application;

图4为本申请实施例提供的ST-LSTM网络示意图；4 is a schematic diagram of an ST-LSTM network provided by an embodiment of the present application;

图5为本申请实施例提供的一种基于骨架数据的跌倒伤害程度预测系统的示意图；5 is a schematic diagram of a system for predicting a fall injury degree based on skeleton data provided by an embodiment of the present application;

图6为本申请实施例提供的一种终端的示意图。FIG. 6 is a schematic diagram of a terminal according to an embodiment of the present application.

具体实施方式Detailed ways

下面结合附图与具体实施方式对本实施例进行阐述。The present embodiment will be described below with reference to the accompanying drawings and specific implementation manners.

本申请实施例提出一种基于人体骨架数据的跌倒伤害程度预测方法，用于评估跌倒事件发生时的高危部位及其伤害严重程度，可适用于日常活动场景中发生的各类跌倒，为急救和后续护理提供参考。The embodiment of the present application proposes a fall injury degree prediction method based on human skeleton data, which is used to evaluate the high-risk parts and the injury severity when a fall event occurs. Follow-up care provides reference.

参见图1，所述方法包括：Referring to Figure 1, the method includes:

S101，对跌倒过程中的骨架数据进行采集，其中采集时间根据跌倒持续时间进行设定。S101: Collect skeleton data during a fall, wherein the collection time is set according to the fall duration.

本申请基于Kinect v2传感器采集人体骨骼节点数据，采样频率30Hz。由于跌倒过程持续时间一般不超过2秒，将数据序列长度设置为2秒，得到60帧图像的骨骼节点数据。对采集的人体骨骼节点序列数据进行帧采样，将输入序列分成20个等长段，从每个等长段中随机选择一帧，得到20帧时序数据，作为训练样本。This application collects human skeleton node data based on the Kinect v2 sensor, and the sampling frequency is 30Hz. Since the duration of the fall process generally does not exceed 2 seconds, the length of the data sequence is set to 2 seconds, and the skeleton node data of 60 frames of images are obtained. The collected human skeleton node sequence data is framed, the input sequence is divided into 20 equal-length segments, and one frame is randomly selected from each equal-length segment to obtain 20 frames of time series data as training samples.

训练样本的标签，即跌倒造成的真实伤害程度，通过跌倒高危部位处关节的帧间速度划分伤害等级，计算如下：The label of the training sample, that is, the true degree of injury caused by the fall, is divided by the frame-to-frame speed of the joint at the high-risk part of the fall, and the calculation is as follows:

其中Δt为帧间间隔时间，ΔS是两个连续帧的关节点的欧式距离：where Δt is the interval time between frames, and ΔS is the Euclidean distance between the joint points of two consecutive frames:

其中

分别为n帧和n-1帧时关节p的三维坐标。in

are the 3D coordinates of joint p at frame n and frame n-1, respectively.

本申请主要考虑头部、髋部和膝盖三个高危部位伤害程序评估，伤害程度分别划分为四个等级，如表1所示。This application mainly considers the injury procedure evaluation of three high-risk parts of the head, hip and knee, and the degree of injury is divided into four grades, as shown in Table 1.

表1头部、髋部及膝盖的伤害等级Table 1 Injury levels for head, hip and knee

S102，对身体骨骼的各个部位进行向量表示。S102, performing vector representation on each part of the body skeleton.

本实施例用p表示关节点坐标，e表示连接两相邻关节点的边坐标向量。如图2所示，分别用p_rsh，p_lsh表示右肩和左肩关节点坐标，分别用p_lhjc，p_lkjc表示左髋关节坐标和左膝关节坐标，用p_rhjc，p_rkjc表示右髋关节坐标和右膝关节坐标,身体各部位向量表示如下：In this embodiment, p represents the coordinate of the joint point, and e represents the edge coordinate vector connecting two adjacent joint points. As shown in Figure 2, p_rsh and p_lsh are used to represent the coordinates of the right shoulder and left shoulder joints, respectively, p_lhjc and p_lkjc are used to represent the coordinates of the left hip joint and left knee joint, and p_rhjc and p_{rkjc are used} to represent the right hip joint. The coordinates and the coordinates of the right knee joint, the vectors of each part of the body are represented as follows:

身体躯干向量表示为：The body torso vector is represented as:

e_boby＝p_clav-p_casi (3)e_boby = p_clav -p_casi (3)

其中p_clav，p_casi分别表示左右肩关节连接起来的中点坐标和左右髋关节连接起来的中点坐标，计算如下：where p_clav and p_casi represent the coordinates of the midpoint connected by the left and right shoulder joints and the coordinates of the midpoint connected by the left and right hip joints, respectively, and are calculated as follows:

肩部向量为：The shoulder vector is:

e_sh＝p_rsh-p_lsh (6)e_sh = p_rsh -p_lsh (6)

左腿骨向量表示：Left leg bone vector representation:

e_lleg＝p_lhjc-p_lkjc (7)e_lleg = p_lhjc -p_lkjc (7)

右腿骨向量表示：The right leg bone vector representation:

e_rleg＝p_rhjc-p_rkjc (8)e_rleg = p_rhjc -p_rkjc (8)

S103，建立跌倒伤害评估模型，所述伤害评估模型包括第一ST-LSTM网络和第二ST-LSTM网络。S103, establish a fall injury assessment model, where the injury assessment model includes a first ST-LSTM network and a second ST-LSTM network.

如图3所示，本实施例采用双层ST-LSTM(时空长短期记忆)结构，基于人体骨架数据预测跌倒伤害部位并评估高危部位的伤害程度。整个网络模型由两层ST-LSTM网络组成，并为各层LSTM引入注意力机制。第一层ST-LSTM网络编码骨架序列，添加自注意力机制对不同关节的信息进行筛选，对跌倒时受伤部位进行检测，并将结果输入第二层ST-LSTM网络；第二层ST-LSTM网络在所有时空步骤上对输入实施注意力，以生成跌倒动作序列的加权注意力表示，对伤害程度进行评估。As shown in FIG. 3 , this embodiment adopts a double-layer ST-LSTM (space-time long short-term memory) structure to predict the injury site of falls and evaluate the injury degree of high-risk sites based on human skeleton data. The entire network model consists of a two-layer ST-LSTM network, and an attention mechanism is introduced for each layer of LSTM. The first layer of ST-LSTM network encodes the skeleton sequence, adds a self-attention mechanism to filter the information of different joints, detects the injured part when falling, and inputs the results into the second layer of ST-LSTM network; the second layer of ST-LSTM The network applies attention to the input over all spatiotemporal steps to generate a weighted attention representation of the fall action sequence, assessing the degree of injury.

在ST-LSTM模型中，同一帧框架中的骨架关节和身体部位被布置成链状，往空间方向传递，在不同帧框架上的相应关节以序列(时间方向)实施传递，如图4所示。本实施例关注M个关节点坐标和N个身体部位向量，则获取的骨架数据表示应为(M+N)×3的矩阵。加入时间序列T，将骨架节点数据坐标折叠，表示为一个T×3(M+N)的矩阵，节点间向量表示和节点表示一致。In the ST-LSTM model, the skeleton joints and body parts in the same frame are arranged in a chain shape, and transferred in the spatial direction, and the corresponding joints on different frames are transferred in a sequence (time direction), as shown in Figure 4 . In this embodiment, M joint point coordinates and N body part vectors are concerned, and the acquired skeleton data representation should be a matrix of (M+N)×3. The time series T is added, and the skeleton node data coordinates are folded and represented as a T×3(M+N) matrix, and the vector representation between nodes is consistent with the node representation.

每个ST-LSTM单元都有一个新的输入x_j,t(表示帧t时的关节或身体部位向量j)、同一关节点或身体向量在前一个时间步长的隐藏表示(h_j,t-1)，以及前一个关节在同一个时间步长中的隐藏表示(h_j-1,t)。ST-LSTM单元包含一个输入门i_j,t、输出门o_j,t和两个遗忘门，分别对应于上下文信息的两个输入通道：Each ST-LSTM cell has a new input x_j,t (representing the joint or body part vector j at frame t), a hidden representation of the same joint point or body vector at the previous time step (h_{j,t -1} ), and the hidden representation (h_j-1,t ) of the previous joint in the same time step. The ST-LSTM unit contains an input gate i_j,t , an output gate o_j,t and two forgetting gates, corresponding to the two input channels of context information:

代表时间域，

代表空间域。ST-LSTM公式如下：

represents the time domain,

represents the spatial domain. The ST-LSTM formula is as follows:

h_j,t＝o_j,t⊙tanh(c_j,t) (11)h_j,t =o_j,t ⊙tanh(c_j,t ) (11)

c_j,t和h_j,t分别表示单元状态和隐藏单元在时空步骤(j,t)处的表示，u_j,t表示调制后的输入，σ是sigmoid激活函数，W为仿射变换模型参数，⊙表示按元素乘法，tanh表示tanh激活函数。c_j,t and h_j,t denote the representation of the unit state and the hidden unit at the spatiotemporal step (j,t), respectively, u_j,t denotes the modulated input, σ is the sigmoid activation function, and W is the affine transformation model parameters, ⊙ means element-wise multiplication, and tanh means the tanh activation function.

第一层ST-LSTM网络对跌倒受伤部位进行预判，输入的时空步长(j,t)是在帧数t时关节点j的三维坐标，不同关节点的信息不同，因此采用注意力网络对关键节点进行自适应关注，隐藏状态h_j,t包含空间结构信息和时间动态信息，有利于指导关键节点的选择。首先将所有节点的信息聚合：The first layer of ST-LSTM network predicts the injured part of the fall. The input spatiotemporal step size (j, t) is the three-dimensional coordinate of the joint point j at the frame number t. The information of different joint points is different, so the attention network is used. Adaptive attention is made to key nodes, and the hidden state h_{j, t} contains spatial structure information and temporal dynamic information, which is beneficial to guide the selection of key nodes. First aggregate the information of all nodes:

其中W_e1为参数矩阵。利用Sigmoid对信息量进行归一化，得到不同关节点在不同时间点的隐藏状态信息的注意力权重：where_We1 is the parameter matrix. Use Sigmoid to normalize the amount of information to obtain the attention weights of the hidden state information of different joints at different time points:

α_j,t＝Sigmoid(U_Stanh(W_hh_j,t+W_qq_j,t+b_u)+b_s)α_j,t =Sigmoid(U_S tanh(W_h h_j,t +W_q q_j,t +b_u )+b_s )

W_h,W_q,U_S都是可学习的参数矩阵，b_s,b_u是偏差。W_h , W_q , and U_S are all learnable parameter matrices, and b_s ,_bu are biases.

利用注意力权重，对每列进行加权求和，得到不同关节点的加权特征表示：Using the attention weight, each column is weighted and summed to obtain the weighted feature representation of different joints:

将加权后的输出状态

输入至全连接层，获得特征向量b＝{b₁,b₂,b₃}。使用softmax将全连接层输出的分类数值计算网络的预测值，即受伤部位的概率分布：The weighted output state

Input to the fully connected layer to obtain the feature vector b={b₁ , b₂ , b₃ }. Use softmax to calculate the predicted value of the network from the classification value output by the fully connected layer, that is, the probability distribution of the injured part:

softmax层输出的最终结果

表示输入数据经过计算预测为第m个部位的概率，并用

表示在时空步骤(j,t)处预测的三个不同部位的伤害概率分布。The final result output by the softmax layer

Indicates the probability that the input data is predicted to be the mth position after calculation, and uses

Represents the predicted injury probability distribution for three different parts at the spatiotemporal step (j, t).

第二层ST-LSTM结合第一层ST-LSTM得到的伤害部位信息并实施注意力机制，学习跌倒部位对跌倒伤害程度的相关性，对跌倒伤害程度进行评估。The second layer of ST-LSTM combines the injury site information obtained by the first layer of ST-LSTM and implements an attention mechanism to learn the correlation between the fall site and the degree of fall injury, and to evaluate the degree of fall injury.

将第一层ST-LSTM得到的隐藏表示及输出标签概率分布作为第二层的输入：The hidden representation and output label probability distribution obtained by the first layer of ST-LSTM are used as the input of the second layer:

对第二层ST-LSTM网络实施注意力，通过上下文信息，评估第二层ST-LSTM在每个时空步骤的输入信息量大小。第二层ST-LSTM输出隐藏状态为H_j,t，得到注意力权重：Implement attention on the second-layer ST-LSTM network, and evaluate the input information size of the second-layer ST-LSTM at each spatiotemporal step through contextual information. The second layer of ST-LSTM outputs the hidden state as H_j,t and obtains the attention weight:

p_j,t＝tanh(H_j,t)p_j,t =tanh(H_j,t )

通过softmax函数计算注意力权重概率向量β_j,tCalculate the attention weight probability vector β_j,t by the softmax function

得到加权输出表示为：The weighted output is obtained as:

将第二层ST-LSTM的加权输出

通过全连接网络映射到类标签向量上获得特征向量y＝{y₁,y₂,y₃,y₄}，通过softmax分类器预测跌倒伤害程度：The weighted output of the second layer ST-LSTM

The feature vector y={y₁ , y₂ , y₃ , y₄ } is obtained by mapping the fully connected network to the class label vector, and the degree of fall injury is predicted by the softmax classifier:

最终输出概率最大的作为伤害等级评估结果。The final output probability is the largest as the damage level evaluation result.

本申请实施例还对所述跌倒伤害评估模型进行训练优化，具体地将两层分类器联合进行训练，使用训练样本训练模型，采用负对数似然函数作为损失函数表示模型预测值与样本真实值的误差值：The embodiment of the present application also performs training and optimization on the fall injury assessment model, specifically, jointly training two layers of classifiers, using training samples to train the model, and using the negative log-likelihood function as the loss function to represent the model prediction value and the sample real value. The error value of the value:

第一层ST-LSTM的目标函数表示为：The objective function of the first layer ST-LSTM is expressed as:

其中K为样本数，M为伤害部位类别数(本方案M＝3)，B表示输出向量，Where K is the number of samples, M is the number of injury site categories (M=3 in this scheme), B is the output vector,

B_im表示第i个样本的真实受伤部位是否为m，

表示模型预测第i个样本伤害部位为m的概率。B_im indicates whether the real injured part of the ith sample is m,

Indicates the probability that the model predicts that the ith sample injury site is m.

第二层ST-LSTM的损失函数表示为：The loss function of the second layer ST-LSTM is expressed as:

其中K为样本数，C为伤害等级(本方案C＝4)，Y表示输出向量，y_ij表示第i个样本的真实伤害等级是否为j，

表示模型预测第i个样本伤害等级为j的概率。Where K is the number of samples, C is the damage level (C=4 in this scheme), Y is the output vector, y_ij is whether the real damage level of the ith sample is j,

Indicates the probability that the model predicts the damage level of the ith sample to be j.

整个模型损失函数为:The entire model loss function is:

Q＝L(B)+L(Y)Q=L(B)+L(Y)

通过反向传播最小化损失函数，完成模型训练。Model training is completed by minimizing the loss function through backpropagation.

S104，将处理后的数据输入所述跌倒伤害评估模型，获取伤害关键部位及其伤害程度评估结果。S104, input the processed data into the fall injury assessment model, and obtain the assessment result of the key injury part and the injury degree thereof.

实时采集人体活动过程的骨架数据，计算跌倒过程关节最大帧间速度和各关节向量，输入至S103训练好的跌倒伤害评估模型，即可得到伤害关键部位及其伤害程度评估结果。The skeleton data of the human activity process is collected in real time, the maximum inter-frame velocity of the joints and each joint vector during the fall process are calculated, and input to the fall injury assessment model trained in S103, the key parts of the injury and the assessment results of the injury degree can be obtained.

本实施例提供一种基于人体骨架数据的跌倒伤害程度预测方法。整个网络结构由两层ST-LSTM网络组成，有效提取各关节和身体部位的时空特征；网络第一层通过注意力机制表现不同跌倒方式对跌倒伤害发生部位的影响，对跌倒受伤关键部位进行预判；网络第二层结合第一层得到的伤害部位分布信息实施注意力机制，学习跌倒部位对跌倒伤害程度的相关性，对高危部位的伤害程度进行评估。This embodiment provides a method for predicting a fall injury degree based on human skeleton data. The entire network structure consists of a two-layer ST-LSTM network, which can effectively extract the spatiotemporal features of each joint and body part; the first layer of the network expresses the impact of different falling methods on the site of fall injury through the attention mechanism, and predicts the key parts of the fall injury. The second layer of the network implements the attention mechanism based on the distribution information of the injured parts obtained from the first layer, learns the correlation between the falling parts and the degree of fall injury, and evaluates the injury degree of the high-risk parts.

与上述实施例提供的一种基于人体骨架数据的跌倒伤害程度预测方法相对应，本申请还提供了一种基于人体骨架数据的跌倒伤害程度预测系统的实施例，参见图5，基于人体骨架数据的跌倒伤害程度预测系统20包括：数据采集模块201、数据处理模块202、模型建立模块203和评估模块204。Corresponding to the method for predicting the degree of fall injury based on human skeleton data provided by the above-mentioned embodiment, the present application also provides an embodiment of a system for predicting the degree of fall injury based on human skeleton data. The fall injury degree prediction system 20 includes: adata acquisition module 201 , adata processing module 202 , amodel establishment module 203 and anevaluation module 204 .

所述数据采集模块201，用于对跌倒过程中的骨架数据进行采集，其中采集时间根据跌倒持续时间进行设定。所述数据处理模块202，用于对身体骨骼的各个部位进行向量表示。所述模型建立模块203，用于建立跌倒伤害评估模型，所述伤害评估模型包括第一ST-LSTM网络和第二ST-LSTM网络，所述第一ST-LSTM网络用于对跌倒时受伤部位进行检测，所述第二ST-LSTM网络用于对伤害程度进行评估。所述评估模块204，用于将处理后的数据输入训练后所述跌倒伤害评估模型，获取伤害关键部位及其伤害程度评估结果。Thedata collection module 201 is used to collect skeleton data during a fall, wherein the collection time is set according to the fall duration. Thedata processing module 202 is used for vector representation of each part of the body skeleton. Themodel establishment module 203 is used to establish a fall injury assessment model, the injury assessment model includes a first ST-LSTM network and a second ST-LSTM network, and the first ST-LSTM network is used for the injury part during a fall. For detection, the second ST-LSTM network is used to assess the degree of injury. Theevaluation module 204 is configured to input the processed data into the fall injury evaluation model after training, and obtain the evaluation results of the key parts of the injury and the injury degree thereof.

进一步地，所述数据采集模块201包括：采集单元和第一获取单元。Further, thedata collection module 201 includes: a collection unit and a first acquisition unit.

采集单元，用于根据预设频率在所述采集时间内采集人体骨骼节点序列数据。第一获取单元，用于将所述人体骨骼节点序列数据进行帧采样，将输入序列分成预设数量的等长段，从每个等长段中随机选择一帧，获取训练样本。The collection unit is configured to collect human skeleton node sequence data within the collection time according to a preset frequency. The first acquisition unit is configured to perform frame sampling on the human skeleton node sequence data, divide the input sequence into a preset number of equal-length segments, and randomly select a frame from each equal-length segment to obtain a training sample.

模型建立模块203建立的跌倒伤害评估模型包括第一ST-LSTM网络和第二ST-LSTM网络。The fall injury assessment model established by themodel establishment module 203 includes a first ST-LSTM network and a second ST-LSTM network.

所述第一ST-LSTM网络包括：聚合单元，用于将所有节点信息进行聚合。第二获取单元，用于对信息量进行归一化，得到不同关节点在不同时间点的隐藏状态信息的注意力权重。第三获取单元，用于利用注意力权重，对每列进行加权求和，得到不同关节点的加权特征表示。第四获取单元，用于将加权后的输出状态输入至全连接层，获得特征向量。计算单元，用于将全连接层输出的分类数值计算网络的预测值，即受伤部位的概率分布。The first ST-LSTM network includes: an aggregation unit for aggregating all node information. The second obtaining unit is used to normalize the amount of information to obtain the attention weights of the hidden state information of different joint points at different time points. The third acquisition unit is used to perform weighted summation on each column by using the attention weight to obtain the weighted feature representation of different joint points. The fourth obtaining unit is used for inputting the weighted output state to the fully connected layer to obtain a feature vector. The calculation unit is used to calculate the predicted value of the network, that is, the probability distribution of the injured part, with the classification value output by the fully connected layer.

所述第二ST-LSTM网络包括：第五获取单元，用于将第一ST-LSTM网络得到的隐藏表示及输出标签概率分布作为第二ST-LSTM网络的输入。权重评估单元，用于对第二ST-LSTM网络实施注意力，通过上下文信息评估第二ST-LSTM网络在每个时空步骤的输入信息量。第六获取单元，用于获得注意力权重概率向量及其加权输出表示。预测单元，用于将第二ST-LSTM网络的加权输出通过全连接网络映射到类标签向量上获得特征向量，通过softmax分类器预测跌倒伤害程度。输出单元，用于输出概率最大的作为伤害等级评估结果。The second ST-LSTM network includes: a fifth acquisition unit, configured to use the hidden representation and output label probability distribution obtained by the first ST-LSTM network as the input of the second ST-LSTM network. The weight evaluation unit is used to implement attention to the second ST-LSTM network, and evaluate the input information amount of the second ST-LSTM network at each spatiotemporal step through context information. The sixth obtaining unit is used to obtain the attention weight probability vector and its weighted output representation. The prediction unit is used to map the weighted output of the second ST-LSTM network to the class label vector through the fully connected network to obtain the feature vector, and predict the degree of fall injury through the softmax classifier. The output unit is used to output the highest probability as the damage level evaluation result.

本申请实施例还提供了一种终端的实施例，参见图6，终端30包括处理器301、存储器302和通信接口303。This embodiment of the present application further provides an embodiment of a terminal. Referring to FIG. 6 , the terminal 30 includes aprocessor 301 , a memory 302 and acommunication interface 303 .

在图6中，处理器301、存储器302和通信接口303可以通过总线相互连接；总线可以分为地址总线、数据总线、控制总线等。为便于表示，图6中仅用一条粗线表示，但并不表示仅有一根总线或一种类型的总线。In FIG. 6, theprocessor 301, the memory 302 and thecommunication interface 303 can be connected to each other through a bus; the bus can be divided into an address bus, a data bus, a control bus, and the like. For ease of presentation, only one thick line is used in FIG. 6, but it does not mean that there is only one bus or one type of bus.

处理器301通常是控制终端30的整体功能，例如终端30的启动、以及终端30启动后对跌倒过程中的骨架数据进行采集，其中采集时间根据跌倒持续时间进行设定；对身体骨骼的各个部位进行向量表示；建立跌倒伤害评估模型，所述伤害评估模型包括第一ST-LSTM网络和第二ST-LSTM网络，所述第一ST-LSTM网络用于对跌倒时受伤部位进行检测，所述第二ST-LSTM网络用于对伤害程度进行评估；将处理后的数据输入所述跌倒伤害评估模型，获取伤害关键部位及其伤害程度评估结果。Theprocessor 301 usually controls the overall functions of the terminal 30, such as the startup of the terminal 30 and the collection of skeleton data during the fall process after the terminal 30 is started, wherein the collection time is set according to the fall duration; Perform vector representation; establish a fall injury assessment model, the injury assessment model includes a first ST-LSTM network and a second ST-LSTM network, and the first ST-LSTM network is used to detect the injured part during a fall, the said The second ST-LSTM network is used to evaluate the degree of injury; the processed data is input into the fall injury evaluation model to obtain the evaluation results of the key parts of the injury and the degree of injury.

处理器301可以是通用处理器，例如，中央处理器(英文：central processingunit，缩写：CPU)，网络处理器(英文：network processor，缩写：NP)或者CPU和NP的组合。处理器也可以是微处理器(MCU)。处理器还可以包括硬件芯片。上述硬件芯片可以是专用集成电路(ASIC)，可编程逻辑器件(PLD)或其组合。上述PLD可以是复杂可编程逻辑器件(CPLD)，现场可编程逻辑门阵列(FPGA)等。Theprocessor 301 may be a general-purpose processor, for example, a central processing unit (English: central processing unit, abbreviation: CPU), a network processor (English: network processor, abbreviation: NP), or a combination of CPU and NP. The processor may also be a microprocessor (MCU). The processor may also include hardware chips. The above-mentioned hardware chip may be an application specific integrated circuit (ASIC), a programmable logic device (PLD) or a combination thereof. The above-mentioned PLD may be a complex programmable logic device (CPLD), a field programmable logic gate array (FPGA), or the like.

存储器302被配置为存储计算机可执行指令以支持终端30数据的操作。存储器301可以由任何类型的易失性或非易失性存储设备或者它们的组合实现，如静态随机存取存储器(SRAM)，电可擦除可编程只读存储器(EEPROM)，可擦除可编程只读存储器(EPROM)，可编程只读存储器(PROM)，只读存储器(ROM)，磁存储器，快闪存储器，磁盘或光盘。Memory 302 is configured to store computer-executable instructions to support operation of terminal 30 data.Memory 301 may be implemented by any type of volatile or non-volatile storage device or combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read Only Memory (EEPROM), Erasable Programmable Read Only Memory (EEPROM), Programmable Read Only Memory (EPROM), Programmable Read Only Memory (PROM), Read Only Memory (ROM), Magnetic Memory, Flash Memory, Magnetic or Optical Disk.

启动终端30后，处理器301和存储器302上电，处理器301读取并执行存储在存储器302内的计算机可执行指令，以完成上述的基于骨架数据的跌倒伤害程度预测方法实施例中的全部或部分步骤。After starting the terminal 30, theprocessor 301 and the memory 302 are powered on, and theprocessor 301 reads and executes the computer-executable instructions stored in the memory 302, so as to complete all the above-mentioned embodiments of the method for predicting the degree of fall injury based on skeleton data. or part of the steps.

通信接口303用于终端30传输数据，例如实现与Kinect v2传感器的通信等。通信接口303包括有线通信接口，还可以包括无线通信接口。其中，有线通信接口包括USB接口、Micro USB接口，还可以包括以太网接口。无线通信接口可以为WLAN接口，蜂窝网络通信接口或其组合等。Thecommunication interface 303 is used for the terminal 30 to transmit data, for example, to realize the communication with the Kinect v2 sensor. Thecommunication interface 303 includes a wired communication interface, and may also include a wireless communication interface. Wherein, the wired communication interface includes a USB interface, a Micro USB interface, and may also include an Ethernet interface. The wireless communication interface may be a WLAN interface, a cellular network communication interface, a combination thereof, or the like.

在一个示意性实施例中，本申请实施例提供的终端30还包括电源组件，电源组件为终端30的各种组件提供电力。电源组件可以包括电源管理系统，一个或多个电源，及其他与为终端30生成、管理和分配电力相关联的组件。In an exemplary embodiment, the terminal 30 provided in this embodiment of the present application further includes a power supply component, and the power supply component provides power for various components of the terminal 30 . Power components may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power to terminal 30 .

通信组件，通信组件被配置为便于终端30和其他设备之间有线或无线方式的通信。终端30可以接入基于通信标准的无线网络，如WiFi，4G或5G，或它们的组合。通信组件经由广播信道接收来自外部广播管理系统的广播信号或广播相关信息。通信组件还包括近场通信(NFC)模块，以促进短程通信。例如，在NFC模块可基于射频识别(RFID)技术，红外数据协会(IrDA)技术，超宽带(UWB)技术，蓝牙(BT)技术和其他技术来实现。The communication component is configured to facilitate wired or wireless communication between the terminal 30 and other devices. The terminal 30 may access a wireless network based on a communication standard, such as WiFi, 4G or 5G, or a combination thereof. The communication component receives broadcast signals or broadcast related information from an external broadcast management system via a broadcast channel. The communication assembly also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

在一个示意性实施例中，终端30可以被一个或多个应用专用集成电路(ASIC)、数字信号处理器(DSP)、数字信号处理设备(DSPD)、可编程逻辑器件(PLD)、现场可编程门阵列(FPGA)或其他电子元件实现。In one illustrative embodiment, terminal 30 may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable Programmable Gate Array (FPGA) or other electronic component implementation.

需要说明的是，在本文中，诸如“第一”和“第二”等之类的关系术语仅仅用来将一个实体或者操作与另一个实体或操作区分开来，而不一定要求或者暗示这些实体或操作之间存在任何这种实际的关系或者顺序。而且，术语“包括”、“包含”或者其任何其他变体意在涵盖非排他性的包含，从而使得包括一系列要素的过程、方法、物品或者设备不仅包括那些要素，而且还包括没有明确列出的其他要素，或者是还包括为这种过程、方法、物品或者设备所固有的要素。在没有更多限制的情况下，由语句“包括一个……”限定的要素，并不排除在包括所述要素的过程、方法、物品或者设备中还存在另外的相同要素。It should be noted that, in this document, relational terms such as "first" and "second" etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply these There is no such actual relationship or sequence between entities or operations. Moreover, the terms "comprising", "comprising" or any other variation thereof are intended to encompass a non-exclusive inclusion such that a process, method, article or device that includes a list of elements includes not only those elements, but also includes not explicitly listed or other elements inherent to such a process, method, article or apparatus. Without further limitation, an element qualified by the phrase "comprising a..." does not preclude the presence of additional identical elements in a process, method, article or apparatus that includes the element.

当然，上述说明也并不仅限于上述举例，本申请未经描述的技术特征可以通过或采用现有技术实现，在此不再赘述；以上实施例及附图仅用于说明本申请的技术方案并非是对本申请的限制，如来替代，本申请仅结合并参照优选的实施方式进行了详细说明，本领域的普通技术人员应当理解，本技术领域的普通技术人员在本申请的实质范围内所做出的变化、改型、添加或替换都不脱离本申请的宗旨，也应属于本申请的权利要求保护范围。Of course, the above description is not limited to the above examples, and the technical features not described in this application can be realized by or using existing technologies, and will not be repeated here; the above embodiments and drawings are only used to illustrate the technical solutions of this application, not It is a limitation of the present application. If it is replaced, the present application is only described in detail with reference to the preferred embodiments. Those of ordinary skill in the art should understand that those of ordinary skill in the art can make Changes, modifications, additions or substitutions do not depart from the purpose of the present application, and should also belong to the protection scope of the claims of the present application.