CN113017829B - Preoperative planning method, system, medium and device for total knee arthroplasty based on deep learning - Google Patents

Abstract

The invention relates to a preoperative planning method, a preoperative planning system, preoperative planning media and preoperative planning equipment for total knee arthroplasty based on deep learning. The method comprises the following steps: a step of medical image data processing based on deep learning, wherein a three-dimensional image of a bone structure is obtained through medical image processing, and a key axis, a key anatomical site and key anatomical parameters are identified and marked; skeletal structures include femur, tibia, fibula, and patella; the key axis comprises a femur anatomical axis, a femur mechanical axis, a tibia anatomical axis and a tibia mechanical axis; anatomical parameters include tibial and distal femoral angles; a step of visual simulation matching, in which a three-dimensional prosthesis model is subjected to simulation matching with a three-dimensional femur and a three-dimensional tibia, and a simulation matching effect is observed in real time; and when the simulation matching effect meets the operation requirement, the simulation matching is considered to be completed. The method and the system realize automatic segmentation of bone blocks and/or identification and measurement of key axes, key anatomical sites and key anatomical parameters in total knee arthroplasty based on deep learning.

Description

Translated fromChinese

一种基于深度学习的全膝关节置换术的术前规划方法、系统、介质和设备A preoperative planning method, system, and method for total knee arthroplasty based on deep learningMedia and Equipment

技术领域technical field

本发明涉及全膝关节置换技术领域，尤其涉及一种基于深度学习 的全膝关节置换术的术前规划方法、系统、介质和设备。The present invention relates to the technical field of total knee replacement, in particular to a preoperative planning method, system, medium and equipment for total knee replacement based on deep learning.

背景技术Background technique

膝关节是全身主要的承重关节，长期负重且运动量大，属于容易 受伤的部位之一，再加上当前社会人口老龄化不断加剧，这些因素使 得膝关节疾病的发生率较高。全膝关节置换术(Total Knee Arthroplasty， TKA)是一种较成熟的治疗膝关节疾病的技术，能够有效恢复膝关节 功能，极大地提高患者的生活质量。术前规划为医生提供技术支持，便于医生制定手术方案、观察下肢力线。如何更快、更精准地实现术 前规划是具有现实意义的研究方向。The knee joint is the main load-bearing joint of the whole body. It is one of the parts that are prone to injury due to long-term load-bearing and heavy exercise. In addition, the aging of the current social population continues to increase. These factors lead to a high incidence of knee joint diseases. Total knee arthroplasty (Total Knee Arthroplasty, TKA) is a relatively mature technique for the treatment of knee joint diseases, which can effectively restore knee joint function and greatly improve the quality of life of patients. Preoperative planning provides technical support for doctors, making it easier for doctors to formulate surgical plans and observe the line of lower limbs. How to achieve preoperative planning faster and more accurately is a research direction with practical significance.

发明内容Contents of the invention

(一)要解决的技术问题(1) Technical problems to be solved

本发明的一个目的是提供一种全膝关节置换手术术前规划方法；An object of the present invention is to provide a method for preoperative planning of total knee replacement surgery;

本发明的另一个目的是一种全膝关节置换手术术前规划。Another object of the present invention is a preoperative planning for total knee replacement surgery.

(二)技术方案(2) Technical solution

为了实现上述目的，本发明提供了如下技术方案：In order to achieve the above object, the present invention provides the following technical solutions:

1、一种基于深度学习的全膝关节置换术的术前规划方法，所述术 前规划方法基于患者下肢医学图像数据，所述术前规划方法包括：1, a kind of preoperative planning method of the total knee arthroplasty based on deep learning, described preoperative planning method is based on patient lower limb medical image data, and described preoperative planning method comprises:

基于深度学习的医学图像数据处理的步骤，通过所述医学图像处 理获得骨骼结构的三维影像、识别标记出关键轴线、关键解剖位点和 关键解剖参数；所述骨骼结构包括股骨、胫骨、腓骨和髌骨；所述关 键轴线包括股骨解剖轴、股骨机械轴、胫骨解剖轴和胫骨机械轴；所 述关键解剖参数包括胫股角和远端股骨角；Steps of medical image data processing based on deep learning, through the medical image processing to obtain three-dimensional images of skeletal structures, identify and mark key axes, key anatomical sites and key anatomical parameters; the skeletal structures include femur, tibia, fibula and Patella; said key axis comprises femoral anatomical axis, femoral mechanical axis, tibial anatomical axis and tibial mechanical axis; said key anatomical parameter comprises tibiofemoral angle and distal femoral angle;

可视化模拟匹配的步骤，将三维假体与三维股骨和三维胫骨进行 模拟匹配，实时观察模拟匹配效果；当模拟匹配效果符合手术要求时， 视为完成模拟匹配。The steps of visual simulation matching are simulated matching of the 3D prosthesis with the 3D femur and 3D tibia, and the simulated matching effect is observed in real time; when the simulated matching effect meets the surgical requirements, the simulated matching is considered complete.

2、根据技术方案1所述的方法，2. According to the method described in technical scheme 1,

所述医学图像数据处理的步骤包括骨骼三维影像重建的步骤；基 于深度学习的图像分割的步骤；识别标记关键轴线、关键解剖位点和 关键解剖参数的步骤。The steps of the medical image data processing include the steps of bone three-dimensional image reconstruction; the steps of image segmentation based on deep learning; the steps of identifying and marking key axes, key anatomical sites and key anatomical parameters.

3、根据技术方案1所述的方法，3. According to the method described in technical scheme 1,

所述医学图像数据处理的步骤包括骨骼三维影像重建的步骤；图 像分割的步骤；基于深度学习的识别标记关键轴线、关键解剖位点和 关键解剖参数的步骤。The steps of the medical image data processing include the steps of bone three-dimensional image reconstruction; the steps of image segmentation; the steps of identifying and marking key axes, key anatomical sites and key anatomical parameters based on deep learning.

4、根据技术方案1所述的方法，4. According to the method described in technical scheme 1,

所述医学图像数据处理的步骤包括骨骼三维影像重建的步骤；基 于深度学习的图像分割的步骤；基于深度学习的识别标记关键轴线、 关键解剖位点和关键解剖参数的步骤。The steps of the medical image data processing include the steps of bone three-dimensional image reconstruction; the steps of image segmentation based on deep learning; the steps of identifying and marking key axes, key anatomical sites and key anatomical parameters based on deep learning.

5、根据技术方案2或4所述的方法，5. According to the method described in technical scheme 2 or 4,

基于深度学习的图像分割包括：Image segmentation based on deep learning includes:

构建下肢医学图像数据库：获取下肢医学图像数据集，手动标注 出股骨、胫骨、腓骨和髌骨区域；将所述数据集划分为训练集和测试 集；将未标注前的医学图像数据转换成第一格式的图片并保存，将标 注后的数据转换成第二格式的图片并保存；Construct the medical image database of lower extremities: obtain the medical image data set of lower extremities, and manually mark out the femur, tibia, fibula, and patella regions; divide the data set into a training set and a test set; convert the unlabeled medical image data into the first Format pictures and save them, convert the labeled data into pictures in the second format and save them;

建立分割神经网络模型；优选地，所述分割神经网络模型包括粗 分割神经网络和精确分割申请网络；进一步优选地，所述粗分割神经 网络选自FCN、SegNet、Unet、3D-Unet、Mask-RCNN、空洞卷积、 ENet、CRFasRNN、PSPNet、ParseNet、RefineNet、ReSeg、LSTM-CF、DeepMask、DeepLabV1、DeepLabV2、DeepLabV3中的任一种或多种； 进一步优选地，所述精确分割神经网络为EfficientDet、SimCLR、PointRend中的任一种或多种；Establish a segmentation neural network model; preferably, the segmentation neural network model includes a coarse segmentation neural network and an accurate segmentation application network; further preferably, the coarse segmentation neural network is selected from FCN, SegNet, Unet, 3D-Unet, Mask- Any one or more of RCNN, Atrous Convolution, ENet, CRFasRNN, PSPNet, ParseNet, RefineNet, ReSeg, LSTM-CF, DeepMask, DeepLabV1, DeepLabV2, DeepLabV3; Further preferably, the precise segmentation neural network is EfficientDet Any one or more of , SimCLR, PointRend;

模型训练：利用训练集对分割神经网络模型进行训练，并利用测 试集进行测试；和Model training: using the training set to train the segmented neural network model and using the test set for testing; and

利用训练好的分割神经网络模型进行分割。Use the trained segmentation neural network model for segmentation.

6、根据技术方案5所述的方法，其特征在于，6. The method according to technical solution 5, characterized in that,

所述Unet神经网络包括n个上采样层和n个下采样层；The Unet neural network includes n upsampling layers and n downsampling layers;

每个上采样层包括上采样操作层和卷积层；Each upsampling layer includes an upsampling operation layer and a convolutional layer;

每个下采样层包括卷积层和池化层。Each downsampling layer consists of a convolutional layer and a pooling layer.

7、根据技术方案6所述的方法，其特征在于，7. The method according to technical solution 6, characterized in that,

n的取值优选为2-8，进一步优选为3-6，更优选为4-5。The value of n is preferably 2-8, more preferably 3-6, more preferably 4-5.

8、根据技术方案6或7所述的方法，其特征在于，8. The method according to technical solution 6 or 7, characterized in that,

每个上采样层包括1个上采样操作层和2个卷积层，其中的卷积 层中的卷积核大小为3*3，上采样操作层中的卷积核大小为2*2，每个 上采样层中的卷积核个数为512，256，256，128。Each upsampling layer includes 1 upsampling operation layer and 2 convolution layers, where the convolution kernel size in the convolution layer is 3*3, and the convolution kernel size in the upsampling operation layer is 2*2, The number of convolution kernels in each upsampling layer is 512, 256, 256, 128.

9、根据技术方案8所述的方法，其特征在于，9. The method according to technical solution 8, characterized in that,

每个下采样层包括2个卷积层和1个池化层，其中的卷积层中的 卷积核大小为3*3，池化层中的卷积核大小为2*2，每个卷积层中的卷 积核的个数为128，256，256，512。Each downsampling layer includes 2 convolutional layers and 1 pooling layer, where the convolution kernel size in the convolution layer is 3*3, and the convolution kernel size in the pooling layer is 2*2, each The number of convolution kernels in the convolution layer is 128, 256, 256, 512.

10、根据技术方案5至9任一项所述的方法，其特征在于，10. The method according to any one of technical solutions 5 to 9, characterized in that,

将所述数据集按照7:3的比例划分为训练集和测试集。The data set is divided into training set and test set according to the ratio of 7:3.

11、根据技术方案6至9任一项所述的方法，其特征在于，11. The method according to any one of technical solutions 6 to 9, characterized in that,

最后一次上采样结束后设有一个dropout层，droupout率设置为 0.5-0.7；和/或A dropout layer is set after the last upsampling with a dropout rate set to 0.5-0.7; and/or

所有的卷积层后面设有激活层，激活层使用的激活函数为relu函 数。All convolutional layers are followed by an activation layer, and the activation function used by the activation layer is the relu function.

12、根据技术方案5至11任一项所述的方法，其特征在于，12. The method according to any one of technical solutions 5 to 11, characterized in that,

所述训练按照如下方法进行：The training is carried out as follows:

粗分割：将训练集送入粗分割神经网络进行训练；训练过程中， 数据标签的背景像素值设置为0，股骨为1，胫骨为2，腓骨为3，髌 骨为4，训练的batch_size为6，学习率设置为1e-4，优化器使用Adam 优化器，使用的损失函数为DICE loss；可选地，根据训练过程中损失 函数的变化，调整训练批次的大小；Coarse segmentation: Send the training set to the coarse segmentation neural network for training; during the training process, the background pixel value of the data label is set to 0, the femur is 1, the tibia is 2, the fibula is 3, the patella is 4, and the training batch_size is 6 , the learning rate is set to 1e-4, the optimizer uses the Adam optimizer, and the loss function used is DICE loss; optionally, the size of the training batch is adjusted according to the change of the loss function during the training process;

精确分割：送入精确分割神经网络进行精确分割；初始过程包括， 先使用双线性插值上采样粗分割的预测结果，再在特征图中选定多个 最不确定的点，然后计算多个点的特征表示并且预测labels；重复所述 初始过程，直到上采样到需要的大小；Precise Segmentation: Feed into the precise segmentation neural network for precise segmentation; the initial process includes first using bilinear interpolation to upsample the prediction results of the rough segmentation, and then selecting multiple most uncertain points in the feature map, and then calculating multiple The feature representation of the point and predict the labels; repeat the initial process until the upsampling reaches the required size;

优选地，选定置信度为0.5的点作为最不确定的点；Preferably, the selected confidence level is 0.5 as the most uncertain point;

优选地，点的特征通过双线性插值Bilinear计算。Preferably, the features of the points are calculated by bilinear interpolation.

13、根据技术方案1至12任一项所述的方法，其特征在于，13. The method according to any one of technical solutions 1 to 12, characterized in that,

所述下肢医学图像数据为CT扫描数据。The lower limb medical image data is CT scan data.

14、根据技术方案1至13任一项所述的方法，其特征在于，14. The method according to any one of technical solutions 1 to 13, characterized in that,

所述基于深度学习的识别标记关键轴线、关键解剖位点和关键解 剖参数的步骤包括：The steps of identifying and marking key axes, key anatomical sites and key anatomical parameters based on deep learning include:

识别关键解剖位点；优选地，利用MTCNN、locnet、Pyramid Residual Module、Densenet、hourglass、resnet、SegNet、Unet、R-CNN、Fast R-CNN、 Faster R-CNN、R-FCN、SSD中的任一种或多种神经网络模型识别关键 点；Identify key anatomical sites; preferably, using any of MTCNN, locnet, Pyramid Residual Module, Densenet, hourglass, resnet, SegNet, Unet, R-CNN, Fast R-CNN, Faster R-CNN, R-FCN, SSD One or more neural network models identify key points;

利用关键解剖位点获得关键轴线；和Use key anatomical sites to obtain key axes; and

测量关键解剖参数。Measure key anatomical parameters.

15、根据技术方案14所述的方法，其特征在于，15. The method according to technical solution 14, characterized in that,

所述识别关键解剖位点的步骤包括：The step of identifying key anatomical sites includes:

构建数据库：获取下肢医学图像数据集，手动标定关键解剖位点； 将所述数据集划分为训练集和测试集，优选按照7:3的比例划分。Constructing a database: acquiring a lower limb medical image data set, and manually marking key anatomical sites; dividing the data set into a training set and a test set, preferably at a ratio of 7:3.

建立识别神经网络模型；Establish a recognition neural network model;

模型训练：利用训练集对识别神经网络模型进行训练，并利用测 试集进行测试；Model training: use the training set to train the recognition neural network model, and use the test set to test;

利用训练好的神经网络模型进行关键解剖位点的识别。Use the trained neural network model to identify key anatomical sites.

16、根据技术方案1至15任一项所述的方法，16. The method according to any one of technical solutions 1 to 15,

所述三维假体包括三维股骨假体和三维胫骨假体；优选地，还包 括胫骨垫；和The three-dimensional prosthesis includes a three-dimensional femoral prosthesis and a three-dimensional tibial prosthesis; preferably, also includes a tibial pad; and

所述模拟匹配包括：The simulated matching includes:

(i)将三维股骨假体植入股骨，将三维胫骨假体植入胫骨；优选 地，还包括将胫骨垫植入假体间隙；(i) implant the three-dimensional femoral prosthesis into the femur, and implant the three-dimensional tibial prosthesis into the tibia; preferably, also include implanting the tibial pad into the prosthesis gap;

(ii)选择三维股骨假体和三维胫骨假体，选择模拟手术条件；(ii) Select a three-dimensional femoral prosthesis and a three-dimensional tibial prosthesis, and select simulated surgical conditions;

(iii)根据三维假体与骨骼的匹配关系智能截骨，观察三维假体与 骨骼的模拟匹配效果；(iii) Intelligent osteotomy according to the matching relationship between the three-dimensional prosthesis and the bone, and observe the simulated matching effect of the three-dimensional prosthesis and the bone;

(iv)若是模拟匹配效果不符合手术需求，则重复步骤ii至步骤iii， 直至模拟匹配效果符合手术要求。(iv) If the simulated matching effect does not meet the surgical requirements, repeat step ii to step iii until the simulated matching effect meets the surgical requirements.

17、根据技术方案16所述的方法，17. According to the method described in technical solution 16,

在步骤ii中：In step ii:

选择三维股骨假体包括选择股骨假体类型和/或股骨假体型号和/ 或三维空间位置；Selecting a three-dimensional femoral prosthesis includes selecting a femoral prosthesis type and/or a femoral prosthesis model and/or a three-dimensional space position;

选择三维胫骨假体包括选择胫骨假体类型和/或胫骨假体型号和/ 或三维空间位置；和/或Selecting a three-dimensional tibial prosthesis includes selecting a tibial prosthesis type and/or a tibial prosthesis size and/or a three-dimensional position; and/or

选择模拟手术条件包括选择股骨手术参数和/或选择胫骨手术参数； 所述股骨手术参数包括股骨远端截骨量、股骨后髁截骨量、内外旋角、 内外翻角和股骨假体屈曲角；所述胫骨手术参数包括胫骨截骨量、内 外旋角、内外翻角和后倾角。Selection of simulated operation conditions includes selecting femoral surgery parameters and/or selecting tibial surgery parameters; said femoral surgery parameters include distal femoral osteotomy amount, posterior femoral condyle osteotomy amount, internal and external rotation angle, valgus angle and femoral prosthesis flexion angle ; The tibial surgical parameters include tibial osteotomy, internal and external rotation angle, valgus and posterior inclination angle.

18、根据技术方案1至17任一项所述的方法，18. The method according to any one of technical solutions 1 to 17,

所述骨骼结构可进行任意组合显示和/或进行透明度的切换和/或 进行图像缩放和/或图像旋转和/或图像移动；可选地，所述透明度包括 透明和不透明两种。The skeletal structure can be displayed in any combination and/or can be switched and/or image zoomed and/or image rotated and/or image moved; optionally, the transparency includes both transparent and opaque.

19、根据技术方案1至18任一项所述的方法，19. The method according to any one of technical solutions 1 to 18,

在如下一个或多个状态下观察模拟匹配效果：Observe simulated matching effects in one or more of the following states:

(a)截骨状态或非截骨状态；(a) Osteotomy state or non-osteotomy state;

(b)骨骼透明状态或不透明状态；(b) bone transparent state or opaque state;

(c)腓骨显示或不显示状态。(c) Fibula showing or not showing state.

20、根据技术方案1至19任一项所述的方法，20. The method according to any one of technical solutions 1 to 19,

21、根据技术方案1至20任一项所述的方法，其特征在于，21. The method according to any one of technical solutions 1 to 20, characterized in that,

所述关键解剖位点还包括股骨内髁凹点、股骨外髁最高点、股骨 内外后髁最低点、胫骨平台内侧低点和外侧高点、后交叉韧带中点和 胫骨结节内侧缘点、股骨远端最低点中任一种或多种；所述关键轴线 还包括通髁线、后髁连线、胫骨膝关节线、股骨矢状轴、股骨膝关节 线中的任一种或多种；Described key anatomical site also comprises femoral medial condyle concave point, femoral lateral condyle highest point, femoral medial posterior condyle lowest point, tibial plateau medial low point and lateral high point, posterior cruciate ligament midpoint and tibial tuberosity medial edge point, Any one or more of the lowest points of the distal end of the femur; the key axis also includes any one or more of the condylar line, the posterior condyle line, the tibial knee joint line, the femoral sagittal axis, and the femoral knee joint line ;

优选地，所述关键解剖参数还包括股骨后髁角。Preferably, the key anatomical parameters also include the posterior femoral condyle angle.

22、根据技术方案20所述的方法，22. According to the method described in technical solution 20,

在透明度为不透明的状态下标记出关键轴线。Keylines are marked with opaque transparency.

23、根据技术方案1至22任一项所述的方法，23. The method according to any one of technical solutions 1 to 22,

通过所述医学图像处理获得骨骼结构的三维影像和二维影像；所 述二维影像包括横断面影像、矢状面影像和冠状面影像；进一步优选 地，横断面影像、矢状面影像和冠状面影像三轴联动。Three-dimensional images and two-dimensional images of bone structures are obtained through the medical image processing; the two-dimensional images include cross-sectional images, sagittal plane images and coronal plane images; further preferably, cross-sectional images, sagittal plane images and coronal plane images The three-axis linkage of the surface image.

24、根据技术方案2至23任一项所述的方法，24. The method according to any one of technical solutions 2 to 23,

在标记关键轴线后，观察关键轴线是否对位，并将不对位的关键 轴线手动标记；优选地，独立显示出股骨或胫骨，进一步优选地，调 整股骨或胫骨的观察角度，然后再进行关键轴线手动标记。After marking the key axis, observe whether the key axis is aligned, and manually mark the key axis that is not aligned; preferably, display the femur or tibia independently, and more preferably, adjust the viewing angle of the femur or tibia, and then perform the key axis Manually tagged.

25、根据技术方案1至24任一项所述的方法，25. The method according to any one of technical solutions 1 to 24,

所述方法还包括可视化术后模拟的步骤，用于模拟全膝关节置换 术的术后肢体运动情况。The method also includes the step of visualizing a postoperative simulation for simulating postoperative limb motion in total knee arthroplasty.

26、根据技术方案1至25任一项所述的方法，26. The method according to any one of technical solutions 1 to 25,

所述方法还包括将符合手术需求的模拟匹配数据导出，形成术前 规划报告的步骤。The method also includes the step of exporting the simulated matching data meeting the surgical requirements to form a preoperative planning report.

27、一种基于深度学习的全膝关节置换的术前规划系统，所述术 前规划系统包括：27. A preoperative planning system for total knee replacement based on deep learning, the preoperative planning system comprising:

医学图像数据处理模块，用于通过医学图像处理获得骨骼结构的 三维影像、识别标记出关键轴线、关键解剖位点和关键解剖参数；所 述骨骼结构包括股骨、胫骨、腓骨和髌骨；所述关键解剖位点包括股 骨髓腔的不同层面上的中心点、胫骨髓腔的不同层面上的中心点、髋 关节中心点、膝关节中心点、髁间棘的中心点、踝关节中心点；所述关键轴线包括股骨解剖轴、股骨机械轴、胫骨解剖轴和胫骨机械轴； 所述关键解剖参数包括胫股角和远端股骨角；The medical image data processing module is used to obtain a three-dimensional image of the bone structure through medical image processing, identify and mark key axes, key anatomical sites and key anatomical parameters; the bone structure includes femur, tibia, fibula and patella; the key Anatomical sites include the center point on different levels of the femoral medullary cavity, the center point on different levels of the tibial medullary cavity, the center point of the hip joint, the center point of the knee joint, the center point of the intercondylar spine, and the center point of the ankle joint; Key axes include femoral anatomical axis, femoral mechanical axis, tibial anatomical axis and tibial mechanical axis; Described key anatomical parameters include tibiofemoral angle and distal femoral angle;

模拟匹配模块，用于将三维假体与三维股骨和三维胫骨进行模拟 匹配，实时观察模拟匹配效果；和The simulated matching module is used to simulate the matching of the three-dimensional prosthesis with the three-dimensional femur and the three-dimensional tibia, and observe the simulated matching effect in real time; and

显示模块：用于显示骨骼结构的三维影像、关键轴线、关键解剖 位点、关键解剖参数和模拟匹配效果。Display module: used to display 3D images of bone structure, key axes, key anatomical points, key anatomical parameters and simulation matching effects.

28、根据技术方案27所述的术前规划系统，28. According to the preoperative planning system described in technical solution 27,

所述医学图像数据处理模块包括：The medical image data processing module includes:

三维重建单元；3D reconstruction unit;

图像分割单元；Image segmentation unit;

识别标记单元。Identify marked units.

29、根据技术方案27所述的术前规划系统，29. According to the preoperative planning system described in technical solution 27,

所述术前规划系统还包括：The preoperative planning system also includes:

图像组合模块，用于将骨骼结构任意组合；The image combination module is used to combine the bone structure arbitrarily;

图像透明度切换模块，用于切换骨骼结构的透明度；Image transparency switching module, used to switch the transparency of bone structure;

图像缩放模块，用于缩放骨骼结构的三维影像和/或二维影像；Image scaling module, for scaling the three-dimensional image and/or the two-dimensional image of the bone structure;

图像旋转模块，用于将图像按照任意轴进行旋转；和/或an image rotation module for rotating the image along any axis; and/or

图像移动模块，用于将图像进行移动。The image moving module is used to move the image.

30、根据技术方案27至29任一项所述的术前规划系统，30. The preoperative planning system according to any one of technical solutions 27 to 29,

所述术前规划系统还包括：The preoperative planning system also includes:

数据导入模块；Data import module;

术后模拟模块；和/或post-operative simulation module; and/or

数据导出模块。Data export module.

31、一种设备，其特征在于，包括：31. A device, characterized by comprising:

一个或多个处理器；one or more processors;

存储装置，用于存储一个或多个程序；storage means for storing one or more programs;

当所述一个或多个程序被所述一个或多个处理器执行，使得所述 一个或多个处理器实现技术方案1至26任一项所述的方法。When the one or more programs are executed by the one or more processors, the one or more processors are made to implement the method described in any one of technical solutions 1 to 26.

32、一种计算机可读存储介质，其上存储有计算机程序，其特征 在于，32. A computer-readable storage medium having a computer program stored thereon, characterized in that,

所述程序被处理器执行时实现技术方案1至26任一项所述的方法。When the program is executed by the processor, the method described in any one of technical solutions 1 to 26 is implemented.

(三)有益效果(3) Beneficial effects

本发明的上述技术方案具有如下优点：The technical scheme of the present invention has the following advantages:

本发明提供的术前规划方法和系统基于深度学习实现了股骨、胫 骨、腓骨和髌骨的自动分割。与现有技术相比(如，目前国内外针对 骨关节CT图像分割方法大部分需要在每一张CT图像中进行手动定位 或手动分割，费时费力，且效率低)，本发明提高了分割效率以及准确 率。本发明提供的术前规划方法和系统基于深度学习实现了关键轴线 和关键解剖参数自动识别和测量。The preoperative planning method and system provided by the present invention realize automatic segmentation of femur, tibia, fibula and patella based on deep learning. Compared with the prior art (for example, at present, most domestic and foreign CT image segmentation methods for bone joints require manual positioning or manual segmentation in each CT image, which is time-consuming, laborious, and inefficient), the present invention improves the segmentation efficiency and accuracy. The preoperative planning method and system provided by the present invention realize automatic identification and measurement of key axes and key anatomical parameters based on deep learning.

本发明提供的术前规划系统智能高效，医生学习时间短，无需经 过长时间、大体量手术的培训即可掌握；而且，成本较低，无需复杂 设备。The preoperative planning system provided by the present invention is intelligent and efficient, with short learning time for doctors, and can be mastered without long-term and large-volume surgery training; moreover, the cost is low and no complicated equipment is required.

利用本发明提供的术前规划方法和系统可以在术前确定植入假体 的尺寸和位置，并且能虚拟测试假体是否达到性能要求，以便最优化 关节面重建和假体位置的确定；为医生提供技术支持，使外科手术更 准确、更安全；促进外科手术向智能化、精准化、微创化方向发展。Using the preoperative planning method and system provided by the present invention, the size and position of the implanted prosthesis can be determined before the operation, and whether the prosthesis meets the performance requirements can be tested virtually, so as to optimize the joint surface reconstruction and the determination of the position of the prosthesis; for Doctors provide technical support to make surgical operations more accurate and safer; promote the development of surgical operations in the direction of intelligence, precision, and minimal invasiveness.

附图说明Description of drawings

图1示意性地示出了本发明提供的术前规划方法的流程图；Fig. 1 schematically shows the flowchart of the preoperative planning method provided by the present invention;

图2示意性地示出了本发明提供的术前规划系统的框图；Fig. 2 schematically shows the block diagram of the preoperative planning system provided by the present invention;

图3是分割后四类骨骼结构组合显示的三维影像，a和b分别为不 同角度下的三维影像；Fig. 3 is the three-dimensional image that four kinds of skeletal structure combinations display after segmentation, and a and b are three-dimensional images under different angles respectively;

图4是只显示股骨时的股骨三维影像，a和b分别为不同角度下的 三维影像；Fig. 4 is the three-dimensional image of the femur when only showing the femur, and a and b are three-dimensional images under different angles respectively;

图5是只显示胫骨时的胫骨三维影像，a和b分别为不同角度下的 三维影像；Fig. 5 is the three-dimensional image of the tibia when only showing the tibia, and a and b are the three-dimensional images under different angles respectively;

图6是胫骨平台放大后的三维影像；Figure 6 is an enlarged three-dimensional image of the tibial plateau;

图7是标记关键轴线后的结果图；Figure 7 is the result graph after marking the key axis;

图8是截骨前的模拟匹配的界面(显像效果为透明)；Fig. 8 is the interface of simulation matching before osteotomy (imaging effect is transparent);

图9是截骨后的模拟匹配的界面(显像效果为不透明)；Fig. 9 is the simulated matching interface after osteotomy (imaging effect is opaque);

图10是在不同角度下的图像，a为股骨，b为胫骨；Figure 10 is the images at different angles, a is the femur, b is the tibia;

图11是术后模拟的结果图；Fig. 11 is the result figure of postoperative simulation;

图12示意性地示出了本发明提供的设备的结构图。Fig. 12 schematically shows the structure diagram of the device provided by the present invention.

图中：101：医学图像数据处理模块；201：模拟匹配模块；301： 显示模块；401：数据导入模块；501：可视化术后模拟模块。In the figure: 101: medical image data processing module; 201: simulation matching module; 301: display module; 401: data import module; 501: visual postoperative simulation module.

具体实施方式Detailed ways

为使本发明的目的、技术方案和优点更加清楚，下面将结合本发 明实施例，对本发明的技术方案进行清楚、完整地描述。显然，所描 述的实施例是本发明的一部分实施例，而不是全部的实施例。基于本 发明中的实施例，本领域普通技术人员在没有做出创造性劳动的前提 下所获得的所有其他实施例，都属于本发明保护的范围。In order to make the object, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Apparently, the described embodiments are some, not all, embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

﹤第一方面﹥﹤The first aspect﹥

本发明在第一方面提供了一种基于深度学习的全膝关节置换术的 术前规划方法，所述术前规划方法基于患者下肢医学图像数据，参考 图1，本发明提供的这一术前规划方法包括如下步骤：In the first aspect, the present invention provides a preoperative planning method for total knee arthroplasty based on deep learning. The preoperative planning method is based on medical image data of the patient's lower limbs. Referring to FIG. The planning method includes the following steps:

S1、基于深度学习的医学图像数据处理的步骤，通过所述医学图 像处理获得骨骼结构的三维影像、识别标记出关键轴线、关键解剖位 点和关键解剖参数；所述骨骼结构包括股骨、胫骨、腓骨和髌骨；所述关键轴线包括股骨解剖轴、股骨机械轴、胫骨解剖轴和胫骨机械轴； 所述关键解剖位点包括股骨髓腔的不同层面上的中心点、胫骨髓腔的 不同层面上的中心点、髋关节中心点、膝关节中心点、髁间棘的中心 点、踝关节中心点；所述关键解剖参数包括胫股角和远端股骨角；S1, the step of medical image data processing based on deep learning, through the medical image processing to obtain a three-dimensional image of bone structure, identify and mark key axes, key anatomical sites and key anatomical parameters; the bone structure includes femur, tibia, fibula and patella; the key axes include femoral anatomical axis, femoral mechanical axis, tibial anatomical axis and tibial mechanical axis; Center point, hip joint center point, knee joint center point, center point of intercondylar spine, ankle joint center point; Described key anatomy parameter comprises tibiofemoral angle and distal end femoral angle;

S2、可视化模拟匹配的步骤，将三维假体与三维股骨和三维胫骨 进行模拟匹配，实时观察模拟匹配效果；当模拟匹配效果符合手术要 求时，视为完成模拟匹配。S2. The step of visual simulation matching, simulate and match the 3D prosthesis with the 3D femur and 3D tibia, and observe the simulated matching effect in real time; when the simulated matching effect meets the surgical requirements, the simulated matching is considered as completed.

关于S1：About S1:

继续参考图1，所述医学图像数据处理的步骤包括骨骼三维影像重 建的步骤；图像分割的步骤；识别标记出关键轴线、关键解剖位点和 关键解剖参数的步骤。需要说明的是，本发明对医学图像数据处理步 骤所包括的三个步骤没有顺序上的限定。在获得患者的医学图像数据后，可以先进行三维影像重建，再进行分割、识别标记，也可以先进 行分割，再进行三维影像重建、识别标记，本发明在此对可以实现的 顺序不一一列举说明。Continuing to refer to Fig. 1, the step of described medical image data processing comprises the step of bone three-dimensional image reconstruction; The step of image segmentation; The step of identifying and marking key axis, key anatomical site and key anatomical parameters. It should be noted that the present invention does not limit the order of the three steps included in the medical image data processing steps. After obtaining the medical image data of the patient, the three-dimensional image reconstruction can be carried out first, and then the segmentation and identification marks can be carried out, or the segmentation can be carried out first, and then the three-dimensional image reconstruction and identification marks can be carried out. List instructions.

通过三维影像重建获得股骨、胫骨、腓骨和髌骨这四类骨骼的三 维影像。无需说明的是，若是在分割之前进行三维影像重建，则获得 的三维影像中的骨骼结构是存在连结的。通过图像分割至少能够获得 股骨、胫骨、腓骨和髌骨这四类骨骼结构，分割出的这四类骨骼结构 无连结。通过识别标记的步骤至少识别标记出股骨和胫骨上的股骨解剖轴、股骨机械轴、胫骨解剖轴和胫骨机械轴，至少获得胫股角和远 端股骨角这些关键解剖参数。Three-dimensional images of four types of bones, femur, tibia, fibula, and patella, were obtained through three-dimensional image reconstruction. Needless to say, if three-dimensional image reconstruction is performed before segmentation, the bone structure in the obtained three-dimensional image is connected. At least four bone structures of femur, tibia, fibula, and patella can be obtained through image segmentation, and these four types of bone structures are not connected. By identifying and marking at least the femoral anatomical axis, the femoral mechanical axis, the tibial anatomical axis, and the tibial mechanical axis on the femur and the tibia, at least key anatomical parameters such as tibiofemoral angle and distal femoral angle are obtained.

本发明在图像分割的步骤和/或识别标记的步骤可通过深度学习技 术实现AI图像分割和/或AI识别标记关键轴线、关键解剖位点和关键解剖参数。In the step of image segmentation and/or the step of identifying markers, the present invention can realize AI image segmentation and/or AI identifying and marking key axes, key anatomical sites and key anatomical parameters through deep learning technology.

关于基于深度学习的图像分割：About image segmentation based on deep learning:

在一些优选的实施方式中，所述基于深度学习的图像分割的步骤 包括：In some preferred embodiments, the step of the image segmentation based on deep learning includes:

构建下肢医学图像数据库：获取下肢医学图像数据集，手动标注 出股骨、胫骨、腓骨和髌骨区域；将所述数据集划分为训练集和测试 集，优选按照7:3的比例进行划分；将未标注前的医学图像数据(如二维横断面影像dicom格式的数据)转换成第一格式(如jpg格式)的图 片并保存，将标注后的数据转换成第二格式(如png格式)的图片并 保存；需要说明的是，第一格式和第二格式不相同；Construct the lower limb medical image database: obtain the lower limb medical image data set, and manually mark out the femur, tibia, fibula and patella regions; divide the data set into a training set and a test set, preferably in a ratio of 7:3; Convert the medical image data before labeling (such as data in dicom format of two-dimensional cross-sectional images) into pictures in the first format (such as jpg format) and save them, and convert the labeled data into pictures in the second format (such as png format) and save it; it should be noted that the first format and the second format are not the same;

建立分割神经网络模型；Build a segmentation neural network model;

模型训练：利用训练集对分割神经网络模型进行训练，并利用测 试集进行测试；和Model training: using the training set to train the segmented neural network model and using the test set for testing; and

利用训练好的分割神经网络模型进行分割。Use the trained segmentation neural network model for segmentation.

关于分割神经网络模型：About segmenting neural network models:

在一些优选的实施方式中，分割神经网络模型包括粗分割神经网 络和精确分割申请网络；所述粗分割神经网络选自FCN、SegNet、Unet、 3D-Unet、Mask-RCNN、空洞卷积、ENet、CRFasRNN、PSPNet、ParseNet、 RefineNet、ReSeg、LSTM-CF、DeepMask中的任一种，优选为Unet； 所述精确分割神经网络为EfficientDet、SimCLR、PointRend。In some preferred embodiments, the segmentation neural network model includes a coarse segmentation neural network and an accurate segmentation application network; the coarse segmentation neural network is selected from FCN, SegNet, Unet, 3D-Unet, Mask-RCNN, hole convolution, ENet , CRFasRNN, PSPNet, ParseNet, RefineNet, ReSeg, LSTM-CF, DeepMask, preferably Unet; the precise segmentation neural network is EfficientDet, SimCLR, PointRend.

以所述分割神经网络模型为Unet+PointRend为例，利用Unet神经 网络进行粗分割，利用PointRend神经网络进行精确分割。具体地，所 述Unet神经网络包括n个上采样层和n个下采样层；每个上采样层包 括上采样操作层和卷积层；每个下采样层包括卷积层和池化层。n的取 值优选为2-8，进一步优选为3-6，更优选为4-5。每个上采样层优选包 括1个上采样操作层和2个卷积层，其中的卷积层中的卷积核大小为 3*3，上采样操作层中的卷积核大小为2*2，每个上采样层中的卷积核 个数为512，256，256，128。每个下采样层优选包括2个卷积层和1 个池化层，其中的卷积层中的卷积核大小为3*3，池化层中的卷积核大 小为2*2，每个卷积层中的卷积核的个数为128，256，256，512。Taking the segmentation neural network model as Unet+PointRend as an example, the Unet neural network is used for rough segmentation, and the PointRend neural network is used for precise segmentation. Specifically, the Unet neural network includes n upsampling layers and n downsampling layers; each upsampling layer includes an upsampling operation layer and a convolutional layer; each downsampling layer includes a convolutional layer and a pooling layer. The value of n is preferably 2-8, more preferably 3-6, more preferably 4-5. Each upsampling layer preferably includes 1 upsampling operation layer and 2 convolution layers, wherein the convolution kernel size in the convolution layer is 3*3, and the convolution kernel size in the upsampling operation layer is 2*2 , the number of convolution kernels in each upsampling layer is 512, 256, 256, 128. Each downsampling layer preferably includes 2 convolution layers and 1 pooling layer, wherein the convolution kernel size in the convolution layer is 3*3, and the convolution kernel size in the pooling layer is 2*2, each The number of convolution kernels in each convolutional layer is 128, 256, 256, 512.

在一些优选的实施方式中，最后一次上采样结束后设有一个dropout层，droupout率设置为0.5-0.7。In some preferred embodiments, a dropout layer is provided after the last upsampling, and the dropout rate is set to 0.5-0.7.

在一些优选的实施方式中，所有的卷积层后面设有激活层，激活 层使用的激活函数为relu函数。In some preferred embodiments, all convolutional layers are followed by an activation layer, and the activation function used by the activation layer is a relu function.

关于模型训练：About model training:

训练过程中，将训练集全部送入Unet神经网络进行训练；训练过 程中，数据标签的背景像素值设置为0，股骨为1，胫骨为2，腓骨为 3，髌骨为4，训练的batch_size为6，学习率设置为1e-4，优化器使用 Adam优化器，使用的损失函数为DICE loss，可以根据训练过程中损失函数的变化，调整训练批次的大小；During the training process, all the training sets are sent to the Unet neural network for training; during the training process, the background pixel value of the data label is set to 0, the femur is 1, the tibia is 2, the fibula is 3, and the patella is 4. The batch_size of the training is 6. The learning rate is set to 1e-4, the optimizer uses the Adam optimizer, and the loss function used is DICE loss. The size of the training batch can be adjusted according to the change of the loss function during the training process;

完成粗分割后，送入PointRend神经网络进行精确分割；初始过程 包括，先使用双线性插值上采样粗分割的预测结果，再在特征图中选 定多个最不确定的点，然后计算多个点的特征表示并且预测labels；重复所述初始过程，直到上采样到需要的大小；After the rough segmentation is completed, it is sent to the PointRend neural network for precise segmentation; the initial process includes first using bilinear interpolation to upsample the prediction results of the rough segmentation, and then selecting multiple most uncertain points in the feature map, and then calculating the multiple The feature representation of points and predict labels; repeat the initial process until the upsampling reaches the required size;

优选地，选定置信度为0.5的点作为最不确定的点；Preferably, the selected confidence level is 0.5 as the most uncertain point;

优选地，点的特征通过双线性插值Bilinear计算。Preferably, the features of the points are calculated by bilinear interpolation.

关于基于深度学习的识别标记：Regarding deep learning-based recognition markers:

在一些优选的实施方式中，所述基于深度学习的识别标记的步骤 包括：In some preferred embodiments, the step of identifying mark based on deep learning comprises:

识别关键解剖位点；Identify key anatomical sites;

利用关键解剖位点获得关键轴线；和Use key anatomical sites to obtain key axes; and

测量关键解剖参数。Measure key anatomical parameters.

关于识别关键解剖位点：Regarding the identification of key anatomical sites:

本发明需要识别的关键解剖位点包括股骨髓腔的不同层面上的中 心点、胫骨髓腔的不同层面上的中心点、髋关节中心点、膝关节中心 点、髁间棘的中心点、踝关节中心点，在一些优选的实施方式中，还 包括股骨内髁凹点、股骨外髁最高点、股骨内外后髁最低点、胫骨平 台内侧低点和外侧高点、后交叉韧带中点和胫骨结节内侧缘点、股骨远端最低点等。The key anatomical sites that need to be identified in the present invention include the center point on different levels of the femoral medullary cavity, the center point on different levels of the tibial medullary cavity, the center point of the hip joint, the center point of the knee joint, the center point of the intercondylar spine, the ankle The joint center point, in some preferred embodiments, also includes the concave point of the medial femoral condyle, the highest point of the lateral femoral condyle, the lowest point of the medial and medial posterior femoral condyle, the medial low point and the lateral high point of the tibial plateau, the midpoint of the posterior cruciate ligament and the tibial The medial edge of the tubercle, the lowest point of the distal femur, etc.

识别关键解剖位点的步骤包括：Steps to identify key anatomical sites include:

构建数据库：获取下肢医学图像数据集，手动标定关键解剖位点； 将所述数据集划分为训练集和测试集，优选按照7:3的比例划分。Constructing a database: acquiring a lower limb medical image data set, and manually marking key anatomical sites; dividing the data set into a training set and a test set, preferably at a ratio of 7:3.

建立关键点识别神经网络模型：所述识别神经网络模型为MTCNN、 locnet、Pyramid Residual Module、Densenet、hourglass、resnet、SegNet、 Unet、R-CNN、Fast R-CNN、Faster R-CNN、R-FCN、SSD中的任一 种或多种。Establish a key point recognition neural network model: the recognition neural network model is MTCNN, locnet, Pyramid Residual Module, Densenet, hourglass, resnet, SegNet, Unet, R-CNN, Fast R-CNN, Faster R-CNN, R-FCN , SSD in any one or more.

以hourglass为例，其网络细节包括：Taking hourglass as an example, its network details include:

首先Conv层和Max Pooling层用于将特征的分辨率进行缩放；First, the Conv layer and the Max Pooling layer are used to scale the resolution of the feature;

每一个Max Pooling(降采样)处，网络进行分叉，上下两路在不同 尺度空间进行卷积操作提取特征；At each Max Pooling (down-sampling), the network is bifurcated, and the upper and lower channels perform convolution operations in different scale spaces to extract features;

得到最低分辨率特征后，网络开始进行upsampling，并逐渐结合 不同尺度的特征信息；对较低分辨率可以采用最近邻上采样方式，将 两个不同的特征集进行逐元素相加；After obtaining the lowest resolution features, the network starts upsampling, and gradually combines feature information of different scales; for lower resolutions, the nearest neighbor upsampling method can be used to add two different feature sets element by element;

整个hourglass是对称的，获取低分辨率特征过程中每有一个网络 层，则在上采样的过程中相应低就会有一个对应网络层；The entire hourglass is symmetrical, and for each network layer in the process of obtaining low-resolution features, there will be a corresponding network layer in the process of upsampling;

得到hourglass网络模块输出后，再采用两个连续的1×1Conv层 进行处理，得到最终的网络输出；输出为heatmaps的集合，每一个 heatmap表征了关键点在每个像素点存在的概率。After obtaining the output of the hourglass network module, two consecutive 1×1 Conv layers are used for processing to obtain the final network output; the output is a set of heatmaps, and each heatmap represents the probability of key points existing in each pixel.

模型训练：利用训练集对识别神经网络模型进行训练，并利用测 试集进行测试。Model training: use the training set to train the recognition neural network model, and use the test set to test.

以hourglass为例，在进行训练时，输入像素值为0-255的正投影 图像和label.txt，可以通过每张图片的名称找到互相对应的点的坐标； 若直接用目标点的坐标进行学习，神经网络需要自行将空间位置转换 为坐标，是一种比较难学习的训练方式，所以将这些点生成高斯图， 用heatmap去监督，即网络的输出是一个与输入大小相同尺寸的特征 图，在检测点的位置为1，其他位置为0，多个点的检测就输出多个通 道的特征图；网络使用Adam优化，学习率为1e-5，batch_size为4，损失函数使用L2正则化，可以根据训练过程中损失函数的变化，调整 训练批次的大小，得到关键点位的坐标值。Taking hourglass as an example, when training, input an orthographic projection image with a pixel value of 0-255 and label.txt, and you can find the coordinates of the corresponding points through the name of each image; if you directly use the coordinates of the target point for learning , the neural network needs to convert the spatial position into coordinates by itself, which is a relatively difficult training method to learn, so these points are generated into a Gaussian map, and the heatmap is used to supervise, that is, the output of the network is a feature map of the same size as the input, The position of the detection point is 1, and the other positions are 0. The detection of multiple points will output the feature map of multiple channels; the network uses Adam optimization, the learning rate is 1e-5, the batch_size is 4, and the loss function uses L2 regularization. According to the change of the loss function during the training process, the size of the training batch can be adjusted to obtain the coordinate values of the key points.

利用训练好的神经网络模型进行关键解剖位点的识别。Use the trained neural network model to identify key anatomical sites.

关于利用关键解剖位点获得关键轴线：Regarding the use of key anatomical points to obtain key axes:

对于股骨解刨轴，可以通过拟合股骨髓腔的不同层面上的中心点 而得到。同样地，对于胫骨解刨轴，可以通过拟合胫骨髓腔的不同层 面上的中心点而得到。拟合的方法可以为最小二乘法、梯度下降、高 斯牛顿、列-马算法中的任一种。For the femoral anatomy axis, it can be obtained by fitting the center points on different levels of the femoral medullary cavity. Similarly, for the tibial anatomy axis, it can be obtained by fitting the center points on different levels of the tibial medullary cavity. The fitting method can be any one of the least square method, gradient descent, Gauss-Newton, and Lemma algorithm.

对于其它种类的关键轴线，可以利用确定的两个端点而得到。如， 股骨机械轴的两个端点-髋关节中心点和膝关节中心点-已被识别出来， 可以通过这两点确定股骨机械轴线。For other kinds of critical axes, it can be obtained by using the determined two endpoints. For example, two endpoints of the femoral mechanical axis - the center of the hip and the center of the knee - have been identified, from which the femoral mechanical axis can be determined.

测量关键解剖参数：Measure key anatomical parameters:

在该步骤可以自动测量的关键解剖参数包括胫股角、远端股骨角， 还可以自动测量出股骨后髁角。Key anatomical parameters that can be automatically measured in this step include the tibiofemoral angle, the distal femoral angle, and the posterior femoral condyle angle can also be automatically measured.

本发明通过所述医学图像处理不仅可以获得骨骼结构的三维影像， 还可以获得二维影像；所述二维影像包括横断面影像、矢状面影像和 冠状面影像，并且横断面影像、矢状面影像和冠状面影像可以三轴联 动。The present invention can obtain not only three-dimensional images of bone structures but also two-dimensional images through the medical image processing; the two-dimensional images include transverse images, sagittal images and coronal images, and transverse images, sagittal images The plane image and the coronal plane image can be linked in three axes.

对于本发明来说，通过医学图像处理获得的骨骼结构的三维影像 可以进行任意组合，从而实现骨骼结构灵活多样的显示方式。显示的 情形包括如下任一种：只显示股骨；只显示胫骨；只显示腓骨；只显 示髌骨；同时显示股骨和胫骨；同时显示股骨和腓骨；同时显示股骨 和髌骨；同时显示胫骨和腓骨；同时显示胫骨和髌骨；同时显示腓骨和髌骨；同时显示股骨、胫骨和腓骨；同时显示股骨、胫骨和髌骨； 同时显示股骨、腓骨和髌骨；同时显示胫骨、腓骨和髌骨；同时显示 股骨、胫骨、腓骨和髌骨。For the present invention, the three-dimensional images of the bone structure obtained through medical image processing can be combined arbitrarily, thereby realizing flexible and diverse display modes of the bone structure. The displayed situation includes any of the following: display only femur; display only tibia; display only fibula; display only patella; display both femur and tibia; display both femur and fibula; display both femur and patella; Show tibia and patella; show fibula and patella together; show femur, tibia and fibula together; show femur, tibia and patella together; show femur, fibula and patella together; show tibia, fibula and patella together; show femur, tibia, fibula together and patella.

对于本发明来说，通过医学图像处理获得的骨骼结构的三维影像 可以进行透明度的变换，使得影像表现出多样的显像效果。具体来说， 透明度可以在透明和不透明之间进行切换。例如，只显示股骨时，股 骨的显像效果可以选择透明，也可以选择不透明。例如，只显示胫骨实，胫骨的显像效果可以选择透明，也可以选择不透明。例如，同时 显示股骨和胫骨时，两类骨骼的显像效果可以选择透明，也可以选择 不透明。例如，同时显示股骨和腓骨时，两类骨骼的显像效果可以选择透明，也可以选择不透明。例如，同时显示股骨、胫骨和腓骨时， 三类骨骼的显像效果可以选择透明，也可以选择不透明。例如，同时 显示股骨、胫骨、腓骨和髌骨时，骨骼的显像效果可以选择透明，也 可以选择不透明。For the present invention, the three-dimensional image of the skeletal structure obtained through medical image processing can be transformed in transparency, so that the image shows various imaging effects. Specifically, transparency can be toggled between transparent and opaque. For example, when only the femur is displayed, the imaging effect of the femur can be transparent or opaque. For example, if only the tibia is displayed, the display effect of the tibia can be transparent or opaque. For example, when displaying femur and tibia at the same time, the imaging effects of the two types of bones can be transparent or opaque. For example, when displaying femur and fibula at the same time, the visualization effect of the two types of bones can be transparent or opaque. For example, when displaying femur, tibia, and fibula at the same time, the imaging effects of the three types of bones can be transparent or opaque. For example, when displaying femur, tibia, fibula and patella at the same time, you can choose transparent or opaque for the imaging effect of bones.

对于本发明来说，通过医学图像处理获得的骨骼结构的三维影像 可以进行图像缩放。如，只显示股骨时，可以进行股骨图像的缩放(缩 小或放大，以下同)。如，只显示胫骨时，可以进行胫骨图像的缩放。 如，同时显示股骨和胫骨时，可以进行股骨和胫骨图像的缩放。如，同时显示股骨、胫骨和腓骨时，可以进行这三类骨骼图像的缩放。如，同时显示股骨、胫骨、腓骨和髌骨时，可以进行这骨骼图像的缩放。在一些优选的实施方式中，二维影像(包括横断面影像、矢状面影像 和冠状面影像)也可以进行图像的缩放，如，横断面影像、矢状面影 像和冠状面影像同时放大或缩小。For the present invention, the three-dimensional image of the bone structure obtained through medical image processing can be image-scaled. For example, when only the femur is displayed, the image of the femur can be zoomed (reduced or enlarged, the same below). For example, when only the tibia is displayed, the image of the tibia can be zoomed. For example, when displaying femur and tibia at the same time, the femur and tibia images can be zoomed. For example, when displaying femur, tibia and fibula at the same time, these three types of bone images can be zoomed. For example, when displaying the femur, tibia, fibula, and patella at the same time, the bone image can be zoomed. In some preferred embodiments, the two-dimensional image (including the transverse image, the sagittal image and the coronal image) can also be scaled, for example, the transverse image, the sagittal image and the coronal image can be zoomed in simultaneously or zoom out.

对于本发明来说，通过医学图像处理获得的股骨结构的三维影像 可以按照任意轴进行旋转，还可以进行图像移动。如，只显示股骨时， 可以将股骨按照任意轴进行旋转。如，只显示胫骨时，可以将胫骨按 照任意轴进行旋转。如，同时显示股骨和胫骨时，可以将股骨和胫骨 按照任意轴进行旋转。如，同时显示股骨、胫骨和腓骨时，可以将这三类骨骼按照任意轴进行旋转。如，同时显示股骨、胫骨、腓骨和髌 骨时，可以将这骨骼结构按照任意轴进行旋转。For the present invention, the three-dimensional image of the femoral structure obtained through medical image processing can be rotated according to any axis, and the image can also be moved. For example, when only the femur is displayed, the femur can be rotated along any axis. For example, when only the tibia is displayed, the tibia can be rotated about any axis. For example, when the femur and tibia are displayed simultaneously, the femur and tibia can be rotated along any axis. For example, when displaying femur, tibia, and fibula at the same time, these three types of bones can be rotated according to any axis. For example, when simultaneously displaying the femur, tibia, fibula, and patella, the bone structure can be rotated about any axis.

总的来说，灵活多样的显示方式更加直观地显示了骨骼的立体结 构，使得医生(或其它医护人员)可以多角度、多层次地观察骨骼结 构的影像。需要说明的是，透明的含义为图像透明度(transparency) 为0.3-0.75，不透明的含义为图像透明度为0.8-1。In general, the flexible and diverse display methods more intuitively display the three-dimensional structure of the bone, allowing doctors (or other medical staff) to observe images of the bone structure from multiple angles and levels. It should be noted that transparent means that the image transparency (transparency) is 0.3-0.75, and opaque means that the image transparency is 0.8-1.

本发明通过识别标记步骤实现关键轴线、关键解剖位点、关键解 剖参数的识别标记。关键轴线包括股骨解剖轴、股骨机械轴、胫骨解 剖轴、胫骨机械轴。在一些优选的实施方式中，关键轴线还包括通髁 线、后髁连线、胫骨膝关节线、股骨矢状轴、股骨膝关节线中的任一 种或多种。关键解剖位点包括股骨髓腔的不同层面上的中心点、胫骨髓腔的不同层面上的中心点、髋关节中心点、膝关节中心点、髁间棘 的中心点、踝关节中心点，还可以包括股骨内髁凹点、股骨外髁最高 点、股骨内外后髁最低点、胫骨平台内侧低点和外侧高点、后交叉韧 带中点和胫骨结节内侧缘点、股骨远端最低点。关键解剖参数包括胫 股角、远端股骨角。在一些优选的实施方式中，所述关键解剖参数还 包括股骨后髁角。The present invention realizes the identification and marking of key axes, key anatomical sites and key anatomical parameters through the identification and marking step. Key axes include femoral anatomical axis, femoral mechanical axis, tibial anatomical axis, and tibial mechanical axis. In some preferred embodiments, key axis also comprises any one or more in passing condyle line, posterior condyle line, tibial knee joint line, femoral sagittal axis, femoral knee joint line. The key anatomical sites include the center point on different levels of the femoral medullary canal, the center point on different levels of the tibial medullary canal, the center point of the hip joint, the center point of the knee joint, the center point of the intercondylar spine, the center point of the ankle joint, and the It can include the concave point of the medial femoral condyle, the highest point of the lateral femoral condyle, the lowest point of the medial and posterior femoral condyle, the medial and lateral high points of the tibial plateau, the midpoint of the posterior cruciate ligament, the medial edge of the tibial tuberosity, and the lowest point of the distal femur. Key anatomical parameters include tibiofemoral angle, distal femoral angle. In some preferred embodiments, described key anatomy parameter also comprises femoral posterior condyle angle.

在一些优选的实施方式中，在透明度为不透明的状态下标记出关 键轴线。In some preferred embodiments, the critical axis is marked while the transparency is opaque.

在一些优选的实施方式中，在标记关键轴线后，观察关键轴线和/ 或关键解剖位点是否对位，并将不对位的关键轴线和/或关键解剖位点 手动标记；优选地，独立显示出股骨或胫骨，进一步优选地，通过旋 转调整股骨或胫骨的观察角度，然后再进行关键轴线和/或关键解剖位 点的手动标记。In some preferred embodiments, after marking the key axes, observe whether the key axes and/or key anatomical sites are aligned, and manually mark the key axes and/or key anatomical sites that are not aligned; preferably, independently display Femur or tibia, further preferably, adjust the viewing angle of femur or tibia by rotation, and then perform manual marking of key axes and/or key anatomical sites.

需要说明的是，本发明提供的方法中的医学图像数据为CT扫描数 据，该数据为dicom格式的数据。基于全膝关节置换术，CT的扫描范 围为下肢全长，即：髋关节至踝关节。显然地，本发明中的医学图像 数据为下肢全长dicom数据，下肢全长的范围为髋关节至踝关节。It should be noted that the medical image data in the method provided by the present invention is CT scan data, and the data is data in dicom format. Based on total knee arthroplasty, CT scans the entire length of the lower extremity, i.e. from the hip joint to the ankle joint. Obviously, the medical image data in the present invention is the full-length dicom data of the lower limbs, and the full-length range of the lower limbs is from the hip joint to the ankle joint.

本发明中提及的术语均为骨科常规术语，各个术语解释如下：The terms mentioned in the present invention are all orthopedic routine terms, and each term is explained as follows:

股骨解剖轴：股骨骨干中心线。Femoral Anatomical Axis: Centerline of the femoral diaphysis.

股骨机械轴：一端点位于髋关节中心，另一端点位于股骨的膝关 节中心点(股骨髁间窝顶点)。Femoral mechanical axis: one end point is located at the center of the hip joint, and the other end point is located at the center point of the knee joint of the femur (the apex of the intercondylar notch of the femur).

胫骨解剖轴：胫骨骨干中心线。Tibial Anatomical Axis: Centerline of the tibial diaphysis.

胫骨机械轴：一端点位于胫骨膝关节中心(髁间棘的中心)，另一 端点位于胫骨踝关节中心(内外踝外侧骨皮质连线的中点)。Tibial mechanical axis: one end point is located at the center of the tibial knee joint (the center of the intercondylar spine), and the other end point is located at the center of the tibial ankle joint (the midpoint of the line connecting the inner and outer malleolus).

通髁线：股骨内髁凹与外髁最高点之间的连线。Condylar line: the line between the concave of the medial condyle of the femur and the highest point of the lateral condyle.

后髁连线：股骨内外后髁最低点之间的连线。Posterior condyle line: the line between the lowest points of the inner and outer posterior condyles of the femur.

股骨膝关节线：股骨远端最低点的连线。Femoral-knee joint line: The line connecting the lowest points of the distal femur.

胫骨膝关节线：胫骨平台内侧低点和外侧高点的连线。Tibial Knee Line: The line connecting the medial low point and the lateral high point of the tibial plateau.

股骨矢状轴：后交叉韧带止点中心与胫骨结节内缘的连线。Femoral sagittal axis: line connecting the center of the insertion of the posterior cruciate ligament and the medial border of the tibial tuberosity.

胫股角(又称mTFA)：股骨机械轴和胫骨机械轴形成的夹角。Tibiofemoral angle (also known as mTFA): The angle formed by the mechanical axis of the femur and the mechanical axis of the tibia.

远端股骨角：股骨机械轴与股骨解剖轴之间的夹角。Distal Femoral Angle: The angle between the mechanical axis of the femur and the anatomical axis of the femur.

股骨后髁角(又称PCA)：股骨通髁线与后髁连线在横断面的投影 线之间的夹角。Posterior femoral condyle angle (also known as PCA): the angle between the line through the femoral condyle and the projection line of the posterior condyle on the transverse plane.

关于S2：About S2:

在一些优选的实施方式中，所述三维假体包括三维股骨假体和三 维胫骨假体；和In some preferred embodiments, the three-dimensional prosthesis comprises a three-dimensional femoral prosthesis and a three-dimensional tibial prosthesis; and

所述模拟匹配包括：The simulated matching includes:

(i)将三维股骨假体植入股骨(指的是股骨三维影像)，将三维胫 骨假体植入胫骨(指的是胫骨三维影像)；可以将可视化三维假体用颜 色与骨骼结构区分开来；(i) The 3D femoral prosthesis is implanted in the femur (refer to the 3D image of the femur) and the 3D tibial prosthesis is implanted in the tibia (refer to the 3D image of the tibia); the visual 3D prosthesis can be distinguished from the bone structure by color Come;

(ii)选择三维股骨假体和三维胫骨假体，选择模拟手术条件；(ii) Select a three-dimensional femoral prosthesis and a three-dimensional tibial prosthesis, and select simulated surgical conditions;

(iii)根据三维假体与骨骼的匹配关系智能截骨，观察模拟匹配效 果；(iii) Intelligent osteotomy according to the matching relationship between the three-dimensional prosthesis and the bone, and observe the simulated matching effect;

(iv)若是模拟匹配效果不符合手术需求，则重复步骤ii至步骤iii， 直至模拟匹配效果符合手术要求。(iv) If the simulated matching effect does not meet the surgical requirements, repeat step ii to step iii until the simulated matching effect meets the surgical requirements.

优选地，在步骤ii中，选择三维股骨假体包括选择股骨假体类型 和/或股骨假体型号(型号代表大小，以下同)。优选地，在步骤ii中， 选择三维胫骨假体包括选择胫骨假体类型和/或胫骨假体型号。优选地， 在步骤ii中，还可以选择三维胫骨垫类型和/或型号。需要说明的是， 存储的股骨假体类型以及其型号、胫骨假体类型以及其型号、胫骨垫 类型以及其型号中所提及的类型和型号为市售产品(目前市场上已有 的全膝关节置换用假体)的商品类型和型号。如，股骨假体类型有 ATTUNE-PS、ATTUNE-CR、SIGMA-PS150等。如，ATTUNE-PS的 型号有1、2、3、3N、4、4N、5、5N、6、6N。如，SIGMA-PS150的 型号有1、1.5、2、2.5、3、4、4N、5、6。如，胫骨假体类型有ATTUNE-FB、 ATTUNE-RP、SIGMA-MBT等。如，ATTUNE-FB的型号有1、2、3、 4、5、6、7、8、9、10。如，SIGMA-MBT的型号有1、1.5、2、2.5、 3、4、5、6、7。本发明在此不一一举例说明。Preferably, in step ii, selecting a three-dimensional femoral prosthesis includes selecting a femoral prosthesis type and/or a femoral prosthesis model (model represents size, the same below). Preferably, in step ii, selecting a three-dimensional tibial prosthesis includes selecting a tibial prosthesis type and/or a tibial prosthesis model. Preferably, in step ii, the type and/or model of the three-dimensional tibial pad can also be selected. It should be noted that the types and models of the stored femoral prosthesis and its model, the type of tibial prosthesis and its model, the type of tibial pad and its model are commercially available products (total knee prosthesis currently on the market Joint replacement prosthesis) type and model number. For example, the types of femoral prosthesis include ATTUNE-PS, ATTUNE-CR, SIGMA-PS150, etc. For example, the models of ATTUNE-PS are 1, 2, 3, 3N, 4, 4N, 5, 5N, 6, 6N. For example, the models of SIGMA-PS150 are 1, 1.5, 2, 2.5, 3, 4, 4N, 5, 6. For example, the types of tibial prosthesis include ATTUNE-FB, ATTUNE-RP, SIGMA-MBT and so on. For example, the models of ATTUNE-FB are 1, 2, 3, 4, 5, 6, 7, 8, 9, and 10. For example, the models of SIGMA-MBT are 1, 1.5, 2, 2.5, 3, 4, 5, 6, and 7. The present invention is not illustrated here one by one.

优选地，在步骤ii中，选择模拟手术条件包括选择股骨手术参数 和/或选择胫骨手术参数；所述股骨手术参数包括股骨远端截骨量、股 骨后髁截骨量、内外旋角、内外翻角和股骨假体屈曲角；所述胫骨手术参数包括胫骨截骨量、内外旋角、内外翻角和后倾角。Preferably, in step ii, the selection of simulated operation conditions includes selection of femoral operation parameters and/or selection of tibial operation parameters; Angle of rotation and femoral prosthesis flexion angle; The tibial surgical parameters include the amount of tibial osteotomy, internal and external rotation angle, internal and external valgus angle and posterior inclination angle.

在一些优选的实施方式中，在如下一个或多个状态下观察模拟匹 配效果：In some preferred embodiments, the simulated matching effect is observed in one or more of the following states:

(a)截骨状态或非截骨状态；(a) Osteotomy state or non-osteotomy state;

(b)骨骼透明状态或不透明状态；(b) bone transparent state or opaque state;

(c)腓骨显示或不显示状态。(c) Fibula showing or not showing state.

关于S3：About S3:

在一些优选的实施方式中，所述方法还包括S3：可视化术后模拟 的步骤，用于模拟全膝关节置换术的术后肢体运动情况。In some preferred embodiments, the method also includes S3: a step of visual postoperative simulation, which is used to simulate postoperative limb movement in total knee arthroplasty.

在一些优选的实施方式中，所述方法(图1未示出)还包括将符 合手术需求的模拟匹配数据导出，形成术前规划报告的步骤，便于医 生进行术前部署。In some preferred embodiments, the method (not shown in FIG. 1 ) also includes the step of exporting the simulated matching data that meets the surgical requirements to form a preoperative planning report, which is convenient for doctors to perform preoperative deployment.

﹤第二方面﹥﹤The second aspect﹥

本发明在第二方面提供了一种基于深度学习的全膝关节置换的术 前规划系统，参考图2，术前规划系统包括：In a second aspect, the present invention provides a preoperative planning system for total knee replacement based on deep learning. Referring to FIG. 2, the preoperative planning system includes:

医学图像数据处理模块101，用于通过医学图像处理获得骨骼结构 的三维影像、识别标记出关键轴线、关键解剖位点和关键解剖参数； 所述骨骼结构包括股骨、胫骨、腓骨和髌骨；所述关键解剖位点包括股骨髓腔的不同层面上的中心点、胫骨髓腔的不同层面上的中心点、 髋关节中心点、膝关节中心点、髁间棘的中心点、踝关节中心点；所 述关键轴线包括股骨解剖轴、股骨机械轴、胫骨解剖轴和胫骨机械轴； 所述关键解剖参数包括胫股角和远端股骨角；The medical image data processing module 101 is used to obtain a three-dimensional image of the bone structure through medical image processing, identify and mark key axes, key anatomical sites and key anatomical parameters; the bone structure includes femur, tibia, fibula and patella; the The key anatomical points include the center points on different levels of the femoral medullary cavity, the center points on different levels of the tibial medullary cavity, the center point of the hip joint, the center point of the knee joint, the center point of the intercondylar spine, and the center point of the ankle joint; The key axis includes femoral anatomical axis, femoral mechanical axis, tibial anatomical axis and tibial mechanical axis; the key anatomical parameters include tibiofemoral angle and distal femoral angle;

模拟匹配模块201，用于将三维假体与三维股骨和三维胫骨进行模 拟匹配，实时观察模拟匹配效果；和The simulation matching module 201 is used to simulate the matching of the three-dimensional prosthesis with the three-dimensional femur and the three-dimensional tibia, and observe the simulation matching effect in real time; and

显示模块301：用于显示骨骼结构的三维影像、关键轴线、关键解 剖位点、关键解剖参数和模拟匹配效果。Display module 301: used to display the three-dimensional image of the bone structure, key axes, key anatomical points, key anatomical parameters and simulation matching effects.

在一些优选的实施方式中，所述医学图像数据处理模块101包括：In some preferred embodiments, the medical image data processing module 101 includes:

三维重建单元，用于获得骨骼结构的三维影像；a three-dimensional reconstruction unit for obtaining a three-dimensional image of the bone structure;

图像分割单元，用于分割出股骨、胫骨、腓骨和髌骨；The image segmentation unit is used to segment femur, tibia, fibula and patella;

识别标记单元，用于识别标记出关键轴线、关键解剖位点和关键 解剖参数。The identification marking unit is used to identify and mark key axes, key anatomical sites and key anatomical parameters.

在一些优选的实施方式中，所述术前规划系统还包括数据导入模 块404，用于导入医学图像数据。In some preferred embodiments, the preoperative planning system also includes a data import module 404, which is used to import medical image data.

在一些优选的实施方式中，所述术前规划系统还包括可视化术后 模拟模块501，用于模拟全膝关节置换术的术后肢体运动情况。In some preferred embodiments, the preoperative planning system also includes a visualized postoperative simulation module 501, which is used to simulate postoperative limb movement in total knee arthroplasty.

在一些优选的实施方式中，所述术前规划系统还包括图像组合模 块，用于将骨骼结构任意组合。在一些优选的实施方式中，所述术前 规划系统还包括图像透明度切换模块，用于切换骨骼结构的透明度。 在一些优选的实施方式中，所述术前规划系统还包括图像缩放模块， 用于缩放骨骼结构的三维影像和/或二维影像。在一些优选的实施方式 中，所述术前规划系统还包括图像旋转模块，用于将图像按照任意轴 进行旋转。在一些优选的实施方式中，所述术前规划系统还包括图像移动模块，用于将图像进行移动。In some preferred embodiments, the preoperative planning system also includes an image combination module, which is used to combine bone structures arbitrarily. In some preferred embodiments, the preoperative planning system also includes an image transparency switching module, which is used to switch the transparency of the bone structure. In some preferred embodiments, the preoperative planning system further includes an image scaling module, configured to scale the three-dimensional image and/or the two-dimensional image of the bone structure. In some preferred embodiments, the preoperative planning system also includes an image rotation module, which is used to rotate the image along any axis. In some preferred embodiments, the preoperative planning system further includes an image moving module, which is used to move the image.

在一些优选的实施方式中，所述术前规划系统还包括数据导出模 块，用于将符合手术需求的模拟匹配数据导出，形成术前规划报告。In some preferred embodiments, the preoperative planning system also includes a data export module, which is used to export the simulated matching data that meets the surgical requirements to form a preoperative planning report.

初次之外，本系统可以实现的更多的功能或者更为具体的功能请 见第一方面内容。Except for the first time, please refer to the first aspect for more functions or more specific functions that this system can realize.

﹤第三方面﹥﹤The third aspect﹥

一种设备，包括：A device comprising:

一个或多个处理器；one or more processors;

存储装置，用于存储一个或多个程序；storage means for storing one or more programs;

当所述一个或多个程序被所述一个或多个处理器执行，使得所述 一个或多个处理器实现本发明在第一方面提供的所述术前规划方法。When the one or more programs are executed by the one or more processors, the one or more processors are made to implement the preoperative planning method provided in the first aspect of the present invention.

﹤第四方面﹥﹤The fourth aspect﹥

一种计算机可读存储介质，其上存储有计算机程序，A computer-readable storage medium on which a computer program is stored,

所述计算机程序被处理器执行时实现本发明在第一方面提供的术 前规划方法。When the computer program is executed by the processor, the preoperative planning method provided by the present invention in the first aspect is realized.

以下再结合附图3至图11进行更为具体的说明：A more specific description will be given below in conjunction with accompanying drawings 3 to 11:

导入数据：利用数据导入模块404将CT扫描获得的下肢全长 dicom数据导入术前规划系统中。Import data: use the data import module 404 to import the full-length dicom data of the lower extremity obtained by CT scanning into the preoperative planning system.

基于深度学习的医学图像数据处理：利用医学图像数据处理模块 101进行该步骤，通过医学图像处理获得骨骼结构的三维影像和二维影 像、识别标记出关键轴线、关键解剖位点关键解剖参数；所述骨骼结 构包括股骨、胫骨、腓骨和髌骨；关键解剖位点包括股骨髓腔的不同 层面上的中心点、胫骨髓腔的不同层面上的中心点、髋关节中心点、 膝关节中心点、髁间棘的中心点、踝关节中心点，还包括股骨内髁凹点、股骨外髁最高点、股骨内外后髁最低点、胫骨平台内侧低点和外 侧高点、后交叉韧带中点和胫骨结节内侧缘点、股骨远端最低点；所 述关键轴线包括股骨解剖轴、股骨机械轴、胫骨解剖轴、胫骨机械轴， 还包括通髁线、后髁连线、胫骨膝关节线、股骨矢状轴、股骨膝关节 线中的任一种或多种；所述关键解剖参数包括胫股角和远端股骨角， 还包括股骨后髁角。Medical image data processing based on deep learning: use the medical image data processing module 101 to perform this step, obtain 3D images and 2D images of bone structures through medical image processing, identify and mark key axes, key anatomical sites and key anatomical parameters; The skeletal structures described above include the femur, tibia, fibula, and patella; key anatomical sites include the center point on different levels of the medullary canal of the femur, the center point on different levels of the medullary canal of the tibia, the center point of the hip joint, the center point of the knee joint, the condyle The center point of the interspinous spine, the center point of the ankle joint, also includes the concave point of the medial femoral condyle, the highest point of the lateral femoral condyle, the lowest point of the medial and medial posterior femoral condyle, the medial low point and lateral high point of the tibial plateau, the midpoint of the posterior cruciate ligament, and the tibial knot The medial border point of the femur, the lowest point of the distal femur; the key axis includes the femoral anatomical axis, the femoral mechanical axis, the tibial anatomical axis, the tibial mechanical axis, and also includes the condylar line, the posterior condyle line, the tibial knee joint line, and the femoral sagittal axis. Any one or more in the shape axis, the femoral knee joint line; The key anatomical parameters include the tibiofemoral angle and the distal femoral angle, and also include the posterior femoral condyle angle.

具体地，该步骤包括：Specifically, this step includes:

骨骼三维影像重建的步骤Steps of bone 3D image reconstruction

利用三维重建单元，根据下肢全长dicom数据进行三维影像重建， 获得下肢骨骼三维影像，可以通过显示模块301显示出来。需要说明 的是，三维影像重建可以利用现有方法实现，因此，三维重建单元可以为现有的能够实现三维影像重建的单元。The 3D reconstruction unit is used to reconstruct the 3D image according to the full-length dicom data of the lower limb to obtain a 3D image of the skeleton of the lower limb, which can be displayed by the display module 301 . It should be noted that the three-dimensional image reconstruction can be realized by using existing methods, therefore, the three-dimensional reconstruction unit can be an existing unit capable of realizing three-dimensional image reconstruction.

基于深度学习的图像分割的步骤Steps of Image Segmentation Based on Deep Learning

按照如下方法实现股骨、胫骨、腓骨和髌骨这四类骨骼结构的分 割：Segmentation of the four bone structures of femur, tibia, fibula and patella is achieved as follows:

构建下肢医学图像数据库：获取下肢CT图像数据集，手动标注出 股骨、胫骨、腓骨和髌骨区域；将数据集按照7:3的比例划分为训练集 和测试集；将未标注前的二维横断面影像dicom数据转换成jpg格式的 图片并保存，将标注后的数据转换成png格式的图片并保存。此处以 二维横断面数据进行说明，还可以使用二维矢状面和二维冠状面数据。Construct the medical image database of lower extremities: obtain CT image data sets of lower extremities, and manually mark the femur, tibia, fibula, and patella regions; divide the data set into training set and test set according to the ratio of 7:3; The face image dicom data is converted into a picture in jpg format and saved, and the marked data is converted into a picture in png format and saved. Two-dimensional cross-sectional data are described here, but two-dimensional sagittal and two-dimensional coronal data can also be used.

建立分割神经网络模型，分割神经网络模型为Unet+PointRend， 利用Unet神经网络进行粗分割，利用PointRend神经网络进行精确分 割；所述Unet神经网络包括4个上采样层和4个下采样层；每个上采 样层优选包括1个上采样操作层和2个卷积层，其中的卷积层中的卷 积核大小为3*3，上采样操作层中的卷积核大小为2*2，每个上采样层 中的卷积核个数为512，256，256，128；每个下采样层优选包括2个 卷积层和1个池化层，其中的卷积层中的卷积核大小为3*3，池化层中 的卷积核大小为2*2，每个卷积层中的卷积核的个数为128，256，256， 512；最后一次上采样结束后设有一个dropout层，droupout率设置为 0.5-0.7；所有的卷积层后面设有激活层，激活层使用的激活函数为relu 函数。Set up segmentation neural network model, segmentation neural network model is Unet+PointRend, utilize Unet neural network to carry out coarse segmentation, utilize PointRend neural network to carry out precise segmentation; Described Unet neural network comprises 4 upper sampling layers and 4 lower sampling layers; Each upsampling layer preferably includes 1 upsampling operation layer and 2 convolution layers, wherein the convolution kernel size in the convolution layer is 3*3, and the convolution kernel size in the upsampling operation layer is 2*2, The number of convolution kernels in each upsampling layer is 512, 256, 256, 128; each downsampling layer preferably includes 2 convolution layers and 1 pooling layer, where the convolution kernel in the convolution layer The size is 3*3, the size of the convolution kernel in the pooling layer is 2*2, and the number of convolution kernels in each convolution layer is 128, 256, 256, 512; after the last upsampling, set A dropout layer, the dropout rate is set to 0.5-0.7; all convolutional layers are followed by an activation layer, and the activation function used by the activation layer is the relu function.

模型训练，包括：Model training, including:

粗分割训练：将训练集全部送入Unet神经网络进行训练；训练过 程中，数据标签的背景像素值设置为0，股骨为1，胫骨为2，腓骨为 3，髌骨为4，训练的batch_size为6，学习率设置为1e-4，优化器使用Adam优化器，使用的损失函数为DICE loss，根据训练过程中损失函数的变化，调整训练批次的大小；Coarse segmentation training: Send all the training sets to the Unet neural network for training; during the training process, the background pixel value of the data label is set to 0, the femur is 1, the tibia is 2, the fibula is 3, and the patella is 4. The batch_size of the training is 6. The learning rate is set to 1e-4, the optimizer uses the Adam optimizer, and the loss function used is DICE loss. According to the change of the loss function during the training process, the size of the training batch is adjusted;

完成粗分割后，送入PointRend神经网络进行精确分割；初始过程包括，先使用双线性插值上采样粗分割的预测结果，再在特征图中选 定多个置信度为0.5的点作为最不确定的点，然后通过双线性插值 Bilinear计算多个点的特征表示并且预测labels；重复所述初始过程， 直到上采样到需要的大小。After the rough segmentation is completed, it is sent to the PointRend neural network for precise segmentation; the initial process includes first using bilinear interpolation to upsample the prediction results of the rough segmentation, and then selecting multiple points with a confidence of 0.5 in the feature map as the least Determined points, then calculate the feature representation of multiple points and predict labels through bilinear interpolation Bilinear; repeat the initial process until the upsampling reaches the required size.

利用训练好的分割神经网络模型进行分割。Use the trained segmentation neural network model for segmentation.

上述分割过程可在图像分割单元中实现，分割出的这四类骨骼结 构无连结，并且边缘清晰。The above segmentation process can be realized in the image segmentation unit, and the four types of skeletal structures segmented are unconnected and have clear edges.

基于深度学习的识别标记的步骤Steps for identifying markers based on deep learning

步骤包括：Steps include:

(1)识别关键解剖位点。(1) Identify key anatomical sites.

识别关键解剖位点的步骤包括：Steps to identify key anatomical sites include:

构建数据库：获取下肢医学图像数据集，手动标定关键点；将所 述数据集按照7:3的比例划分为训练集和测试集。Construct the database: obtain the medical image data set of the lower limbs, manually mark the key points; divide the data set into a training set and a test set according to the ratio of 7:3.

建立识别神经网络模型：所述识别神经网络模型为hourglass， hourglass的网络细节在此不再详述。Establishing a recognition neural network model: the recognition neural network model is hourglass, and the network details of the hourglass will not be described in detail here.

模型训练：在进行训练时，输入像素值为0-255的正投影图像和 label.txt，可以通过每张图片的名称找到互相对应的点的坐标；若直接 用目标点的坐标进行学习，神经网络需要自行将空间位置转换为坐标， 是一种比较难学习的训练方式，所以将这些点生成高斯图，用heatmap去监督，即网络的输出是一个与输入大小相同尺寸的特征图，在检测 点的位置为1，其他位置为0，多个点的检测就输出多个通道的特征图； 网络使用Adam优化，学习率为1e-5，batch_size为4，损失函数使用 L2正则化，可以根据训练过程中损失函数的变化，调整训练批次的大 小，得到关键点位的坐标值。Model training: During training, input an orthographic projection image with a pixel value of 0-255 and label.txt, and you can find the coordinates of the corresponding points through the name of each image; if you directly use the coordinates of the target point for learning, the neural The network needs to convert the spatial position into coordinates by itself, which is a relatively difficult training method to learn, so these points are generated into a Gaussian map, and the heatmap is used to supervise, that is, the output of the network is a feature map with the same size as the input. The position of the point is 1, and the other positions are 0. The detection of multiple points will output the feature map of multiple channels; the network uses Adam optimization, the learning rate is 1e-5, the batch_size is 4, and the loss function uses L2 regularization, which can be based on During the training process, the loss function changes, the size of the training batch is adjusted, and the coordinate values of the key points are obtained.

利用训练好的神经网络模型进行关键解剖位点的识别。Use the trained neural network model to identify key anatomical sites.

(2)利用关键解剖位点获得关键轴线：(2) Using key anatomical points to obtain key axes:

对于股骨解刨轴，可以通过拟合股骨髓腔的不同层面上的中心点 而得到。对于胫骨解刨轴，可以通过拟合胫骨髓腔的不同层面上的中 心点而得到。拟合的方法为最小二乘法、梯度下降、高斯牛顿、列-马 算法中的任一种。For the femoral anatomy axis, it can be obtained by fitting the center points on different levels of the femoral medullary cavity. For the tibial anatomy axis, it can be obtained by fitting the center points on different levels of the tibial medullary cavity. The fitting method is any one of the least squares method, gradient descent, Gauss-Newton, and Lemma-Marine algorithm.

对于其它种类的关键轴线，可以利用确定的两个端点而得到。如， 股骨机械轴的两个端点-髋关节中心点和膝关节中心点-已被识别出来， 可以通过这两点确定股骨机械轴线。For other kinds of critical axes, it can be obtained by using the determined two endpoints. For example, two endpoints of the femoral mechanical axis - the center of the hip and the center of the knee - have been identified, from which the femoral mechanical axis can be determined.

(3)测量关键解剖参数。(3) Measure key anatomical parameters.

上述识别标记步骤在识别标记单元实现。The above steps of identifying and marking are realized in the identifying and marking unit.

需要说明的是，本发明对医学图像数据处理步骤所包括的三个步 骤没有顺序上的限定。本发明在此处为了具体说明医学图像数据处理 的步骤而给出了包含顺序的处理步骤，但不应理解为处理顺序的限定。It should be noted that the present invention does not limit the order of the three steps included in the medical image data processing steps. In order to specifically illustrate the steps of medical image data processing, the present invention provides the processing steps including the sequence, but it should not be understood as the limitation of the processing sequence.

四类骨骼结构(股骨、胫骨、腓骨和髌骨)通过图像组合模块可 以进行任意组合，通过图像透明度切换模块可以进行透明度的变换， 通过图像缩放模块可以进行图像缩放、通过图像旋转模块可以进行图 像旋转。图3为分割后四类骨骼组合在一起的三维影像，显影效果为 不透明(可切换为透明状态)，其中a图和b图的角度不同，在观察时可以选择不同的角度进行观察。由于本发明将股骨、胫骨、腓骨和髌 骨这四类骨骼结构进行了分割，显然，这四类股骨结构可以任意进行 组合。图4为只显示股骨的股骨三维影像，显影效果为不透明(可切 换为透明状态)，其中a图和b图的角度不同。图5为只显示胫骨的胫 骨三维影像，显影效果为不透明(可切换为透明状态)，其中a图和b 图的角度不同。此处只结合附图列举了四类骨骼组合在一起显示、只 显示股骨、只显示胫骨的情况，还可以只显示腓骨，还可以只显示髌 骨，还可以同时显示股骨和胫骨等。图6为5b胫骨平台处的放大图。 当然，任意的组合方式下的三维影像均可进行放大或缩小。如，当只 显示股骨时，可以进行放大或缩小。如，同时显示股骨和胫骨时，可以进行放大或缩小。同时显示股骨、胫骨和腓骨时，可以进行放大或 缩小。同时显示股骨、胫骨、腓骨和髌骨时，可以进行放大或缩小。Four types of bone structures (femur, tibia, fibula, and patella) can be combined arbitrarily through the image combination module, transparency can be transformed through the image transparency switching module, image scaling can be performed through the image scaling module, and image rotation can be performed through the image rotation module . Figure 3 is a three-dimensional image of the four types of bones combined after segmentation. The development effect is opaque (can be switched to a transparent state). The angles of a and b are different, and different angles can be selected for observation. Since the present invention has segmented these four types of bone structures of femur, tibia, fibula and patella, obviously, these four types of femur structures can be combined arbitrarily. Figure 4 is a three-dimensional image of the femur that only shows the femur, and the development effect is opaque (can be switched to a transparent state), where the angles of a and b are different. Figure 5 is a three-dimensional image of the tibia showing only the tibia, and the development effect is opaque (can be switched to a transparent state), where the angles of a and b are different. Here, only the four types of bones are combined to display, only the femur and the tibia are displayed in conjunction with the accompanying drawings, and only the fibula can be displayed, the patella can only be displayed, and the femur and tibia can also be displayed at the same time. Figure 6 is an enlarged view of the tibial plateau at 5b. Of course, the three-dimensional images in any combination can be enlarged or reduced. For example, when only the femur is displayed, it can be zoomed in or out. For example, when displaying both the femur and the tibia, zoom in or zoom out is possible. You can zoom in or out when displaying the femur, tibia, and fibula at the same time. Zooming in or out is possible when simultaneously displaying the femur, tibia, fibula, and patella.

图7显示了标记有关键轴线、关键解剖位点和关键解剖参数后的 结果图。可以观察各个关键解剖位点和/或关键轴线的位置是否正确， 若不正确，可以手动标记关节解剖位点和/或关键轴线(通过手动标记 关键解剖位点而实现)。Figure 7 shows the resulting plot with key axes, key anatomical sites and key anatomical parameters marked. It can be observed whether the position of each key anatomical site and/or key axis is correct, if not, the joint anatomical site and/or key axis can be manually marked (realized by manually marking the key anatomical site).

可视化模拟匹配Visual simulation matching

将三维假体与三维股骨和三维胫骨进行模拟匹配，实时观察模拟 匹配效果；当模拟匹配效果符合手术要求时，视为完成模拟匹配。三 维假体包括三维股骨假体和三维胫骨假体；该步骤可以具体按照如下 方法进行：The three-dimensional prosthesis was simulated and matched with the three-dimensional femur and three-dimensional tibia, and the simulated matching effect was observed in real time; when the simulated matching effect met the surgical requirements, the simulated matching was considered complete. The three-dimensional prosthesis includes a three-dimensional femoral prosthesis and a three-dimensional tibial prosthesis; this step can be specifically carried out as follows:

(i)根据前期的分割识别标记结果，自动将三维股骨假体植入股 骨，将三维胫骨假体植入胫骨，将胫骨垫植入假体间隙；(i) According to the results of previous segmentation and identification marks, automatically implant the three-dimensional femoral prosthesis into the femur, implant the three-dimensional tibial prosthesis into the tibia, and implant the tibial pad into the gap between the prosthesis;

(ii)选择三维股骨假体的类型和型号，调整其三维空间位置；选 择三维股骨假体的类型和型号，调整其三维空间位置；胫骨垫的类型 和型号；选择模拟手术条件，模拟手术条件包括股骨手术参数和胫骨 手术参数，股骨手术参数包括股骨远端截骨量、股骨后髁截骨量、内外旋角、内外翻角和股骨假体屈曲角；胫骨手术参数包括胫骨截骨量、 内外旋角、内外翻角和后倾角；(ii) Select the type and model of the three-dimensional femoral prosthesis and adjust its three-dimensional space position; select the type and model of the three-dimensional femoral prosthesis and adjust its three-dimensional space position; the type and model of the tibial pad; select the simulated surgical conditions and simulate the surgical conditions Including femoral surgical parameters and tibial surgical parameters, femoral surgical parameters include distal femoral osteotomy amount, posterior femoral condyle osteotomy amount, internal and external rotation angle, varus angle and femoral prosthesis flexion angle; tibial surgical parameters include tibial osteotomy amount, Internal and external pronation, valgus and caster angles;

(iii)根据三维假体与骨骼的匹配关系智能截骨，观察模拟匹配效 果；(iii) Intelligent osteotomy according to the matching relationship between the three-dimensional prosthesis and the bone, and observe the simulated matching effect;

可以在如下一个或多个状态下观察模拟匹配效果：Simulated matching effects can be observed in one or more of the following states:

(a)截骨状态或非截骨状态；(a) Osteotomy state or non-osteotomy state;

(b)骨骼透明状态或不透明状态；(b) bone transparent state or opaque state;

(c)腓骨显示或不显示状态；(c) fibula display or non-display status;

(iv)若是模拟匹配效果不符合手术需求，则重复步骤ii至步骤iii： 重新选择假体类型和/或型号和/或模拟手术条件，然后进行模拟截骨， 观察模拟匹配效果，直至模拟匹配效果符合手术要求。(iv) If the simulated matching effect does not meet the surgical requirements, repeat steps ii to iii: reselect the type and/or model of the prosthesis and/or simulated surgical conditions, then perform simulated osteotomy, and observe the simulated matching effect until the simulated match The effect meets the surgical requirements.

可视化模拟匹配的步骤在模拟匹配模块201中进行，图8显示了 模拟匹配的界面，状态是截骨前，显影效果为透明(可切换)。图9显 示了截骨后的结果图，显影效果为不透明(可切换)。在模拟匹配的过 程中，如图10所示，可以利用图像旋转模块调节图像角度，多方位进 行观察。The step of visual simulation matching is carried out in the simulation matching module 201, and Fig. 8 has shown the interface of simulation matching, and state is before osteotomy, and developing effect is transparent (switchable). Figure 9 shows the result map after osteotomy, and the development effect is opaque (switchable). During the simulation matching process, as shown in Figure 10, the image rotation module can be used to adjust the image angle and observe in multiple directions.

术后模拟postoperative simulation

利用术后模拟模块进行术后模拟501，如图11所示，观察截骨后 假体与股骨和胫骨的整体匹配效果，观察全膝关节置换术术后肢体运 动情况(图中未示出)。Use the postoperative simulation module to perform postoperative simulation 501, as shown in Figure 11, to observe the overall matching effect of the prosthesis with the femur and tibia after osteotomy, and to observe the movement of limbs after total knee arthroplasty (not shown in the figure) .

此外，在完成术后模拟后，还可以利用数据导出模块将术前规划 的数据导出，这些数据包括可视化模拟匹配过程中的假体(股骨、胫 骨和胫骨垫)类型和型号、模拟手术条件，形成术前规划报告。In addition, after the postoperative simulation is completed, the data export module can also be used to export the data of the preoperative planning, which includes the type and model of the prosthesis (femur, tibia and tibial pad) in the visual simulation matching process, simulated surgical conditions, Form a preoperative planning report.

图12位本发明的实施例提供的一种设备的结构示意图，该设备包 括存储器10、处理器20、输入装置30和输出装置40。设备中的处理 器20的数量可以是一个或多个，图12中以一个处理器20为例；设备 中的存储器10、处理器20、输入装置30和输出装置40可以通过总线 或其它方式连接，图12中以通过总线50连接为例。FIG. 12 is a schematic structural diagram of a device provided by an embodiment of the present invention. The device includes a memory 10, a processor 20, an input device 30 and an output device 40. The number of processors 20 in the device can be one or more, one processor 20 is taken as an example in Figure 12; the memory 10, processor 20, input device 30 and output device 40 in the device can be connected by bus or other , in FIG. 12, the connection through the bus 50 is taken as an example.

存储器10作为一种计算机可读存储介质，可用于存储软件程序、 计算机可执行程序以及模块，如本发明实施例中的术前规划方法对应 的程序指令/模块。处理器20通过运行存储在存储器10中的软件程序、 指令以及模块，从而执行设备的各种功能应用以及数据处理，即实现上述的图像分割方法。Memory 10, as a computer-readable storage medium, can be used to store software programs, computer-executable programs and modules, such as program instructions/modules corresponding to the preoperative planning method in the embodiment of the present invention. The processor 20 executes various functional applications and data processing of the device by running the software programs, instructions and modules stored in the memory 10 , that is, implements the above-mentioned image segmentation method.

存储器10可主要包括存储程序区和存储数据区，其中，存储程序 区可存储操作系统、至少一个功能所需的应用程序；存储数据区可存 储根据设备的使用所创建的数据等。此外，存储器10可以包括高速随机存取存储器，还可以包括非易失性存储器，例如至少一个磁盘存储 器件、闪存器件、或其他非易失性固态存储器件。在一些实例中，存 储器10可进一步包括相对于处理器20远程设置的存储器，这些远程 存储器可以通过网络连接至设备。上述网络的实例包括但不限于互联 网、企业内部网、局域网、移动通信网及其组合。The memory 10 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system and at least one application required by a function; the data storage area can store data created according to the use of the device, and the like. In addition, the memory 10 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid-state storage devices. In some instances, memory 10 may further include memory located remotely from processor 20, and these remote memories may be connected to the device via a network. Examples of the aforementioned networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

输入装置30可用于接收输入的数字或字符信息，以及产生与装置 的用户设置以及功能控制有关的键信号输入。输出装置40可包括显示 屏等显示设备。The input device 30 can be used to receive input numerical or character information, and to generate key signal input related to user settings and function control of the device. The output device 40 may include a display device such as a display screen.

本发明实施例还提供一种包含计算机可执行指令的存储介质，所 述计算机可执行指令在由计算机处理器执行时用于执行一种术前规划 方法，该方法包括：The embodiment of the present invention also provides a storage medium containing computer-executable instructions, the computer-executable instructions are used to perform a preoperative planning method when executed by a computer processor, the method comprising:

基于深度学习的医学图像数据处理的步骤，通过所述医学图像处 理获得四类骨骼结构的三维影像、识别标记出关键轴线、关键解剖位 点、关键解剖参数；四类骨骼结构包括股骨、胫骨、腓骨和髌骨；关 键解剖位点包括股骨髓腔的不同层面上的中心点、胫骨髓腔的不同层 面上的中心点、髋关节中心点、膝关节中心点、髁间棘的中心点、踝关节中心点；关键轴线包括股骨解剖轴、股骨机械轴、胫骨解剖轴和 胫骨机械轴；所述关键解剖参数包括胫股角和远端股骨角；更为具体 的方法见第一方面内容；The steps of medical image data processing based on deep learning, through the medical image processing, obtain three-dimensional images of four types of skeletal structures, identify and mark key axes, key anatomical sites, and key anatomical parameters; the four types of skeletal structures include femur, tibia, Fibula and patella; key anatomical sites include the center of the different levels of the femoral canal, the center of the different levels of the tibial canal, the center of the hip, the center of the knee, the center of the intercondylar spine, the ankle Center point; key axis includes femoral anatomical axis, femoral mechanical axis, tibial anatomical axis and tibial mechanical axis; described key anatomical parameters include tibiofemoral angle and distal femoral angle; more specific methods see the first aspect content;

可视化模拟匹配的步骤，将三维假体与三维股骨和三维胫骨进行 模拟匹配，实时观察模拟匹配效果；当模拟匹配效果符合手术要求时， 视为完成模拟匹配。更为具体的方法见第一方面内容。The steps of visual simulation matching are simulated matching of the 3D prosthesis with the 3D femur and 3D tibia, and the simulated matching effect is observed in real time; when the simulated matching effect meets the surgical requirements, the simulated matching is considered complete. For more specific methods, see the first aspect.

当然，本发明所提供的一种包含计算机可执行指令的存储介质， 其计算机可执行指令不限于如上所述的方法操作，还可以执行本发明 任意一种术前规划方法中的相关操作。Of course, the present invention provides a storage medium containing computer-executable instructions, and its computer-executable instructions are not limited to the above-mentioned method operations, and can also perform related operations in any preoperative planning method of the present invention.

通过以上关于实施方式的描述，所属领域的技术人员可以清楚地 了解到，本发明可借助软件及必需的通用硬件来实现，当然也可以通 过硬件实现，但很多情况下前者是更佳的实施方式。依据这样的理解， 本发明的技术方案本质上或者说对现有技术做出贡献的部分可以以软 件产品的形式体现出来，该计算机软件产品可以存储在计算机可读存储介质中，如计算机的软盘、只读存储器(Read-Only Memory，ROM)、 随机存取存储器(RandomAccess Memory，RAM)、闪存(FLASH)、硬 盘或光盘等，包括若干指令用以使得一台计算机设备(可以是个人计算 机，服务器，或者网络设备等)执行本发明各个实施例所述的方法。Through the above description about the implementation mode, those skilled in the art can clearly understand that the present invention can be realized by means of software and necessary general-purpose hardware, and of course it can also be realized by hardware, but in many cases the former is a better implementation mode . According to such an understanding, the essence of the technical solution of the present invention or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a floppy disk of a computer , read-only memory (Read-Only Memory, ROM), random access memory (RandomAccess Memory, RAM), flash memory (FLASH), hard disk or CD, etc., including several instructions to make a computer device (can be a personal computer, A server, or a network device, etc.) executes the methods described in various embodiments of the present invention.

最后应说明的是：以上实施例仅用以说明本发明的技术方案，而 非对其限制；尽管参照前述实施例对本发明进行了详细的说明，本领 域的普通技术人员应当理解：其依然可以对前述各实施例所记载的技 术方案进行修改，或者对其中部分技术特征进行等同替换；而这些修 改或者替换，并不使相应技术方案的本质脱离本发明各实施例技术方案的精神和范围。Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: it can still be Modifications are made to the technical solutions described in the foregoing embodiments, or equivalent replacements are made to some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims (8)

1. A pre-operative planning method of total knee arthroplasty based on deep learning, the pre-operative planning method being based on patient lower limb medical image data, the pre-operative planning method comprising:

a step of medical image data processing based on deep learning, wherein a three-dimensional image of a bone structure is obtained through the medical image processing, and a key axis, a key anatomical site and key anatomical parameters are identified and marked; the bone structure includes a femur, tibia, fibula, and patella; the key anatomical sites comprise central points on different layers of femoral marrow cavities, central points on different layers of tibial marrow cavities, hip joint central points, knee joint central points, intercondylar spines central points and ankle joint central points; the key axis comprises a femur anatomical shaft, a femur mechanical shaft, a tibia anatomical shaft and a tibia mechanical shaft; the key anatomical parameters include a tibial femoral angle and a distal femoral angle;

a step of visual simulation matching, in which a three-dimensional prosthesis is subjected to simulation matching with a three-dimensional femur and a three-dimensional tibia, and a simulation matching effect is observed in real time; when the simulation matching effect meets the operation requirement, the simulation matching is considered to be completed;

the three-dimensional prosthesis comprises a three-dimensional femoral prosthesis and a three-dimensional tibial prosthesis; and

The simulated matching includes:

(i) Automatically implanting a three-dimensional femoral prosthesis into the femur, implanting a three-dimensional tibial prosthesis into the tibia;

(ii) Selecting a three-dimensional femoral prosthesis and a three-dimensional tibial prosthesis, and selecting simulated operation conditions;

(iii) Intelligent osteotomy is performed according to the matching relation between the three-dimensional prosthesis and the bone, and the simulation matching effect of the three-dimensional prosthesis and the bone is observed;

(iv) If the simulation matching effect does not meet the operation requirement, repeating the steps ii to iii until the simulation matching effect meets the operation requirement;

in step ii:

selecting a three-dimensional femoral prosthesis includes selecting a femoral prosthesis type and/or a femoral prosthesis model and/or a three-dimensional spatial location;

selecting a three-dimensional tibial prosthesis includes selecting a tibial prosthesis type and/or a tibial prosthesis model and/or a three-dimensional spatial location; and/or

Selecting the simulated surgical conditions includes selecting femoral surgical parameters and/or selecting tibial surgical parameters; the femur operation parameters comprise femur distal end osteotomy quantity, femur back condyle osteotomy quantity, internal and external rotation angles, internal and external turning angles and femur prosthesis buckling angles; the tibial operation parameters comprise tibial osteotomy quantity, internal and external rotation angles, internal and external turning angles and a back inclination angle;

the Unet neural network comprises n up-sampling layers and n down-sampling layers;

Each upsampling layer comprises an upsampling operation layer and a convolution layer;

each downsampling layer comprises a convolution layer and a pooling layer;

n has a value of 4-5;

each up-sampling layer comprises 1 up-sampling operation layer and 2 convolution layers, wherein the convolution kernel size in the convolution layers is 3*3, the convolution kernel size in the up-sampling operation layers is 2 x 2, and the number of the convolution kernels in each up-sampling layer is 512, 256, 256 and 128;

each downsampling layer comprises 2 convolution layers and 1 pooling layer, wherein the convolution kernel size in the convolution layers is 3*3, the convolution kernel size in the pooling layers is 2 x 2, and the number of the convolution kernels in each convolution layer is 128, 256, 256 and 512;

dividing the data set into a training set and a testing set according to the proportion of 7:3;

setting a dropout layer after the last up-sampling is finished, wherein the dropout rate is set to be 0.5-0.7; and/or

An activation layer is arranged behind all the convolution layers, and an activation function used by the activation layer is a relu function;

training was performed as follows:

rough segmentation: the training set is sent into a rough segmentation neural network for training; in the training process, the background pixel value of the data tag is set to be 0, the femur is 1, the tibia is 2, the fibula is 3, the patella is 4, the training batch_size is 6, the learning rate is set to be 1e-4, the optimizer uses an Adam optimizer, and the loss function used is DICE loss; according to the change of the loss function in the training process, the size of the training batch is adjusted;

Accurate segmentation: sending into an accurate segmentation neural network for accurate segmentation; the initial process comprises the steps of up-sampling a rough segmentation prediction result by using bilinear interpolation, selecting a plurality of points which are the most uncertain in a feature map, calculating feature representation of the points and predicting labels; repeating the initial process until the up-sample is to the required size;

selecting a point with a confidence level of 0.5 as the most uncertain point;

the characteristics of the points are calculated through Bilinear interpolation Bilinear;

the lower limb medical image data are CT scan data.

2. The method of claim 1, wherein the step of determining the position of the substrate comprises,

the step of medical image data processing comprises the step of bone three-dimensional image reconstruction; a step of image segmentation based on deep learning; a step of identifying a key axis, a key anatomical site, and a key anatomical parameter; or (b)

The step of medical image data processing comprises the step of bone three-dimensional image reconstruction; dividing the image; a step of identifying a key axis, a key anatomical site, and a key anatomical parameter based on the deep learning; or (b)

The step of medical image data processing comprises the step of bone three-dimensional image reconstruction; a step of image segmentation based on deep learning; a step of identifying a key axis, a key anatomical site and a key anatomical parameter based on the deep learning.

3. The method of claim 2, wherein the step of determining the position of the substrate comprises,

image segmentation based on deep learning includes:

constructing a lower limb medical image database: acquiring a lower limb medical image dataset, and manually marking femur, tibia, fibula and patella areas; dividing the data set into a training set and a testing set; converting medical image data before labeling into a picture in a first format and storing the picture, and converting the labeled data into a picture in a second format and storing the picture;

establishing a segmentation neural network model; the segmentation neural network model comprises a coarse segmentation neural network and a precise segmentation neural network; the rough segmentation neural network is selected from one or more of FCN, segNet, unet, 3D-Unet, mask-RCNN, cavity convolution, ENet, CRFasRNN, PSPNet, parseNet, refineNet, reSeg, LSTM-CF, deepMask, deepLabV1, deep LabV2 and deep LabV 3; the precisely segmented neural network is selected from any one or more of EEfficientDet, simCLR, pointRend;

model training: training the split neural network model by using a training set, and testing by using a testing set; and

and performing segmentation by using the trained segmentation neural network model.

4. A method according to claim 2 or 3, characterized in that,

the step of identifying the key axis, key anatomical site and key anatomical parameter based on the deep learning comprises:

identifying a critical anatomical site; identifying key points by utilizing any one or more neural network models of MTCNN, locnet, pyramid Residual Module, densenet, hourglass, resnet, segNet, unet, R-CNN, fast R-CNN, R-FCN and SSD;

obtaining a critical axis using the critical anatomical site; and

key anatomical parameters are measured.

5. The method of claim 1, wherein the step of determining the position of the substrate comprises,

the skeleton structure can be displayed in any combination and/or switched in transparency and/or zoomed in and/or rotated and/or moved in images; the transparency includes transparent and opaque; and/or

The key axis also comprises any one or more of a intercondylar line, a posterior condylar line, a tibia knee joint line, a femur sagittal axis and a femur knee joint line; the key anatomical parameters also include the femoral posterior condylar angle.

6. A pre-operative planning system for total knee replacement based on deep learning, the pre-operative planning system comprising:

The medical image data processing module is used for obtaining a three-dimensional image of a bone structure, identifying and marking a key axis, a key anatomical site and key anatomical parameters; the bone structure includes a femur, tibia, fibula, and patella; the key axis comprises a femur anatomical shaft, a femur mechanical shaft, a tibia anatomical shaft and a tibia mechanical shaft; the key anatomical parameters include a tibial femoral angle and a distal femoral angle;

the simulation matching module is used for performing simulation matching on the three-dimensional prosthesis, the three-dimensional femur and the three-dimensional tibia, and observing the simulation matching effect in real time; and

and a display module: the method is used for displaying three-dimensional images, key axes, key anatomical sites, key anatomical parameters and simulated matching effects of the bone structure;

the three-dimensional prosthesis comprises a three-dimensional femoral prosthesis and a three-dimensional tibial prosthesis; and

the simulated matching includes:

(i) Automatically implanting a three-dimensional femoral prosthesis into the femur, implanting a three-dimensional tibial prosthesis into the tibia;

(ii) Selecting a three-dimensional femoral prosthesis and a three-dimensional tibial prosthesis, and selecting simulated operation conditions;

(iii) Intelligent osteotomy is performed according to the matching relation between the three-dimensional prosthesis and the bone, and the simulation matching effect of the three-dimensional prosthesis and the bone is observed;

(iv) If the simulation matching effect does not meet the operation requirement, repeating the steps ii to iii until the simulation matching effect meets the operation requirement;

In step ii:

selecting a three-dimensional femoral prosthesis includes selecting a femoral prosthesis type and/or a femoral prosthesis model and/or a three-dimensional spatial location;

selecting a three-dimensional tibial prosthesis includes selecting a tibial prosthesis type and/or a tibial prosthesis model and/or a three-dimensional spatial location; and/or

Selecting the simulated surgical conditions includes selecting femoral surgical parameters and/or selecting tibial surgical parameters; the femur operation parameters comprise femur distal end osteotomy quantity, femur back condyle osteotomy quantity, internal and external rotation angles, internal and external turning angles and femur prosthesis buckling angles; the tibial operation parameters comprise tibial osteotomy quantity, internal and external rotation angles, internal and external turning angles and a back inclination angle;

the Unet neural network comprises n up-sampling layers and n down-sampling layers;

each upsampling layer comprises an upsampling operation layer and a convolution layer;

each downsampling layer comprises a convolution layer and a pooling layer;

n has a value of 4-5;

each up-sampling layer comprises 1 up-sampling operation layer and 2 convolution layers, wherein the convolution kernel size in the convolution layers is 3*3, the convolution kernel size in the up-sampling operation layers is 2 x 2, and the number of the convolution kernels in each up-sampling layer is 512, 256, 256 and 128;

each downsampling layer comprises 2 convolution layers and 1 pooling layer, wherein the convolution kernel size in the convolution layers is 3*3, the convolution kernel size in the pooling layers is 2 x 2, and the number of the convolution kernels in each convolution layer is 128, 256, 256 and 512;

Dividing the data set into a training set and a testing set according to the proportion of 7:3;

setting a dropout layer after the last up-sampling is finished, wherein the dropout rate is set to be 0.5-0.7; and/or

An activation layer is arranged behind all the convolution layers, and an activation function used by the activation layer is a relu function;

training was performed as follows:

rough segmentation: the training set is sent into a rough segmentation neural network for training; in the training process, the background pixel value of the data tag is set to be 0, the femur is 1, the tibia is 2, the fibula is 3, the patella is 4, the training batch_size is 6, the learning rate is set to be 1e-4, the optimizer uses an Adam optimizer, and the loss function used is DICE loss; according to the change of the loss function in the training process, the size of the training batch is adjusted;

accurate segmentation: sending into an accurate segmentation neural network for accurate segmentation; the initial process comprises the steps of up-sampling a rough segmentation prediction result by using bilinear interpolation, selecting a plurality of points which are the most uncertain in a feature map, calculating feature representation of the points and predicting labels; repeating the initial process until the up-sample is to the required size;

selecting a point with a confidence level of 0.5 as the most uncertain point;

the characteristics of the points are calculated through Bilinear interpolation Bilinear;

The lower limb medical image data are CT scan data.

7. An apparatus, comprising:

one or more processors;

a storage means for storing one or more programs;

when executed by the one or more processors, causes the one or more processors to implement the method of any of claims 1 to 5.

8. A computer readable storage medium, on which a computer program is stored, characterized in that the program, when being executed by a processor, implements the method of any one of claims 1 to 5.