(5.4) judging whether the DiceLoss value calculated in the step (5.3) is smaller than a preset threshold value or not, if so, stopping iteration, and finishing the fine registration of the single vertebral image; otherwise, setting the objective function of the Adam parameter optimization algorithm as Dice Loss, setting the parameter vector as the current fine registration parameter, then repeating the steps (5.2) - (5.4), and searching the optimal fine registration parameter with the smallest DiceLoss value through the Adam parameter optimization algorithm, thereby completing the fine registration of the single vertebral block diagram;

(5.5) judging whether the precise registration of all the single vertebral images is finished, if not, repeating the steps (5.2) - (5.4) until the precise registration of all the single vertebral images is finished; otherwise, entering the step (5.6);

and (5.6) carrying out spatial transformation on all the single vertebral images according to the corresponding optimized fine registration parameters, and combining according to the positions before segmentation so as to realize the CT level registration of the spine.

The invention aims to realize the following steps:

the invention relates to a 2D/3D spine CT level registration method based on a deep learning network, which mainly comprises two steps of coarse registration and fine registration; firstly, generating deformation of a 3D CT sequence, generating a DRR image through projection of an X-ray imaging calculation model, and then randomly selecting the DRR image to train a deep learning network; then deforming the 3D image to be registered before the operation, generating a DRR through projection of an X-ray imaging model, and inputting the DRR and the 2D reference image in the operation into a depth learning network to obtain a coarse registration parameter; and finally, based on the coarse registration parameters, finishing the precise registration of a plurality of vertebrae in the preoperative 3D image to be registered through an Adam parameter optimization algorithm, and realizing the CT level registration of the spine.

Meanwhile, the 2D/3D spine CT level registration method based on the deep learning network also has the following beneficial effects:

(1) the invention adopts a hierarchical registration mode, not only integrates a deep learning network, but also matches a classical parameter optimization mode, and ensures that the registration precision is more excellent through the combination of two registration methods, and not only performs rigid registration on the vertebra, but also considers the deformation between the vertebrae.

(2) Compared with the traditional mode that the vertebra is taken as a whole rigid body, the mode of circularly and accurately registering a plurality of vertebrae is adopted, the single-block step-by-step registration accuracy is higher, because 2D images in operation and 3D before operation are considered, the posture change of a patient under imaging equipment causes fine deformation between the vertebrae, if the vertebrae are taken as a rigid body, the registration result is difficult to avoid to be rough, and the accurate registration of a plurality of vertebrae solves the problem.

(3) The segmentation of the vertebrae into a plurality of vertebrae before the fine registration is not mentioned in the conventional registration method, and the purpose of the segmentation is to improve the efficiency of the registration by performing the registration on a plurality of vertebrae in the fine registration, namely segmenting each 3D vertebra into single blocks.

Drawings

FIG. 1 is a flowchart of a deep learning network-based 2D/3D spine CT level registration method of the present invention;

FIG. 2 is an X-ray imaging computational model;

fig. 3 is a diagram of a deep learning convolutional network architecture.

Detailed Description

The following description of the embodiments of the present invention is provided in order to better understand the present invention for those skilled in the art with reference to the accompanying drawings. It is to be expressly noted that in the following description, a detailed description of known functions and designs will be omitted when it may obscure the subject matter of the present invention.

Examples

For convenience of description, the related terms appearing in the detailed description are explained:

GPU (graphics Processing Unit): a graphics processor;

drr (digital reconstructed radiograms): digitally reconstructing a radiological image;

adam parameter optimization algorithm: adaptive motion Estimation.

FIG. 1 is a flow chart of a deep learning network-based 2D/3D spine CT level registration method of the invention.

In this embodiment, as shown in fig. 1, the 2D/3D spine CT level registration method based on a deep learning network of the present invention includes the following steps:

s1, acquiring an X-ray image as an intraoperative 2D reference image in the training and registration process, and acquiring a medical CT sequence as an preoperative 3D image in the training process;

s2, constructing a training image set;

s2.1, inputting the preoperative 3D image into a rigid body transformation model, and randomly transforming six-dimensional rigid body transformation parameter T ═ T (T)_x,t_y,t_z,r_x,r_y,r_z) Generating a group of three-dimensional image sequences, and then inputting the three-dimensional image sequences into an X-ray imaging calculation model for projection so as to generate a DRR image sequence; wherein, t_xRepresenting a translation parameter, t, on the X-axis_yRepresenting a translation parameter, t, in the Y-axis_zRepresenting a translation parameter in the Z-axis, r_xRepresenting a rotation parameter along the X-axis, r_yRepresenting a rotation parameter along the Y-axis, r_zRepresenting a rotation parameter along the Z-axis; then, in this embodiment, the rotation matrices around the X-axis, the Y-axis, and the Z-axis can be expressed by the following equations, respectively:

the translation matrix is represented as: t is_l(t_x,t_y,t_z)^T；

If the image is first rotated around the X-axis, Y-axis, and Z-axis in sequence, and then translated, the pixel coordinates before and after the rigid body transformation can be expressed as:

wherein, (x, y, z)^TRepresenting the spatial coordinates of a certain pixel point in the floating image,

representing the space coordinate of the pixel point after rigid body transformation;

in this embodiment, as shown in fig. 2, the X-Ray imaging calculation model may be implemented by using a Ray-Casting algorithm based on a GPU, and the model specifically includes:

wherein, I represents the energy of the X-ray after attenuation, I₀Denotes the initial energy of X-rays, μ_iRepresents the linear attenuation coefficient of the ith voxel tissue, d_iRepresenting the distance traveled by the ray in the ith voxel;

s2.2, combining the DRR image sequences in pairs, wherein one image is used as a reference image, the other image is used as a floating image, and the two images form a training sample to form a training image set;

as shown in fig. 3, an 8-layer CNN model is built as a deep learning network model and trained;

a first layer input layer inputting a floating image and a reference image;

the second layer is the first convolution layer, the convolution kernel size is 5 x 20, no padding, the step size is 1, and the output matrix size of the layer is 152 x 296 x 20;

the third layer was the first pooling layer, with maximum pooling, a pooling window size of 2 x 2, step size of 2, the layer output matrix of 76 x 148 x 20;

the fourth layer is the second convolution layer, the convolution kernel size is 5 x 20, no padding, the step size is 1, and the output matrix size of the layer is 72 x 144 x 20;

the fifth layer is the second pooling layer, maximum pooling is used, the pooling window size is 2 x 2, the step size is 2, the layer output matrix is 36 x 72 x 20;

the sixth layer is a full connection layer, 250 ReLU activation function units are provided, and the number of output nodes is 250;

the seventh layer is a second full link layer and is provided with 6 ReLU activation function units, and the number of output nodes is 6;

the eighth layer is an output layer which outputs 6 parameters, namely (t)_x,t_y,t_z,r_x,r_y,r_z)；

The method comprises the steps of sequentially inputting a floating image and a reference image in a training image set to a deep learning network model for training, subtracting the reference image from the floating image to obtain a residual image in the model training process, continuously extracting high-order characteristic information of the residual image through a network, seeking a deformation rule from the floating image to the reference image, outputting accurate 6 individual variable parameters, training by utilizing a TensorFlow frame and accelerating training by using a high-performance GPU and a CUDA (compute unified device architecture), wherein the specific training process is similar to a general deep learning network training process and is not repeated herein.

S4, carrying out coarse registration by using a deep learning network model;

according to the method of the step S2.1, a DRR image is generated by using the preoperative 3D image to be registered and is used as a floating image, and the floating image and the intraoperative 2D reference image are input into the trained depth learning network model together, so that the rough registration transformation parameters of the preoperative 3D image to be registered are output;

s5, performing fine registration of the single vertebra through an Adam parameter optimization algorithm;

s5.1, performing vertebra segmentation on the preoperative 3D image to be registered by using a Grow Cut region growing algorithm, so that each segmented sub-image only comprises one vertebra, and obtaining a plurality of single vertebra images;

s5.2, taking the rough registration transformation parameters as initial registration parameters of each single vertebral image, then carrying out rigid body transformation on the single vertebral image through the initial registration parameters, and then carrying out projection through an X-ray imaging calculation model to generate a DRR image of the single vertebral image as a floating image;

s5.3, calculating a DiceLoss value between the floating image of the single vertebra and the intraoperative 2D reference image;

s5.4, judging whether the Dice Loss value calculated in the step S5.3 is smaller than a preset threshold value or not, if so, stopping iteration, and finishing the precise registration of the single vertebral image; otherwise, setting the objective function of the Adam parameter optimization algorithm as Dice Loss, setting the parameter vector as the current fine registration parameter, then repeating the steps (5.2) - (5.4), and searching the optimal fine registration parameter with the smallest DiceLoss value through the Adam parameter optimization algorithm, thereby completing the fine registration of the single vertebral block diagram;

s5.5, judging whether all the single vertebral images are accurately registered, if not, repeating the steps S5.2-S5.4 until the accurate registration of all the single vertebral images is completed; otherwise, go to step S5.6;

and S5.6, performing spatial transformation on all the single vertebral images according to the correspondingly optimized fine registration parameters, and combining the single vertebral images according to the positions before segmentation so as to realize the CT level registration of the spine.

Although illustrative embodiments of the present invention have been described above to facilitate the understanding of the present invention by those skilled in the art, it should be understood that the present invention is not limited to the scope of the embodiments, and various changes may be made apparent to those skilled in the art as long as they are within the spirit and scope of the present invention as defined and defined by the appended claims, and all matters of the invention which utilize the inventive concepts are protected.

Claims

Translated fromChinese

1.一种基于深度学习网络的2D/3D脊椎CT层级配准方法，其特征在于，包括以下步骤：1. a 2D/3D spine CT level registration method based on deep learning network, is characterized in that, comprises the following steps:

(1)、获取X射线图像作为训练及配准过程的术中2D参考图像，获取医学CT序列作为训练过程的术前3D图像；(1) Acquire an X-ray image as an intraoperative 2D reference image for the training and registration process, and acquire a medical CT sequence as a preoperative 3D image for the training process;

(2)、构建训练图像集；(2), build a training image set;

(2.1)、将术前3D图像输入至刚体变换模型，通过随机变换六维的刚体变换参数T＝(t_x,t_y,t_z,r_x,r_y,r_z)，生成一组三维的图像序列，然后再输入至X射线成像计算模型进行投影，从而生成DRR图像序列；其中，t_x表示在X轴上的平移参数，t_y表示在Y轴上的平移参数，t_z表示在Z轴上的平移参数，r_x表示沿X轴的旋转参数，r_y表示沿Y轴的旋转参数，r_z表示沿Z轴的旋转参数；(2.1) Input the preoperative 3D image into the rigid body transformation model, and generate a set of three-dimensional transformation parameters T=(t_x , t_y ,_t_z , r_x , ry , r_z ) by randomly transforming the six-dimensional rigid body transformation parameters , and then input it into the X-ray imaging calculation model for projection to generate a DRR image sequence; where t_x represents the translation parameter on the X axis,_{ty represents the translation parameter on the Y axis, and t z}_represents the translation parameter on the Y axis. The translation parameter on the Z axis, r_x represents the rotation parameter along the X axis, ry represents the rotation parameter along the Y axis, and r_z represents the rotation parameter along the_Z axis;

(2.2)、对DRR图像序列进行两两组合，一张作为参考图像，另一张作为浮动图像，两张图像构成一个训练样本，从而组成训练图像集；(2.2) Combine the DRR image sequences in pairs, one as a reference image and the other as a floating image, and the two images constitute a training sample, thereby forming a training image set;

(3)、搭建深度学习网络模型并训练；(3), build a deep learning network model and train it;

搭建8层CNN模型作为深度学习网络模型，然后将训练图像集中的参考图像与浮动图像依次输入，用于模型训练，当模型收敛时能够准确输出浮动图像对应的形变参数；Build an 8-layer CNN model as a deep learning network model, and then input the reference images and floating images in the training image set in turn for model training. When the model converges, the deformation parameters corresponding to the floating images can be accurately output;

(4)、利用深度学习网络模型进行粗配准；(4) Coarse registration using deep learning network model;

按照步骤(2.1)所述方法，利用术前待配准3D图像生成一幅DRR图像，并作为浮动图像，再将浮动图像与术中2D参考图像一起输入至训练完成的深度学习网络模型，从而输出术前待配准3D图像的粗配准变换参数；According to the method described in step (2.1), use the preoperative 3D image to be registered to generate a DRR image and use it as a floating image, and then input the floating image together with the intraoperative 2D reference image to the deep learning network model after training, so as to Output the coarse registration transformation parameters of the preoperative 3D image to be registered;

(5)、通过Adam参数优化算法进行单块椎骨的精配准；(5) Accurate registration of a single vertebra by Adam parameter optimization algorithm;

(5.1)、利用用Grow Cut区域生长算法对术前待配准3D图像进行椎骨分割，使分割的每幅子图中仅包含一块椎骨，从而得到多幅单块椎骨图；(5.1) Use the Grow Cut region growth algorithm to segment the vertebrae on the preoperative 3D image to be registered, so that each segmented sub-image contains only one vertebra, thereby obtaining multiple single-block vertebra maps;

(5.2)、将粗配准变换参数作为每幅单块椎骨图的初始配准参数，然后将单块椎骨经初始配准参数的刚体变换后，再通过X射线成像计算模型进行投影，生成单块椎骨图的DRR图像，作为浮动图像；(5.2), take the coarse registration transformation parameters as the initial registration parameters of each single vertebra map, and then transform the single vertebra through the rigid body transformation of the initial registration parameters, and then perform projection through the X-ray imaging calculation model to generate a single vertebra. DRR images of block vertebrae, as floating images;

(5.3)、计算单块椎骨的浮动图像与术中2D参考图像之间的DiceLoss值；(5.3), calculate the DiceLoss value between the floating image of a single vertebra and the intraoperative 2D reference image;

其中，|X|表示浮动图像的像素矩阵X中所有元素之和，|Y|表示参考图像的像素矩阵Y中所有元素之和，|X∩Y|表示像素矩阵X与像素矩阵Y对应元素点乘后再求所有元素之和；Among them, |X| represents the sum of all elements in the pixel matrix X of the floating image, |Y| represents the sum of all elements in the pixel matrix Y of the reference image, and |X∩Y| represents the corresponding element point of the pixel matrix X and the pixel matrix Y After multiplying, find the sum of all elements;

(5.4)、判断步骤(5.2)中计算得到的DiceLoss值是否小于预设阈值，如果小于，则迭代停止，完成单块椎骨图的精配准；否则，设置Adam参数优化算法的目标函数为DiceLoss，参数向量设置为当前精配准参数，然后重复步骤(5.2)～(5.4)，通过Adam参数优化算法搜寻DiceLoss值最小时的最优精配准参数，从而完成该单块椎骨图的精配准；(5.4), determine whether the DiceLoss value calculated in step (5.2) is less than the preset threshold, if it is less than, the iteration stops to complete the precise registration of the single vertebral map; otherwise, the objective function of the Adam parameter optimization algorithm is set to DiceLoss , the parameter vector is set as the current fine registration parameters, and then repeat steps (5.2) to (5.4), through the Adam parameter optimization algorithm to search for the optimal fine registration parameters when the DiceLoss value is the smallest, so as to complete the fine registration of the single vertebral map allow;

(5.5)、判断所有单块椎骨图是否都完成了精配准，如果未完成，则重复步骤(5.2)～(5.4)，直到完成所有单块椎骨图的精配准；否则，进入步骤(5.6)；(5.5), determine whether all single vertebrae have completed the fine registration, if not, repeat steps (5.2) to (5.4) until the fine registration of all single vertebrae is completed; otherwise, go to step ( 5.6);

(5.6)、将所有的单块椎骨图根据对应优化后的精配准参数进行空间变换，再按照分割前的位置进行组合，从而实现脊椎CT层级配准。(5.6) Perform spatial transformation on all single vertebral images according to the corresponding optimized precise registration parameters, and then combine them according to the positions before segmentation, so as to realize spinal CT level registration.

2.根据权利要求1所述的基于深度学习网络的2D/3D脊椎CT层级配准方法，其特征在于，所述深度学习网络模型的具体结构为：2. the 2D/3D spine CT level registration method based on deep learning network according to claim 1, is characterized in that, the concrete structure of described deep learning network model is:

第一层输入层，输入浮动图像和参考图像；The first layer of input layer, input floating image and reference image;

第二层是第一卷积层，卷积核大小为5*5*20，不填充，步长为1，该层输出矩阵大小为152*296*20；The second layer is the first convolution layer, the size of the convolution kernel is 5*5*20, no padding, the stride is 1, and the output matrix size of this layer is 152*296*20;

第三层是第一池化层，采用最大值池化，池化窗口尺寸为2*2，步长为2，该层输出矩阵为76*148*20；The third layer is the first pooling layer, using maximum pooling, the pooling window size is 2*2, the step size is 2, and the output matrix of this layer is 76*148*20;

第四层是第二卷积层，卷积核尺寸为5*5*20，不填充，步长为1，该层输出矩阵大小为72*144*20；The fourth layer is the second convolution layer, the size of the convolution kernel is 5*5*20, no padding, the step size is 1, and the output matrix size of this layer is 72*144*20;

第五层是第二池化层，采用最大值池化，池化窗口尺寸为2*2，步长为2，该层输出矩阵为36*72*20；The fifth layer is the second pooling layer, using maximum pooling, the pooling window size is 2*2, the step size is 2, and the output matrix of this layer is 36*72*20;

第六层是全连接层，有250个ReLU激活函数单元，输出结点个数为250个；The sixth layer is a fully connected layer with 250 ReLU activation function units and 250 output nodes;

第七层为第二个全链接层，有6个ReLU激活函数单元，输出结点个数为6个；The seventh layer is the second full link layer, with 6 ReLU activation function units, and the number of output nodes is 6;

第八层为输出层，输出6个参数，即(t_x,t_y,t_z,r_x,r_y,r_z)。The eighth layer is the output layer, which outputs 6 parameters, namely (t_x ,_{ty , t z}_,_r_x , ry , r_z ).