技术领域technical field
本发明涉及医学图像处理领域，尤其涉及基于多路卷积神经网络的MRI图像心脏结构分割方法。The invention relates to the field of medical image processing, in particular to a method for segmenting cardiac structures in MRI images based on a multi-channel convolutional neural network.
背景技术Background technique
根据世界卫生组织统计，心血管疾病是全球致死率最高的疾病，2016年约有1970万人死于心血管疾病。在临床中，心脏功能分析对于心脏疾病诊断、风险评估、患者管理、治疗决策具有重要作用。这通常是借助心脏数字图像通过评估一系列临床指标例如心室体积、射血分数、每搏量、心肌质量等，来定量分析全局或局部心脏功能。由于对软组织的良好判别性，通过电影MRI图像进行左、右心室射血分数、每搏量、左心室质量、心肌厚度的评估成为心脏功能分析的金标准.而这些定量指标的评估，需要对舒张及收缩末期这两个阶段的左心室内膜和心外膜，以及右室内膜进行准确的分割。在临床实践中，医生手动分割不仅费时费力，依赖医生经验，不同医生甚至同一医生的两次分割结果还都具有很大的可变性。因此，迫切需要准确的自动分割方法。According to the statistics of the World Health Organization, cardiovascular disease is the disease with the highest mortality rate in the world, and about 19.7 million people died of cardiovascular disease in 2016. In clinical practice, cardiac function analysis plays an important role in heart disease diagnosis, risk assessment, patient management, and treatment decision-making. This is usually done with the aid of digital images of the heart to quantify global or regional cardiac function by assessing a range of clinical parameters such as ventricular volume, ejection fraction, stroke volume, myocardial mass, etc. Due to the good discrimination of soft tissues, the evaluation of left and right ventricular ejection fraction, stroke volume, left ventricular mass, and myocardial thickness through cine MRI images has become the gold standard for cardiac function analysis. The evaluation of these quantitative indicators requires The endocardium and epicardium of the left ventricle and the endocardium of the right ventricle are accurately segmented in diastole and end-systole. In clinical practice, manual segmentation by doctors is not only time-consuming and laborious, but also depends on the experience of doctors, and the results of two segmentations by different doctors or even the same doctor have great variability. Therefore, accurate automatic segmentation methods are urgently needed.
心脏电影MRI图像中心脏结构的分割目前有人工手动分割方式，传统自动分割方法以及基于深度学习的分割方法。人工手动分割方式是医生在二维图像上逐层进行人工描绘，再根据人工描绘的结果进行进一步的分析和诊断。但人工标注工作量大，耗时费力，可重复性差，使得医生标注能力不足以满足大量潜在病人的需求；同时，标注人员专业水平及经验差异较大，人工分割结果存在较大差异，质量无法保证。The segmentation of cardiac structures in cardiac cine MRI images currently includes manual segmentation methods, traditional automatic segmentation methods, and segmentation methods based on deep learning. The manual segmentation method is that the doctor manually draws layer by layer on the two-dimensional image, and then conducts further analysis and diagnosis according to the results of the manual drawing. However, manual labeling is labor-intensive, time-consuming and laborious, and has poor repeatability, which makes doctors' labeling capabilities insufficient to meet the needs of a large number of potential patients. ensure.
传统自动分割方法，基于图像或可变形模型的方法可通过与用户交互的方式完成分割过程，需要人工进行分割结果的确认及标注结果调整。基于模型的分割方法如活动形状模型、图谱模型，可采用以大量数据构建大体模型的方式来减少用户交互，来完成自动分割过程。然而，基于图像和可变形模型的心脏分割方法通常需要用户交互，鲁棒性差，且分割准确度低。基于模型的方法如活动形状模型、图谱模型的方法虽然可减少用户交互，但是不同人(包括正常人以及有心脏疾病的病人)心脏形状以及动态多种多样，建立包含心室所有可能的形状的通用模型是很困难的，存在模型通用性、泛化性能差的问题。Traditional automatic segmentation methods, methods based on images or deformable models can complete the segmentation process by interacting with users, and require manual confirmation of segmentation results and adjustment of labeling results. Model-based segmentation methods such as active shape models and graph models can reduce user interaction by building a general model with a large amount of data to complete the automatic segmentation process. However, heart segmentation methods based on images and deformable models usually require user interaction, have poor robustness, and suffer from low segmentation accuracy. Although model-based methods such as active shape models and atlas model methods can reduce user interaction, the heart shapes and dynamics of different people (including normal people and patients with heart disease) are diverse, and a general model containing all possible shapes of ventricles can be established. The model is very difficult, and there are problems of model versatility and poor generalization performance.
随着近年深度学习的发展，基于深度学习的分割方法被引入到心脏MRI图像分割中来。基于深度学习的分割方法自动从原始心脏图像中提取特征完成自动分割过程，通常无需用户交互。基于深度学习的方法可以得到较为准确的全自动分割结果，但是现有基于深度学习的心脏分割方法，多采用2D的分割方法并未考虑层间上下文信息，层间上下文信息对于准确分割、提升分割性能是很有价值的。忽略层间上下文信息不符合临床医生的实际工作流程。同时，由于心脏电影MRI图像自身扫描层厚、间距大的特点，直接通过3D分割方法利用层间上下文信息不仅计算开销大而且可能无法带来性能提升。With the development of deep learning in recent years, segmentation methods based on deep learning have been introduced into cardiac MRI image segmentation. The segmentation method based on deep learning automatically extracts features from the original heart image to complete the automatic segmentation process, usually without user interaction. The method based on deep learning can obtain more accurate automatic segmentation results. However, the existing heart segmentation methods based on deep learning mostly use 2D segmentation methods without considering the inter-layer context information. The inter-layer context information is very important for accurate segmentation and improved segmentation. Performance is valuable. Ignoring inter-layer contextual information does not conform to the actual workflow of clinicians. At the same time, due to the characteristics of thick scanning layers and large spacing of cardiac cine MRI images, using inter-layer context information directly through 3D segmentation methods not only has high computational overhead but may not bring about performance improvements.
因此，如何提高心脏电影MRI图像自动分割的准确性、提升分割性能成为急需解决的问题。Therefore, how to improve the accuracy of automatic segmentation of cardiac cine MRI images and improve the segmentation performance has become an urgent problem to be solved.
发明内容Contents of the invention
针对现有技术之不足，本发明提出一种基于多路卷积神经网络的MRI图像心脏结构分割方法，所述方法包括：Aiming at the deficiencies in the prior art, the present invention proposes a method for segmenting the heart structure of an MRI image based on a multi-channel convolutional neural network, the method comprising:
步骤1：收集心脏电影MRI训练数据，包括正常人的正常心脏电影MRI图像和心脏病人的异常心脏电影MRI图像，所述心脏电影MRI训练数据包括心脏舒张及收缩阶段的电影MRI图像；Step 1: Collect cardiac cine MRI training data, including normal cardiac cine MRI images of normal people and abnormal cardiac cine MRI images of heart patients, and the cardiac cine MRI training data includes cine MRI images of diastolic and systolic stages of the heart;
步骤2：由经验丰富的医生手动对心脏电影MRI训练数据中的心脏结构进行逐层标注，并将标注结果作为心脏分割标准结果；Step 2: Manually label the heart structure in the heart movie MRI training data layer by layer by experienced doctors, and use the labeling results as the heart segmentation standard results;
步骤3：心脏区域提取生成对抗网络训练，设计并建立基于生成对抗网络的心脏区域提取模型，从收集到的所述心脏电影MRI训练数据中提取出心脏区域图像，提取出的心脏区域图像由多个心脏MRI切片图像叠加构成；Step 3: Heart region extraction Generative adversarial network training, designing and establishing a heart region extraction model based on a generative adversarial network, extracting heart region images from the collected heart movie MRI training data, and extracting heart region images by multiple A heart MRI slice image is superimposed to form;
步骤4：基于多路卷积神经网络的心脏分割网络训练，设计并建立结合层间上下文信息的深度卷积分割网络，步骤包括：Step 4: Heart segmentation network training based on multi-channel convolutional neural network, design and establish a deep convolutional segmentation network combining inter-layer context information, the steps include:
步骤41：对步骤3提取出的心脏区域进行心脏结构的分割，以迭代的方式将相邻层的MRI切片图像信息以及相邻上层的已有分割结果信息作为层间上下文信息；Step 41: Segment the cardiac structure of the heart region extracted in step 3, and iteratively use the MRI slice image information of the adjacent layer and the existing segmentation result information of the adjacent upper layer as inter-layer context information;
步骤42：将所述层间上下文信息分别输入到各自对应的特征提取分支中，，每个所述特征提取分支采用独立的并行结构，即每个层间上下文特征提取分支采用相同的网络结构但是独立处理其对应的一种层间上下文信息；Step 42: Input the inter-layer context information into their corresponding feature extraction branches, each of the feature extraction branches adopts an independent parallel structure, that is, each inter-layer context feature extraction branch adopts the same network structure but Independently process its corresponding inter-layer context information;
步骤43：各分支通过叠加多个卷积及池化操作提取图像的高层抽象特征，并经特征融合模块进行融合；Step 43: each branch extracts the high-level abstract features of the image by superimposing multiple convolution and pooling operations, and fuses them through the feature fusion module;
步骤44：特征融合模块先将每个所述特征提取分支提取到的高层抽象特征进行串连，然后通过ASPP模块进一步融合这些高层特征，融合后的特征经由解码模块的上采样、局部细节信息补偿以及卷积操作将图像恢复到输入进分割网络时的尺寸，得到端到端的密集多结构同时分割概率图，并由概率图确定各个像素所属类别从而得到最终的分割结果；Step 44: The feature fusion module first concatenates the high-level abstract features extracted by each feature extraction branch, and then further fuses these high-level features through the ASPP module, and the fused features are compensated by upsampling and local detail information of the decoding module And the convolution operation restores the image to the size when it is input into the segmentation network, obtains an end-to-end dense multi-structure simultaneous segmentation probability map, and determines the category of each pixel from the probability map to obtain the final segmentation result;
步骤5：将步骤4心脏结构分割网络得到的分割结果与步骤2得到的心脏标准分割结果进行比较，通过性能评价指标进行分割结果量化评估；Step 5: Compare the segmentation results obtained by the heart structure segmentation network in step 4 with the heart standard segmentation results obtained in step 2, and perform quantitative evaluation of the segmentation results through performance evaluation indicators;
步骤6：收集待分割心脏电影MRI数据，利用步骤3训练好的心脏区域提取模型提取心脏区域，并记录提取位置信息，然后将提取的心脏区域的MRI切片图像输入到步骤4训练好的心脏分割网络中，自心底到心顶依次迭代完成心脏区域体数据的分割，剔除分割结果中各个心脏结构可能存在的不连续的零散区域，得到心脏结构初始分割结果；Step 6: Collect MRI data of the cardiac movie to be segmented, use the cardiac region extraction model trained in step 3 to extract the cardiac region, and record the extracted position information, and then input the extracted MRI slice image of the cardiac region to the cardiac segmentation trained in step 4 In the network, the segmentation of heart region volume data is iteratively completed from the bottom of the heart to the top of the heart, and the discontinuous scattered regions that may exist in each heart structure in the segmentation result are eliminated to obtain the initial segmentation result of the heart structure;
步骤7：根据所述提取位置信息，将所述心脏结构初始分割结果恢复到原图尺寸，得到最终的分割结果。Step 7: According to the extracted position information, restore the initial segmentation result of the heart structure to the size of the original image to obtain the final segmentation result.
根据一种优选的实施方式，步骤3中的所述心脏区域提取模型包括生成器和判别器，According to a preferred embodiment, the heart region extraction model in step 3 includes a generator and a discriminator,
所述生成器以心脏MRI切片图像作为输入，采用编码解码结构，即先通过卷积、池化操作、下采样提取特征，再通过上采样、局部细节补偿以及卷积操作生成与输入的心脏MRI切片图像尺寸一致的伪心脏轮廓图像；The generator takes cardiac MRI slice images as input and adopts an encoding and decoding structure, that is, extracts features through convolution, pooling operations, and downsampling, and then generates and inputs cardiac MRI images through upsampling, local detail compensation, and convolution operations. Pseudo-heart contour image with consistent slice image size;
所述判别器以心脏MRI切片图像和相应的真实心脏区域轮廓图像对或心脏MRI切片图像和生成器生成的伪心脏轮廓图像对作为输入，通过卷积和池化操作提取特征，判别输入的心脏轮廓图像是真实的还是由生成器生成的；The discriminator takes the cardiac MRI slice image and the corresponding real heart region contour image pair or the cardiac MRI slice image and the pseudo heart contour image pair generated by the generator as input, extracts features through convolution and pooling operations, and distinguishes the input heart Whether the contour image is real or generated by the generator;
生成对抗网络训练好后，通过生成器生成准确的各个心脏MRI切片图像对应的心脏轮廓图像，定位出心脏在图像上的位置，从而提取出三维心脏区域图像。After the generative confrontation network is trained, the generator generates accurate cardiac contour images corresponding to each cardiac MRI slice image, locates the position of the heart on the image, and extracts a three-dimensional cardiac region image.
本发明的有益效果在于：The beneficial effects of the present invention are:
1、本发明采用基于生成对抗网络的心脏区域提取模型，能够自动的、更加准确的提取心脏区域，并记录提取位置，无需人工进行交互，通用性、泛化能力较好。1. The present invention adopts a heart region extraction model based on a generative adversarial network, which can automatically and more accurately extract the heart region and record the extraction position without manual interaction, and has good versatility and generalization ability.
2、本发明技术方案在训练分割心脏结构的神经网络中，利用层间上下文信息，即利用相邻层之间的空间关联信息，来提高心脏结构分割的精度和准确度，训练好的神经网络可完成心脏结构的自动分割。2. The technical solution of the present invention uses inter-layer context information, that is, utilizes the spatial correlation information between adjacent layers, to improve the precision and accuracy of cardiac structure segmentation in the neural network for training and segmenting cardiac structures, and the trained neural network Automatic segmentation of cardiac structures can be completed.
3、由于直接通过3D分割方法利用层间上下文信息具有计算开销大，且有限数据集下性能受限的缺点，本发明在2D分割方法框架下，采用独立的并行结构来处理对应的层间上下文信息，有效提高了分割性能。3. Since the use of inter-layer context information directly through the 3D segmentation method has the disadvantages of high computational overhead and limited performance under limited data sets, the present invention uses an independent parallel structure to process the corresponding inter-layer context under the framework of the 2D segmentation method information, which effectively improves the segmentation performance.
附图说明Description of drawings
图1是本发明心脏结构自动分割方法的流程图；Fig. 1 is the flowchart of the heart structure automatic segmentation method of the present invention;
图2是本发明基于生成对抗网络提取心脏的工作原理图；和Fig. 2 is the working principle diagram of extracting the heart based on the generation confrontation network of the present invention; and
图3是本发明基于多路卷积神经网络进行心脏分割网络的工作原理图。Fig. 3 is a working principle diagram of the heart segmentation network based on the multi-channel convolutional neural network in the present invention.
具体实施方式Detailed ways
为使本发明的目的、技术方案和优点更加清楚明了，下面结合具体实施方式并参照附图，对本发明进一步详细说明。应该理解，这些描述只是示例性的，而并非要限制本发明的范围。此外，在以下说明中，省略了对公知结构和技术的描述，以避免不必要地混淆本发明的概念。In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below in combination with specific embodiments and with reference to the accompanying drawings. It should be understood that these descriptions are exemplary only, and are not intended to limit the scope of the present invention. Also, in the following description, descriptions of well-known structures and techniques are omitted to avoid unnecessarily obscuring the concept of the present invention.
本发明中的心脏电影MRI图像是指：由心脏磁共振电影成像技术获取到的图像，是心脏MRI图像的一种。The cardiac cine MRI image in the present invention refers to an image obtained by cardiac magnetic resonance cine imaging technology, which is a kind of cardiac MRI image.
本发明中的ASPP模块是指：带扩张卷积的金字塔池化模块。The ASPP module in the present invention refers to a pyramid pooling module with dilated convolution.
心脏磁共振电影成像技术是一种常用的心脏磁共振成像技术，通常是指在心动周期内的各个特定阶段快速获取多幅图像，以电影的形式呈现显示。心脏磁共振电影成像技术不仅可用于评估心室功能、室壁异常，还可以用于评价心脏的瓣膜形态与功能。Cardiac magnetic resonance cine imaging technology is a commonly used cardiac magnetic resonance imaging technology, which usually refers to the rapid acquisition of multiple images at each specific stage of the cardiac cycle, and presents them in the form of a movie. Cardiac magnetic resonance cine imaging technology can not only be used to evaluate ventricular function and wall abnormalities, but also can be used to evaluate the shape and function of heart valves.
本发明的原图是指核磁共振原始扫描得到的图像，即收集到的心脏电影MRI图像。The original image in the present invention refers to the image obtained by the original nuclear magnetic resonance scan, that is, the collected heart cine MRI image.
针对目前现有技术存在的不足，本发明提出一种基于多路卷积神经网络的MRI图像心脏结构分割方法，如图1所示，本发明的方法包括：Aiming at the deficiencies in the current prior art, the present invention proposes a method for segmenting the heart structure of an MRI image based on a multi-channel convolutional neural network, as shown in Figure 1, the method of the present invention includes:
步骤1：收集心脏电影MRI训练数据，包括正常人的正常心脏电影MRI图像和心脏病人的异常心脏电影MRI图像。心脏电影MRI图像采集了心动周期中心脏舒张及收缩阶段的图像。扫描得到的心脏电影MRI数据是体数据，即由多个切片MRI数据构成。Step 1: Collect cardiac cine MRI training data, including normal cardiac cine MRI images of normal people and abnormal cardiac cine MRI images of heart patients. Cardiac cine MRI images capture images of the diastolic and systolic phases of the cardiac cycle. The scanned cardiac cine MRI data is volume data, that is, composed of multiple slices of MRI data.
同时采用正常人和心脏病人心脏数据是为了保证本发明技术方案的泛化能力，即本发明的自动分割方法既适用于正常心脏结构的的分割，同时也适用于异常心脏结构的分割。The purpose of using the heart data of normal people and heart patients at the same time is to ensure the generalization ability of the technical solution of the present invention, that is, the automatic segmentation method of the present invention is not only suitable for the segmentation of normal cardiac structures, but also suitable for the segmentation of abnormal cardiac structures.
在训练阶段采用了心脏舒张及收缩阶段的数据，一方面这部分数据是有标注的，更重要的是这两个阶段的心脏结构分割是通常在临床上比较关注的。In the training phase, the diastolic and systolic phase data are used. On the one hand, this part of the data is marked, and more importantly, the segmentation of cardiac structures in these two phases is usually more concerned in clinical practice.
步骤2：由经验丰富的医生手动对心脏电影MRI训练数据中的心脏结构进行逐层标注，并将标注结果作为心脏分割标准结果，待分割心脏结构包括正常心脏结构和异常心脏结构。标注的心脏结构主要包括左心室、右心室、心肌，在实际应用中，还可以对左心室内膜和外膜等进行标注。具体的，逐层是指自心底到心顶的MRI切片层，并且对所有的训练数据都进行标注。Step 2: An experienced doctor manually labels the cardiac structure in the cardiac movie MRI training data layer by layer, and uses the labeling result as the standard result of cardiac segmentation. The cardiac structure to be segmented includes normal cardiac structure and abnormal cardiac structure. The annotated cardiac structures mainly include the left ventricle, right ventricle, and myocardium. In practical applications, the endocardium and epicardium of the left ventricle can also be annotated. Specifically, layer by layer refers to the MRI slice layer from the bottom of the heart to the top of the heart, and all training data are labeled.
考虑到实际操作中收集到的心脏电影MRI图像数据都具有较大的扫描范围，覆盖了心脏周围很大的范围，因此心脏区域在图像上所占的显示比例相对较小，考虑到计算有效性并为了在一定程度上规避类别不平衡问题，在进行心脏结构分割之前，需要将心脏区域提取出来，使得提取出来的心脏区域在图像上占据较大的显示比例，这样也减低了后续分割处理的计算量。Considering that the heart cine MRI image data collected in actual operation has a large scanning range and covers a large area around the heart, so the display ratio of the heart area on the image is relatively small. Considering the calculation effectiveness And in order to avoid the problem of category imbalance to a certain extent, the heart region needs to be extracted before the heart structure segmentation, so that the extracted heart region occupies a larger display ratio on the image, which also reduces the subsequent segmentation processing. Calculations.
因此，本发明技术方案在分割之前设计并建立基于生成对抗网络的心脏区域提取模型，并针对心脏电影MRI训练数据训练该模型以提取心脏区域，具体如步骤3所示。Therefore, the technical solution of the present invention designs and establishes a heart region extraction model based on a generative adversarial network before segmentation, and trains the model to extract heart regions with respect to heart movie MRI training data, as specifically shown in step 3.
步骤3：心脏区域提取生成对抗网络训练，设计并建立基于生成对抗网络的心脏区域提取模型，从收集到的心脏电影MRI训练数据中提取出心脏图像，提取出的心脏区域是三维图像，由多个MRI切片图像叠加形成。Step 3: Heart region extraction Generative adversarial network training, design and establish a heart region extraction model based on a generative adversarial network, and extract heart images from the collected heart movie MRI training data. The extracted heart region is a three-dimensional image, which is composed of multiple The MRI slice images are superimposed to form.
生成对抗网络是一种深度学习模型，包括生成器和判别器两部分。生成对抗网络的工作原理如图2所示。心脏区域提取可以手工提取也可以基于一些方法进行自动提取。由于近年来生成对抗网络在图像处理领域取得了较好的性能，因此本发明选择将生成对抗网络应用到心脏区域提取模型中，以实现自动提取同时获得优良的提取性能。Generative confrontation network is a deep learning model, including generator and discriminator two parts. The working principle of Generative Adversarial Network is shown in Figure 2. Heart region extraction can be done manually or automatically based on some methods. Since the GAN has achieved better performance in the field of image processing in recent years, the present invention chooses to apply the GAN to the heart region extraction model to achieve automatic extraction and obtain excellent extraction performance.
生成器以训练数据中的心脏MRI切片图像作为输入，采用编码解码结构，即先通过卷积、池化操作下采样提取特征，再通过上采样、局部细节补偿以及卷积操作生成与输入的心脏MRI切片图像尺寸一致的伪心脏轮廓图像。在心脏区域提取模型训练阶段，生成器生成伪心脏轮廓图像，医生标记的图像为真实心脏区域轮廓图像。生成对抗网络训练的目标是生成器可以生成真实的心脏轮廓图像，即在训练完成后，理想的效果是生成器可以生成与医生标记图像一致的心脏轮廓图像。The generator takes the cardiac MRI slice image in the training data as input, adopts the encoding and decoding structure, that is, firstly extracts features by downsampling through convolution and pooling operations, and then generates and inputs the heart through upsampling, local detail compensation and convolution operations. Pseudo-heart contour images with consistent size of MRI slice images. In the heart region extraction model training stage, the generator generates a pseudo heart contour image, and the image marked by the doctor is the real heart region contour image. The goal of generating an adversarial network training is that the generator can generate a real heart contour image, that is, after the training is completed, the ideal effect is that the generator can generate a heart contour image that is consistent with the doctor's marked image.
现有技术中还没有采用基于生成对抗网络来提取心脏区域的方法，本发明技术方案主要利用生成对抗网络来实现无需人工交互以及无需人工设计特征的心脏区域自动提取。In the prior art, there is no method for extracting heart regions based on generative adversarial networks. The technical solution of the present invention mainly uses generative adversarial networks to realize automatic heart region extraction without manual interaction and without manual design features.
判别器以心脏MRI切片图像和相应的真实心脏区域轮廓图像对或心脏MRI切片图像和生成器生成的伪心脏轮廓图像对作为输入。通过卷积、池化操作提取特征，判别输入到判别器中的心脏区域轮廓图像是真实的还是由生成器生成的。具体的，判别器判别输入的真实心脏区域轮廓图像或伪心脏轮廓图像是真实的还是生成器生成的。The discriminator takes the cardiac MRI slice image and the corresponding real heart region contour image pair or the cardiac MRI slice image and the pseudo heart contour image pair generated by the generator as input. Features are extracted through convolution and pooling operations, and it is judged whether the contour image of the heart region input to the discriminator is real or generated by the generator. Specifically, the discriminator determines whether the input real heart region contour image or pseudo heart contour image is real or generated by the generator.
以使得生成器可生成逼近真实心脏轮廓图像的分割结果，同时以判别器可准确的区分出真实心脏轮廓图像以及生成器生成的伪心脏轮廓图像为优化目标，对心脏区域提取生成对抗网络进行训练，使得生成器最终可生成准确度高的心脏轮廓图像。网络训练好后，应用生成器生成准确的各个心脏MRI切片图像对应的心脏轮廓图像，定位出心脏在图像上的位置，从而将三维心脏区域提取出来。本发明将心脏区域提取问题看作了一个图像像素级二分类问题，即图像分割问题，参考生成对抗网络做图像分割的方法，本发明采用图像对作为判别器的输入。In order to enable the generator to generate segmentation results that are close to the real heart contour image, and at the same time, the discriminator can accurately distinguish the real heart contour image and the pseudo heart contour image generated by the generator as the optimization goal, and the heart region extraction generative confrontation network is trained. , so that the generator can finally generate a heart contour image with high accuracy. After the network is trained, the generator is applied to generate accurate cardiac contour images corresponding to each cardiac MRI slice image, and the position of the heart on the image is located, thereby extracting the three-dimensional cardiac region. The present invention regards the heart region extraction problem as an image pixel-level binary classification problem, that is, the image segmentation problem, referring to the image segmentation method of the generative confrontation network, and the present invention uses image pairs as the input of the discriminator.
一种优选的实施方式，可通过指定训练迭代次数并结合训练损失曲线来判断生成对抗网络是否已经训练完成。In a preferred implementation, it can be judged whether the generative adversarial network has been trained by specifying the number of training iterations combined with the training loss curve.
采用生成对抗网络是为了提取出心脏区域；为了这个目的，分两步，一、需要知道心脏在图像的什么位置上，二、将对应区域提取出来。对于第一步，心脏位置的确定是通过生成对抗网络来实现的，训练好生成对抗网络后，生成器可以生成准确的心脏轮廓图像，也就是知道了图像上哪个位置是心脏，即知道了心脏的位置。第二步，将确定的心脏区域提取出来。The use of generative confrontation network is to extract the heart area; for this purpose, there are two steps, one is to know where the heart is in the image, and the other is to extract the corresponding area. For the first step, the determination of the heart position is achieved by generating a confrontation network. After training the generation confrontation network, the generator can generate an accurate heart contour image, that is, knowing which position on the image is the heart, that is, knowing the heart s position. In the second step, the determined heart region is extracted.
生成对抗网络训练好后，再基于定位的心脏位置，提取出心脏区域，提取的方法包括：对于一个待提取心脏区域的心脏电影MRI图像数据，首先，利用已经训练好的生成器生成各个MRI切片图像对应的心脏轮廓图像，然后，根据轮廓生成包围心脏的矩形区域，取各切片中矩形区域最大的一个并在其基础上长宽各拓展0.3倍以保证可提取到完整的心脏区域，并将此拓展后的矩形区域作为待提取的区域。最后，对所有切片提取该区域，并叠加在一起，从而提取出三维心脏区域图像，以用于后续心脏分割网络的训练。After the generative confrontation network is trained, the heart region is extracted based on the location of the heart. The extraction method includes: for a heart movie MRI image data of the heart region to be extracted, first, use the trained generator to generate each MRI slice The heart contour image corresponding to the image, and then generate a rectangular area surrounding the heart according to the contour, take the largest rectangular area in each slice and expand it by 0.3 times in length and width to ensure that the complete heart area can be extracted, and The expanded rectangular area is used as the area to be extracted. Finally, the region is extracted for all slices and superimposed together to extract a three-dimensional image of the heart region for subsequent training of the heart segmentation network.
步骤4：基于多路卷积神经网络的心脏分割网络训练，设计并建立结合层间上下文信息的深度卷积分割网络。基于多路卷积神经网络进行心脏结构分割的工作原理图如图3所示。Step 4: Based on the multi-channel convolutional neural network heart segmentation network training, design and establish a deep convolutional segmentation network that combines inter-layer context information. The working principle of heart structure segmentation based on multi-channel convolutional neural network is shown in Figure 3.
步骤41：对步骤3提取出的心脏区域进行心脏结构的分割，以迭代的方式将相邻层的MRI切片图像信息以及相邻上层的已有分割结果信息作为层间上下文信息。Step 41: Carry out cardiac structure segmentation on the cardiac region extracted in step 3, and iteratively use the MRI slice image information of the adjacent layer and the existing segmentation result information of the adjacent upper layer as inter-layer context information.
一种优选的实施方案，采用相邻两层，即相邻的上一层和相邻的下一层的MRI切片图像信息以及相邻一个上层的已有分割结果信息作为层间上下文信息。A preferred embodiment uses the MRI slice image information of two adjacent layers, that is, the adjacent upper layer and the adjacent lower layer, and the existing segmentation result information of an adjacent upper layer as inter-layer context information.
步骤42：将这些层间上下文信息分别输入到各自的特征提取分支中，各个层间上下文特征提取分支采用独立的并行结构，即每个层间上下文特征提取分支采用相同的网络结构但是独立处理其对应的一种层间上下文信息。上下文特征提取分支通过卷积和池化操作来进行特征提取。Step 42: Input these inter-layer context information into their respective feature extraction branches, and each inter-layer context feature extraction branch adopts an independent parallel structure, that is, each inter-layer context feature extraction branch adopts the same network structure but independently processes other Corresponding inter-layer context information. The contextual feature extraction branch performs feature extraction through convolution and pooling operations.
具体的，层间上下文特征提取分支的个数根据要处理的层间上下文信息数目来确定。一种优选的实施方式，取相邻上下两层MRI切片图像和1个相邻上层的分割结果信息作为层间上下文信息，则层间上下文特征提取分支为3个。此外，再加上当前待分割的MRI切片自身的特征提取分支，共4个特征提取分支。图3即为本实施方式的4个特征提取分支的心脏结构分割工作原理图，如图3所示，input M[i+1]表示相邻上层MRI切片图像的分割结果，input S[i-1]和input S[i+1]表示相邻上下两层MRI切片图像，input S[i]表示当前待分割的MRI切片图像。Specifically, the number of inter-layer context feature extraction branches is determined according to the number of inter-layer context information to be processed. In a preferred implementation manner, the MRI slice images of two adjacent upper and lower layers and the segmentation result information of one adjacent upper layer are taken as the inter-layer context information, so there are three inter-layer context feature extraction branches. In addition, together with the feature extraction branch of the current MRI slice to be segmented, there are four feature extraction branches in total. Fig. 3 is the heart structure segmentation working principle diagram of 4 feature extraction branches of the present embodiment, as shown in Fig. 3, input M[i+1] represents the segmentation result of adjacent upper layer MRI slice image, input S[i- 1] and input S[i+1] represent the adjacent upper and lower MRI slice images, and input S[i] represents the current MRI slice image to be segmented.
步骤43：各分支通过叠加多个卷积及池化操作提取图像的高层抽象特征，并经特征融合模块进行融合。Step 43: Each branch extracts the high-level abstract features of the image by superimposing multiple convolution and pooling operations, and fuses them through the feature fusion module.
步骤44：特征融合模块先将各个分支提取到的高层抽象特征进行串连，然后通过设计的ASPP模块进一步融合这些高层特征，融合后的特征经由解码模块的上采样、局部细节信息补偿以及卷积操作将图像恢复到输入进心脏结构分割网络时的尺寸，得到端到端的密集多结构联合分割概率图，分割概率图是指MRI切片图像上每一像素点代表待分割图像对应像素点属于对应分类的概率，并由概率图确定各个像素所属类别从而得到最终的分割结果。Step 44: The feature fusion module first concatenates the high-level abstract features extracted by each branch, and then further fuses these high-level features through the designed ASPP module. The fused features are up-sampled, local detail information compensation and convolution by the decoding module The operation restores the image to the size when it is input into the heart structure segmentation network, and obtains an end-to-end dense multi-structure joint segmentation probability map. The segmentation probability map means that each pixel on the MRI slice image represents the corresponding pixel of the image to be segmented and belongs to the corresponding classification The probability of each pixel is determined by the probability map to obtain the final segmentation result.
步骤5：将心脏结构分割网络得到的分割结果与步骤2得到的心脏标准分割结果进行比较，通过性能评价指标进行分割结果量化评估。本发明采用Dice系数和/或ASSD指标对分割结果进行量化评估。同时，通过性能评价指标来评估分割结果的好坏。Step 5: Compare the segmentation results obtained by the heart structure segmentation network with the heart standard segmentation results obtained in step 2, and perform quantitative evaluation of the segmentation results through performance evaluation indicators. The present invention uses the Dice coefficient and/or the ASSSD index to quantitatively evaluate the segmentation result. At the same time, the performance evaluation index is used to evaluate the quality of the segmentation results.
步骤6：收集待分割心脏电影MRI数据，通过步骤3训练好的心脏区域自动提取网络提取心脏区域，并记录提取位置信息。提取出来的心脏区域为三维体数据，是由多个MRI切片图像叠加构成的。然后将提取的心脏区域的MRI切片图像输入到步骤4训练好的深度卷积分割网络中，自心底到心顶依次迭代完成心脏区域体数据的分割，剔除分割结果中各结构可能存在的不连续的零散区域，得到心脏结构初始分割结果。Step 6: Collect the heart movie MRI data to be segmented, extract the heart region through the heart region automatic extraction network trained in step 3, and record the extracted location information. The extracted cardiac region is three-dimensional volume data, which is composed of multiple MRI slice images superimposed. Then input the extracted MRI slice image of the heart region into the deep convolutional segmentation network trained in step 4, iteratively complete the segmentation of the volume data of the heart region from the bottom of the heart to the top of the heart, and eliminate the possible discontinuity of each structure in the segmentation result Scattered regions of the heart structure are obtained to obtain the initial segmentation results of the heart structure.
步骤6是为了应用本发明提出的方法进行心脏结构的分割。Step 6 is to apply the method proposed by the present invention to segment the heart structure.
步骤7：根据提取位置信息，将心脏结构初始分割结果恢复到原图尺寸，得到最终的分割结果。在实际应用中，不恢复到原图的情况下，也可以看到分割结果，只是尺寸与原图不一致，但并不影响看分割结果。Step 7: According to the extracted position information, the initial segmentation result of the heart structure is restored to the original image size, and the final segmentation result is obtained. In practical applications, you can still see the segmentation results without restoring the original image, but the size is inconsistent with the original image, but it does not affect the viewing of the segmentation results.
例如,原图是400*400大小，其中提取出来的心脏是20*20的大小且位于原图的中心位置，这里默认了20*20外的区域都是背景，即默认分割为非心脏结构；心脏结构初始分割得到的是对这20*20的图像的分割结果，而最终要的应该是400*400图像对应的分割结果。因此，在分割时，需要记录心脏提取位置，从而才知道将心脏分割结果对应回原图尺寸的哪个位置。比如中间、左下角、右下角。For example, the original image is 400*400 in size, and the extracted heart is 20*20 in size and located in the center of the original image. Here, by default, the area outside 20*20 is the background, that is, it is divided into non-heart structures by default; The initial segmentation of the heart structure is the segmentation result of the 20*20 image, and the final result should be the segmentation result corresponding to the 400*400 image. Therefore, when segmenting, it is necessary to record the extraction position of the heart, so as to know which position the heart segmentation result corresponds to in the original image size. For example, middle, bottom left, bottom right.
需要注意的是，上述具体实施是示例性的，本领域技术人员可以在本发明公开内容的启发下想出各种解决方案，而这些解决方案也都属于本发明的公开范围并落入本发明的保护范围之内。本领域技术人员应该明白，本发明说明书及其附图均为说明性而并非构成对权利要求的限制。本发明的保护范围由权利要求及其等同物限定。It should be noted that the above specific implementation is exemplary, and those skilled in the art can come up with various solutions inspired by the disclosure of the present invention, and these solutions also belong to the disclosure scope of the present invention and fall into the present invention within the scope of protection. Those skilled in the art should understand that the description and drawings of the present invention are illustrative rather than limiting to the claims. The protection scope of the present invention is defined by the claims and their equivalents.
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910780248.7ACN110599499B (en) | 2019-08-22 | 2019-08-22 | MRI image heart structure segmentation method based on multipath convolutional neural network |
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910780248.7ACN110599499B (en) | 2019-08-22 | 2019-08-22 | MRI image heart structure segmentation method based on multipath convolutional neural network |
| Publication Number | Publication Date |
|---|---|
| CN110599499Atrue CN110599499A (en) | 2019-12-20 |
| CN110599499B CN110599499B (en) | 2022-04-19 |
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910780248.7AExpired - Fee RelatedCN110599499B (en) | 2019-08-22 | 2019-08-22 | MRI image heart structure segmentation method based on multipath convolutional neural network |
| Country | Link |
|---|---|
| CN (1) | CN110599499B (en) |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111598838A (en)* | 2020-04-22 | 2020-08-28 | 中南民族大学 | Automatic heart MR image segmentation method and device, electronic equipment and storage medium |
| CN112116625A (en)* | 2020-08-25 | 2020-12-22 | 澳门科技大学 | Automatic heart CT image segmentation method, device and medium based on contradiction marking method |
| CN112561921A (en)* | 2020-11-10 | 2021-03-26 | 联想(北京)有限公司 | Image segmentation method and device |
| CN112949470A (en)* | 2021-02-26 | 2021-06-11 | 上海商汤智能科技有限公司 | Method, device and equipment for identifying lane-changing steering lamp of vehicle and storage medium |
| CN113222996A (en)* | 2021-03-03 | 2021-08-06 | 中南民族大学 | Heart segmentation quality evaluation method, device, equipment and storage medium |
| CN113362345A (en)* | 2021-06-30 | 2021-09-07 | 武汉中科医疗科技工业技术研究院有限公司 | Image segmentation method and device, computer equipment and storage medium |
| CN113781343A (en)* | 2021-09-13 | 2021-12-10 | 叠境数字科技（上海）有限公司 | Super-resolution image quality improvement method |
| CN114066863A (en)* | 2021-11-22 | 2022-02-18 | 沈阳东软智能医疗科技研究院有限公司 | Method and device for training ventricle region segmentation model and determining template image |
| WO2022050902A1 (en) | 2020-09-02 | 2022-03-10 | Singapore Health Services Pte Ltd | Image segmentation system and method |
| CN116228802A (en)* | 2023-05-05 | 2023-06-06 | 济南科汛智能科技有限公司 | Cardiac MRI auxiliary imaging control method |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106096632A (en)* | 2016-06-02 | 2016-11-09 | 哈尔滨工业大学 | Based on degree of depth study and the ventricular function index prediction method of MRI image |
| CN106600621A (en)* | 2016-12-08 | 2017-04-26 | 温州医科大学 | Space-time cooperation segmentation method based on infant brain tumor multi-modal MRI graph |
| CN106683104A (en)* | 2017-01-06 | 2017-05-17 | 西北工业大学 | Prostate magnetic resonance image segmentation method based on integrated depth convolution neural network |
| US20180218502A1 (en)* | 2017-01-27 | 2018-08-02 | Arterys Inc. | Automated segmentation utilizing fully convolutional networks |
| CN109377520A (en)* | 2018-08-27 | 2019-02-22 | 西安电子科技大学 | Cardiac image registration system and method based on semi-supervised recurrent GAN |
| CN109584254A (en)* | 2019-01-07 | 2019-04-05 | 浙江大学 | A kind of heart left ventricle's dividing method based on the full convolutional neural networks of deep layer |
| US20190223725A1 (en)* | 2018-01-25 | 2019-07-25 | Siemens Healthcare Gmbh | Machine Learning-based Segmentation for Cardiac Medical Imaging |
| CN110120051A (en)* | 2019-05-10 | 2019-08-13 | 上海理工大学 | A kind of right ventricle automatic division method based on deep learning |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106096632A (en)* | 2016-06-02 | 2016-11-09 | 哈尔滨工业大学 | Based on degree of depth study and the ventricular function index prediction method of MRI image |
| CN106600621A (en)* | 2016-12-08 | 2017-04-26 | 温州医科大学 | Space-time cooperation segmentation method based on infant brain tumor multi-modal MRI graph |
| CN106683104A (en)* | 2017-01-06 | 2017-05-17 | 西北工业大学 | Prostate magnetic resonance image segmentation method based on integrated depth convolution neural network |
| US20180218502A1 (en)* | 2017-01-27 | 2018-08-02 | Arterys Inc. | Automated segmentation utilizing fully convolutional networks |
| US20190223725A1 (en)* | 2018-01-25 | 2019-07-25 | Siemens Healthcare Gmbh | Machine Learning-based Segmentation for Cardiac Medical Imaging |
| CN109377520A (en)* | 2018-08-27 | 2019-02-22 | 西安电子科技大学 | Cardiac image registration system and method based on semi-supervised recurrent GAN |
| CN109584254A (en)* | 2019-01-07 | 2019-04-05 | 浙江大学 | A kind of heart left ventricle's dividing method based on the full convolutional neural networks of deep layer |
| CN110120051A (en)* | 2019-05-10 | 2019-08-13 | 上海理工大学 | A kind of right ventricle automatic division method based on deep learning |
| Title |
|---|
| GONGNING LUO ET AL.: "A deep learning network for right ventricle segmentation in short-axis MRI", 《2016 COMPUTING IN CARDIOLOGY CONFERENCE》* |
| 汪天富 等: "基于自组织神经网络的超声心脏图象分割", 《中国生物医学工程学报》* |
| 陈军: "基于深度学习的心脏图像分割方法的研究", 《中国优秀硕士学位论文全文数据库 (信息科技辑)》* |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111598838A (en)* | 2020-04-22 | 2020-08-28 | 中南民族大学 | Automatic heart MR image segmentation method and device, electronic equipment and storage medium |
| CN111598838B (en)* | 2020-04-22 | 2023-04-07 | 中南民族大学 | Automatic heart MR image segmentation method and device, electronic equipment and storage medium |
| CN112116625A (en)* | 2020-08-25 | 2020-12-22 | 澳门科技大学 | Automatic heart CT image segmentation method, device and medium based on contradiction marking method |
| WO2022050902A1 (en) | 2020-09-02 | 2022-03-10 | Singapore Health Services Pte Ltd | Image segmentation system and method |
| EP4208851A4 (en)* | 2020-09-02 | 2024-10-16 | Singapore Health Services Pte Ltd | IMAGE SEGMENTATION SYSTEM AND METHOD |
| CN112561921A (en)* | 2020-11-10 | 2021-03-26 | 联想(北京)有限公司 | Image segmentation method and device |
| CN112561921B (en)* | 2020-11-10 | 2024-07-26 | 联想(北京)有限公司 | Image segmentation method and device |
| CN112949470A (en)* | 2021-02-26 | 2021-06-11 | 上海商汤智能科技有限公司 | Method, device and equipment for identifying lane-changing steering lamp of vehicle and storage medium |
| CN113222996A (en)* | 2021-03-03 | 2021-08-06 | 中南民族大学 | Heart segmentation quality evaluation method, device, equipment and storage medium |
| CN113362345A (en)* | 2021-06-30 | 2021-09-07 | 武汉中科医疗科技工业技术研究院有限公司 | Image segmentation method and device, computer equipment and storage medium |
| CN113781343A (en)* | 2021-09-13 | 2021-12-10 | 叠境数字科技（上海）有限公司 | Super-resolution image quality improvement method |
| CN114066863A (en)* | 2021-11-22 | 2022-02-18 | 沈阳东软智能医疗科技研究院有限公司 | Method and device for training ventricle region segmentation model and determining template image |
| CN114066863B (en)* | 2021-11-22 | 2024-09-13 | 沈阳东软智能医疗科技研究院有限公司 | Method and device for training ventricle area segmentation model and determining template image |
| CN116228802A (en)* | 2023-05-05 | 2023-06-06 | 济南科汛智能科技有限公司 | Cardiac MRI auxiliary imaging control method |
| CN116228802B (en)* | 2023-05-05 | 2023-07-04 | 济南科汛智能科技有限公司 | Cardiac MRI auxiliary imaging control method |
| Publication number | Publication date |
|---|---|
| CN110599499B (en) | 2022-04-19 |
| Publication | Publication Date | Title |
|---|---|---|
| CN110599499B (en) | MRI image heart structure segmentation method based on multipath convolutional neural network | |
| CN110310281B (en) | Mask-RCNN deep learning-based pulmonary nodule detection and segmentation method in virtual medical treatment | |
| Liu et al. | Deep learning-based automated left ventricular ejection fraction assessment using 2-D echocardiography | |
| CN110475505B (en) | Automatic segmentation using full convolution network | |
| JP6993334B2 (en) | Automated cardiac volume segmentation | |
| CN111429474B (en) | Mammary gland DCE-MRI image focus segmentation model establishment and segmentation method based on mixed convolution | |
| CN114037714B (en) | 3D MR and TRUS image segmentation method for prostate system puncture | |
| CN102411795B (en) | Combined display method of multimode image of coronary artery | |
| CN109192305B (en) | An automatic analysis method of cardiac function based on deep recurrent neural network | |
| CN109003280B (en) | A method for segmentation of vascular media and intima from dual-channel intravascular ultrasound images | |
| CN116129235B (en) | A method for cross-modal synthesis of medical images from CT to MRI routine sequence of cerebral infarction | |
| CN114663445B (en) | A 3D cardiac image segmentation method based on multi-scale edge perception | |
| Yan et al. | Cine MRI analysis by deep learning of optical flow: Adding the temporal dimension | |
| CN110363772A (en) | Cardiac MRI Segmentation Method and System Based on Adversarial Network | |
| CN112998756A (en) | Heart blood flow vector imaging method based on ultrasonic image and deep learning | |
| CN113689441A (en) | A Dynamic Segmentation Method of Left Ventricular Ultrasound Based on DeepLabV3 Network | |
| CN118037685A (en) | A method and device for positioning ultrasonic organ structure points based on deep learning | |
| CN116152430A (en) | 3D (three-dimensional) echocardiography left and right ventricle three-dimensional modeling method based on edge reinforcement and self-attention mechanism | |
| CN118037615A (en) | A tumor segmentation-guided magnetic resonance image translation method, system, device and medium | |
| CN115471512A (en) | Medical image segmentation method based on self-supervision contrast learning | |
| Chen et al. | Sam-octa2: Layer sequence octa segmentation with fine-tuned segment anything model 2 | |
| CN112651976B (en) | Method and system for determination of brain network in lesion area based on low-resolution MRI data | |
| CN111242928A (en) | Automatic segmentation, tracking and localization method of atrium based on multi-view learning | |
| CN115439419A (en) | A relative motion learning method and device for identifying myocardial infarction | |
| Lyu et al. | Personalized 3D Myocardial Infarct Geometry Reconstruction from Cine MRI with Explicit Cardiac Motion Modeling |
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee | Granted publication date:20220419 | |
| CF01 | Termination of patent right due to non-payment of annual fee |