技术领域technical field
本发明涉及医疗图像识别领域，具体涉及一种病理切片图像识别方法及设备。The invention relates to the field of medical image recognition, in particular to a pathological slice image recognition method and device.
背景技术Background technique
病理诊断是在观测器官的大体改变、镜下观察组织结构和细胞病变特征而做出的疾病诊断，因此它比临床上根据病史、症状和体征等做出的分析性诊断以及利用各种影像(如超声波、X射线、CT、核磁共振等)所做出的诊断更具有客观性和准确性。Pathological diagnosis is a disease diagnosis made by observing the general changes of organs, observing the tissue structure and the characteristics of cell lesions under a microscope, so it is better than the analytical diagnosis made clinically based on medical history, symptoms and signs, and the use of various images ( Such as ultrasound, X-ray, CT, nuclear magnetic resonance, etc.) The diagnosis made is more objective and accurate.
病理切片本身是通过组织取样、浸蜡、切片和染色等步骤形成的玻璃切片。目前病理切片已实现数字化保存和浏览，通过扫描设备将对玻璃切片进行扫描，形成高清的切片图像。应用者可随时随地对显微切片任何区域进行不同放大倍率的浏览(例如2x,4x,10x,20x,40x,100x)。The pathological section itself is a glass section formed through the steps of tissue sampling, wax immersion, sectioning and staining. At present, pathological slices have been digitally stored and browsed, and glass slices are scanned by scanning equipment to form high-definition slice images. Users can view any area of the microsection at different magnifications (such as 2x, 4x, 10x, 20x, 40x, 100x) anytime, anywhere.
数字化的病理切片原图尺寸较大，通常有几万像素的长宽尺寸。对于机器学习模型而言，对如此大尺寸的图像进行识别会比较困难。目前采用的技术是从病理切片原图中截取一部分图像作为识别目标，利用机器学习模型(深度学习模型)对截取的图像进行分类或分割，确定其是否存在异常内容或标记出异常内容(如肿瘤区域)。但仅凭单个区域而不参考其周围区域进行分类或分割，很难得到准确的识别结果，因此容易会出现假阳性的情况。The original image of digital pathological slides is large in size, usually tens of thousands of pixels in length and width. Recognition of such large-scale images can be difficult for machine learning models. The technology currently used is to intercept a part of the image from the original image of the pathological section as the recognition target, and use a machine learning model (deep learning model) to classify or segment the intercepted image to determine whether there is any abnormal content or to mark the abnormal content (such as tumor area). However, it is difficult to obtain accurate recognition results only by classifying or segmenting a single region without referring to its surrounding regions, so false positives are prone to occur.
发明内容SUMMARY OF THE INVENTION
有鉴于此，本发明提供一种病理切片图像识别方法，包括：In view of this, the present invention provides a method for pathological slice image recognition, comprising:
将病理切片图像划分为多个图像块；dividing the pathological slice image into multiple image blocks;
利用第一神经网络分别从所述多个图像块中提取第一特征图；Using the first neural network to extract first feature maps from the plurality of image blocks respectively;
根据各个所述图像块在所述病理切片图像中的位置拼接各个所述第一特征图形成拼接特征图；stitching each of the first feature maps according to the position of each of the image blocks in the pathological slice image to form a stitching feature map;
分别利用多个第二神经网络从所述拼接特征图中提取第二特征图；Using a plurality of second neural networks to extract a second feature map from the spliced feature map;
将所述第二特征图进行叠加形成特征图集；superimposing the second feature map to form a feature map set;
对所述特征图集进行分类得到分类结果，所述分类结果用于表示所述多个图像块分别是否正常。A classification result is obtained by classifying the feature atlas, and the classification result is used to indicate whether each of the plurality of image blocks is normal.
可选地，在将病理切片图像划分为多个图像块的步骤中采用等分方式得到9个尺寸相等的图像块；Optionally, in the step of dividing the pathological slice image into a plurality of image blocks, an equal division method is used to obtain 9 image blocks of equal size;
在所述利用第一神经网络分别从所述多个图像块中提取第一特征图的步骤中，所述9个尺寸相等的图像块作为所述第一神经网络的输入数据，所述第一神经网络输出9个第一特征图，与所述9个尺寸相等的图像块一一对应；In the step of using the first neural network to extract the first feature map from the plurality of image blocks, the 9 image blocks of equal size are used as the input data of the first neural network, and the first The neural network outputs 9 first feature maps, which are in one-to-one correspondence with the 9 equal-sized image blocks;
在所述分别利用多个第二神经网络从所述拼接特征图中提取第二特征图的步骤中，所述拼接特征图作为9个第二神经网络的输入数据，所述9个第二神经网络分别输出1个第二特征图。In the step of using a plurality of second neural networks to extract a second feature map from the spliced feature map, the spliced feature map is used as input data for 9 second neural networks, and the 9 second neural networks The network outputs a second feature map respectively.
可选地，所述第一特征图是未经全局平均池化处理的特征图。Optionally, the first feature map is a feature map that has not been processed by global average pooling.
可选地，所述第二特征图是未经全局平均池化处理的特征图。Optionally, the second feature map is a feature map that has not been processed by global average pooling.
可选地，对所述特征图集进行分类得到分类结果，包括：Optionally, classify the feature atlas to obtain classification results, including:
对所述特征图集进行全局平均池化处理；performing a global average pooling process on the feature atlas;
利用分类器对全局平均池化处理结果进行分类得到所述分类结果。The classifier is used to classify the global average pooling processing result to obtain the classification result.
本发明还提供一种病理切片图像识别装置，包括：The present invention also provides a pathological slice image recognition device, comprising:
划分模块，用于将病理切片图像划分为多个图像块；A division module, for dividing the pathological slice image into a plurality of image blocks;
第一神经网络，用于分别从所述多个图像块中提取第一特征图；The first neural network is used to extract first feature maps from the plurality of image blocks respectively;
拼接模块，用于根据各个所述图像块在所述病理切片图像中的位置拼接各个所述第一特征图形成拼接特征图；A stitching module, configured to stitch each of the first feature maps according to the position of each of the image blocks in the pathological slice image to form a stitching feature map;
多个第二神经网络，分别用于从所述拼接特征图中提取第二特征图；A plurality of second neural networks are respectively used to extract a second feature map from the spliced feature map;
组合模块，用于将所述第二特征图进行叠加形成特征图集；A combination module, configured to superimpose the second feature maps to form a feature map set;
分类模块，用于对所述特征图集进行分类得到分类结果，所述分类结果用于表示所述多个图像块分别是否正常。A classification module, configured to classify the feature atlas to obtain a classification result, where the classification result is used to indicate whether each of the plurality of image blocks is normal.
可选地，所述划分模块采用等分方式得到9个尺寸相等的图像块；Optionally, the division module obtains 9 image blocks of equal size by means of equal division;
所述9个尺寸相等的图像块作为所述第一神经网络的输入数据，所述第一神经网络输出9个第一特征图，与所述9个尺寸相等的图像块一一对应；The 9 image blocks of equal size are used as the input data of the first neural network, and the first neural network outputs 9 first feature maps, corresponding to the 9 image blocks of equal size;
所述第二神经网络的数量为9个，所述拼接特征图作为9个第二神经网络的输入数据，所述9个第二神经网络分别输出1个第二特征图。The number of the second neural networks is 9, the spliced feature maps are used as input data of the 9 second neural networks, and the 9 second neural networks output 1 second feature map respectively.
可选地，所述第一特征图是未经全局平均池化处理的特征图。Optionally, the first feature map is a feature map that has not been processed by global average pooling.
可选地，所述分类模块包括：Optionally, the classification module includes:
全局平均池化模块，用于对所述特征图集进行全局平均池化处理；A global average pooling module, configured to perform global average pooling processing on the feature atlas;
分类器，用于对全局平均池化处理结果进行分类得到所述分类结果。A classifier, configured to classify the global average pooling processing result to obtain the classification result.
相应地，本发明还提供一种病理切片图像识别设备，包括：至少一个处理器；以及与所述至少一个处理器通信连接的存储器；其中，所述存储器存储有可被所述一个处理器执行的指令，所述指令被所述至少一个处理器执行，以使所述至少一个处理器执行上述病理切片图像识别方法。Correspondingly, the present invention also provides a pathological slice image recognition device, including: at least one processor; and a memory connected to the at least one processor in communication; wherein, the memory stores information that can be executed by the one processor. instructions, the instructions are executed by the at least one processor, so that the at least one processor executes the above pathological slice image recognition method.
根据本发明提供的病理切片图像识别方法及装置，首先通过一个神经网络分别从输入的多个病理切片图像块中提取特征图，进而根据各个图像块在病理切片图像中的位置拼接这些特征图，使得到的拼接特征图体现空间信息；然后再使用多个神经网络并行对该拼接特征图进行识别，其中各个神经网络可分别充分利用图像块的空间信息，进而提取多个特征图，最后对多个特征图进行组合，进而根据组合的特征图集得到分类结果。本发明能够使神经网络非常有效的进行空间信息的利用，在给一张病理切片图像进行分类时，会参考多种不同的空间位置信息，得出更准确的结果。According to the pathological slice image recognition method and device provided by the present invention, firstly, a neural network is used to extract feature maps from multiple input pathological slice image blocks, and then these feature maps are spliced according to the positions of each image block in the pathological slice image, The stitched feature map obtained reflects spatial information; then multiple neural networks are used to identify the stitched feature map in parallel, and each neural network can make full use of the spatial information of the image block, and then extract multiple feature maps. The feature maps are combined, and then the classification result is obtained according to the combined feature map set. The invention enables the neural network to utilize spatial information very effectively, and when classifying a pathological slice image, it will refer to various spatial position information to obtain more accurate results.
附图说明Description of drawings
为了更清楚地说明本发明具体实施方式或现有技术中的技术方案，下面将对具体实施方式或现有技术描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图是本发明的一些实施方式，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。In order to more clearly illustrate the specific implementation of the present invention or the technical solutions in the prior art, the following will briefly introduce the accompanying drawings that need to be used in the specific implementation or description of the prior art. Obviously, the accompanying drawings in the following description The drawings show some implementations of the present invention, and those skilled in the art can obtain other drawings based on these drawings without any creative work.
图1为本发明实施例中的模型结构示意图；Fig. 1 is the model structure schematic diagram in the embodiment of the present invention;
图2为本发明实施例中的病理切片图像识别方法的流程图；Fig. 2 is the flow chart of the pathological slice image recognition method in the embodiment of the present invention;
图3为本发明实施例中的病理切片图像识别装置的结构示意图。Fig. 3 is a schematic structural diagram of a pathological slice image recognition device in an embodiment of the present invention.
具体实施方式Detailed ways
下面将结合附图对本发明的技术方案进行清楚、完整地描述，显然，所描述的实施例是本发明一部分实施例，而不是全部的实施例。基于本发明中的实施例，本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其他实施例，都属于本发明保护的范围。The technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Apparently, the described embodiments are some of the embodiments of the present invention, but not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.
在本发明的描述中，需要说明的是，术语“第一”、“第二”、仅用于描述目的，而不能理解为指示或暗示相对重要性。此外，下面所描述的本发明不同实施方式中所涉及的技术特征只要彼此之间未构成冲突就可以相互结合。In the description of the present invention, it should be noted that the terms "first" and "second" are used for description purposes only, and should not be understood as indicating or implying relative importance. In addition, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.
本发明实施例提供一种模型结构及其训练方法，目的在于获得一个能够对病理切片图像进行分类的模型。如图1所示，该模型包括第一神经网络11、多个第二神经网络12和分类网络13。第一神经网络11和第二神经网络12例如可以是resnet18或类似的卷积神经网络，第二神经网络12的数量与输入的病理切片图像块数量一致，在本实施例中是9个。分类网络13可包括全局平均池化层(全局平均池化，Globel-Average-Pooling，GAP)和分类器。Embodiments of the present invention provide a model structure and a training method thereof, aiming at obtaining a model capable of classifying pathological slice images. As shown in FIG. 1 , the model includes a first neural network 11 , multiple second neural networks 12 and a classification network 13 . The first neural network 11 and the second neural network 12 can be, for example, a resnet 18 or a similar convolutional neural network, and the number of the second neural network 12 is consistent with the number of input pathological slice image blocks, which is 9 in this embodiment. The classification network 13 may include a global average pooling layer (Global-Average-Pooling, GAP) and a classifier.
一个用于训练模型的样本数据中包括多个病理切片图像块，它们是对一个病理切片图像进行划分而得到的。病理切片图像本身取自病理切片原图，原图的尺寸可以是几万像素长宽。在本实施例中，所述病理切片图像的尺寸为768x768像素，是原图中的一部分。将此图像划分为9个相同尺寸的图像块，得到9个256x256像素的图像块。这种划分方式只作为一种优选的实施方式，并非唯一的划分方式，例如也可以划分为4个、6个图像块。A sample data for training a model includes multiple pathological slice image blocks, which are obtained by dividing a pathological slice image. The pathological slice image itself is taken from the original image of the pathological slice, and the size of the original image can be tens of thousands of pixels in length and width. In this embodiment, the size of the pathological slice image is 768x768 pixels, which is a part of the original image. Divide this image into 9 image blocks of the same size, resulting in 9 image blocks of 256x256 pixels. This division method is only used as a preferred implementation manner, and is not the only division method. For example, it can also be divided into 4 or 6 image blocks.
该样本数据中还包括这9个图像块在病理切片图像中的位置信息，以及这9个图像块分别是否包含异常内容的标签信息，异常内容例如是肿瘤病变组织。位置信息是指这9个图像块彼此间的方位，例如以中央的图像块为基准，其它8个图像块相对于中央图像块的位置即左上、正上、右上……右下，具体可以使用数组来表达该位置信息。The sample data also includes position information of the nine image blocks in the pathological slice image, and label information of whether the nine image blocks contain abnormal content, such as tumor lesion tissue. The location information refers to the orientation of these 9 image blocks. For example, based on the central image block, the positions of the other 8 image blocks relative to the central image block are upper left, upper right, upper right...lower right. Specifically, you can use An array to express the location information.
利用上述样本数据对图1所示结构的初始模型进行训练，将9个图像块作为第一神经网络11的输入数据，而输出9个第一特征图(feature map)，与输入的9个图像块一一对应。在一个优选的实施例中，第一神经网络11不需要进行GAP处理，也即第一特征图是GAP之前的特征图。Utilize above-mentioned sample data to train the initial model of structure shown in Fig. 1, use 9 image blocks as the input data of the first neural network 11, and output 9 first feature map (feature map), and input 9 images One-to-one correspondence of blocks. In a preferred embodiment, the first neural network 11 does not need to perform GAP processing, that is, the first feature map is a feature map before GAP.
按照上述位置信息，将这9个第一特征图拼接为1个完整的特征图，得到拼接特征图A，其尺寸取决于第一神经网络11的输出尺寸L(output-size＝L)。第一神经网络11输出了9个第一特征图，则总尺寸是L x L x 9(个)，拼接特征图A的尺寸则是3L x 3L x 1(个)。According to the above position information, the nine first feature maps are spliced into one complete feature map to obtain a spliced feature map A, whose size depends on the output size L (output-size=L) of the first neural network 11 . The first neural network 11 outputs 9 first feature maps, the total size is L x L x 9 (pieces), and the size of the spliced feature map A is 3L x 3L x 1 (pieces).
将拼接特征图A分别输入9个第二神经网络12，每个第二神经网络12都输出一个第二特征图(feature map)，由此得到9个第二特征图。在一个优选的实施例中，第二神经网络12不需要进行GAP处理，也即第二特征图是GAP之前的特征图。The spliced feature map A is respectively input into nine second neural networks 12, and each second neural network 12 outputs a second feature map (feature map), thereby obtaining nine second feature maps. In a preferred embodiment, the second neural network 12 does not need to perform GAP processing, that is, the second feature map is a feature map before GAP.
组合这9个第二特征图得到特征图集B。所述组合与上述拼接是不同的概念，在组合处理中不存在位置信息，而是将这9个第二特征图作为9个通道。如果每个第二神经网络12的输出尺寸为L，则每个第二特征图的尺寸是L x L x 1(个)，而特征图集B的尺寸是L xL x 9。Combine these 9 second feature maps to get feature map set B. The combination is a different concept from the above splicing. There is no position information in the combination process, but the nine second feature maps are used as nine channels. If the output size of each second neural network 12 is L, then the size of each second feature map is L x L x 1 (piece), and the size of feature map set B is L x L x 9.
最后由分类网络13输出最终的分类结果，该结果是一个向量，其中包括9个数值，分别用于表达所输入的9个图像块中是否包含异常内容。具体地，首先对特征图集B进行GAP处理，然后利用分类器(如softmax)对GAP处理后的特征图集B进行分类得到结果。Finally, the classification network 13 outputs the final classification result, which is a vector, which includes 9 values, respectively used to express whether the 9 input image blocks contain abnormal content. Specifically, first perform GAP processing on the feature atlas B, and then use a classifier (such as softmax) to classify the GAP-processed feature atlas B to obtain the result.
利用大量的样本数据重复上述训练过程，使模型根据分类结果与标签信息的比对结果，从而修正第一神经网络11和各个第二神经网络12的参数。当模型性能达到预期时，虽然各个第二神经网络12的结构是相同的，但实际上各个第二神经网络12的参数会有所不同，因为在修正参数过程中它们会逐渐变成对它负责的图像块有利的参数，从而提高分类结果的准确性。A large amount of sample data is used to repeat the above training process, so that the model can modify the parameters of the first neural network 11 and each second neural network 12 according to the comparison result of the classification result and the label information. When the model performance reaches expectations, although the structure of each second neural network 12 is the same, in fact the parameters of each second neural network 12 will be different, because they will gradually become responsible for it in the process of modifying parameters. Favorable parameters of the image block, thereby improving the accuracy of the classification results.
本发明实施例提供一种病理切片图像识别方法，该方法可以由计算机或服务器等电子设备执行，利用深度学习模型对病理切片图像进行识别。关于深度学习模型的结构和训练方式可参照上述实施例提供的技术方案。如图2所示该方法包括如下步骤：An embodiment of the present invention provides a pathological slice image recognition method, which can be executed by an electronic device such as a computer or a server, and uses a deep learning model to recognize the pathological slice image. Regarding the structure and training methods of the deep learning model, reference may be made to the technical solutions provided in the foregoing embodiments. As shown in Figure 2, the method includes the following steps:
S1，将病理切片图像划分为多个图像块。作为一个举例，病理切片图像的尺寸是768x768像素。将此图像划分为9个相同尺寸的图像块，得到9个256x256像素的图像块。这种划分方式只作为一种优选的实施方式，并非唯一的划分方式，例如也可以划分为4个、6个图像块。识别过程的划分方式应当与模型训练数据中的划分方式相同。S1, dividing the pathological slice image into multiple image blocks. As an example, the size of the pathological slice image is 768x768 pixels. Divide this image into 9 image blocks of the same size, resulting in 9 image blocks of 256x256 pixels. This division method is only used as a preferred implementation manner, and is not the only division method. For example, it can also be divided into 4 or 6 image blocks. The recognition process should be partitioned in the same way as in the model training data.
S2，利用第一神经网络分别从多个图像块中提取第一特征图。9个图像块作为第一神经网络的输入数据，而输出9个第一特征图(feature map)，与输入的9个图像块一一对应。在一个优选的实施例中，第一神经网络11不需要进行GAP处理，也即第一特征图是GAP之前的特征图。S2. Using the first neural network to respectively extract first feature maps from multiple image blocks. The 9 image blocks are used as the input data of the first neural network, and 9 first feature maps (feature maps) are output, corresponding to the 9 input image blocks one by one. In a preferred embodiment, the first neural network 11 does not need to perform GAP processing, that is, the first feature map is a feature map before GAP.
S3，根据各个图像块在病理切片图像中的位置拼接各个第一特征图形成拼接特征图。其中的位置可以在步骤S1中被确定，第一特征图的尺寸取决于第一神经网络的输出尺寸，若各个第一特征图的尺寸分别为L x L，则拼接特征图的尺寸为3L x 3L。S3, stitching each first feature map according to the position of each image block in the pathological slice image to form a stitching feature map. The position in which can be determined in step S1, the size of the first feature map depends on the output size of the first neural network, if the size of each first feature map is L x L respectively, then the size of the spliced feature map is 3L x 3L.
S4，分别利用多个第二神经网络从拼接特征图中提取第二特征图。第二神经网络的数量与输入的病理切片图像块数量一致，在本实施例中是9个。9个第二神经网络分别并行对唯一的拼接特征图提取特征，可得到9个第二特征图。S4. Using multiple second neural networks to extract a second feature map from the concatenated feature map. The number of the second neural network is consistent with the number of input pathological slice image blocks, which is 9 in this embodiment. Nine second neural networks extract features from the unique spliced feature map in parallel, and nine second feature maps can be obtained.
S5，将第二特征图进行叠加形成特征图集，9个第二特征图作为特征图集的9个通道。S5. The second feature maps are superimposed to form a feature atlas, and the 9 second feature maps are used as 9 channels of the feature atlas.
S6，对特征图集进行分类得到分类结果，分类结果用于表示多个图像块分别是否正常。例如可以使用分类器进行分类输出一组向量，其中包括9个0-1间的数值，分别是9个图像块是否包含异常内容的置信度。可以将该向量作为最终呈现的结果，也可以采用阈值法做进一步处理，使最终结果中呈现结论性信息。S6, classify the feature atlas to obtain a classification result, and the classification result is used to indicate whether each of the plurality of image blocks is normal. For example, a classifier can be used to classify and output a set of vectors, which include 9 values between 0 and 1, which are the confidence levels of whether the 9 image blocks contain abnormal content. This vector can be used as the final presented result, or a threshold method can be used for further processing, so that conclusive information can be presented in the final result.
作为一个优选的实施方式，在此步骤中首先利用GAP层对特征图集进行GAP处理，然后使用分类器根据GAP处理后的数据得到分类结果。在其它实施例中，也可以采用如全连接层、最大池化层等其它可行的处理方式来替代GAP层对特征图集进行处理。As a preferred implementation manner, in this step, the GAP layer is first used to perform GAP processing on the feature atlas, and then a classifier is used to obtain classification results based on the GAP-processed data. In other embodiments, other feasible processing methods such as fully connected layer and maximum pooling layer may also be used instead of the GAP layer to process the feature atlas.
作为一个举例，例如对特征图集进行分类得到向量(0.1,0，0.8，0.9，0.2，0，0.3，0.7，0.1)，这些数值越接近于1则表示相应的图像块包含异常内容的可能性越大，反之则越小。可以将这个向量作为最终的识别结果，医生可以参考这些置信度信息做出进一步的判断；也可以对该向量做进一步处理，将结果转化为(左上方正常、正上方正常、右上方异常……右下方正常)或类似的结论信息。As an example, for example, the vector (0.1, 0, 0.8, 0.9, 0.2, 0, 0.3, 0.7, 0.1) is obtained by classifying the feature atlas. The closer these values are to 1, the corresponding image block may contain abnormal content. The greater the sex, the smaller the vice versa. This vector can be used as the final recognition result, and the doctor can refer to the confidence information to make further judgments; the vector can also be further processed to convert the result into (normal in the upper left, normal in the upper right, abnormal in the upper right... lower right normal) or similar conclusion information.
根据本发明实施例提供的病理切片图像识别方法，首先通过一个神经网络分别从输入的多个病理切片图像块中提取特征图，进而根据各个图像块在病理切片图像中的位置拼接这些特征图，使得到的拼接特征图体现空间信息；然后再使用多个神经网络并行对该拼接特征图进行识别，其中各个神经网络可分别充分利用图像块的空间信息，进而提取多个特征图，最后对多个特征图进行组合，进而根据组合的特征图集得到分类结果。该方法能够使神经网络非常有效的进行空间信息的利用，在给一张病理切片图像进行分类时，会参考多种不同的空间位置信息，得出更准确的结果。According to the pathological slice image recognition method provided in the embodiment of the present invention, first, a neural network is used to extract feature maps from multiple input pathological slice image blocks, and then these feature maps are spliced according to the positions of each image block in the pathological slice image, The stitched feature map obtained reflects spatial information; then multiple neural networks are used to identify the stitched feature map in parallel, and each neural network can make full use of the spatial information of the image block, and then extract multiple feature maps. The feature maps are combined, and then the classification result is obtained according to the combined feature map set. This method enables the neural network to utilize spatial information very effectively. When classifying a pathological slice image, it will refer to a variety of different spatial position information to obtain more accurate results.
在一个优选的实施例中，第一特征图是未经全局平均池化处理的特征图。也就是说步骤S2中不包括GAP处理，提取在GAP层之前的数据即可。In a preferred embodiment, the first feature map is a feature map that has not been processed by global average pooling. That is to say, the step S2 does not include GAP processing, and it is only necessary to extract the data before the GAP layer.
第一神经网络不进行GAP处理，尽量多的保留整体的空间结构信息留给后续的第二个神经网络使用，使各个第二神经网络充分利用拼接特征图的空间信息，从而提高最终分类结果的准确性。The first neural network does not perform GAP processing, and retains as much overall spatial structure information as possible for the subsequent second neural network, so that each second neural network can make full use of the spatial information of the spliced feature map, thereby improving the accuracy of the final classification result. accuracy.
在一个优选的实施例中，第二特征图是未经全局平均池化处理的特征图。也就是说步骤S4中不包括GAP处理，提取在GAP层之前的数据即可。In a preferred embodiment, the second feature map is a feature map that has not been processed by global average pooling. That is to say, step S4 does not include GAP processing, and only data before the GAP layer can be extracted.
根据各个第二特征图得到特征图集的操作对空间信息没有要求，对最后的分类结果影响不明显，在此选择第二个神经网络不进行GAP处理可以提高整体识别效率。The operation of obtaining the feature atlas according to each second feature map does not require spatial information, and has no obvious impact on the final classification result. Here, choosing the second neural network without GAP processing can improve the overall recognition efficiency.
本实施例还提供一种病理切片图像识别设备，包括：至少一个处理器；以及与所述至少一个处理器通信连接的存储器；其中，所述存储器存储有可被所述一个处理器执行的指令，所述指令被所述至少一个处理器执行，以使所述至少一个处理器执行上述病理切片图像识别方法。This embodiment also provides a pathological slice image recognition device, including: at least one processor; and a memory connected to the at least one processor in communication; wherein, the memory stores instructions that can be executed by the one processor , the instructions are executed by the at least one processor, so that the at least one processor executes the above pathological slice image recognition method.
本发明实施例提供一种病理切片图像识别装置，如图3所示，该装置包括：An embodiment of the present invention provides a pathological slice image recognition device, as shown in Figure 3, the device includes:
划分模块31，用于将病理切片图像划分为多个图像块；A division module 31, configured to divide the pathological slice image into a plurality of image blocks;
第一神经网络32，用于分别从多个图像块中提取第一特征图；The first neural network 32 is used to extract the first feature map from multiple image blocks respectively;
拼接模块33，用于根据各个图像块在病理切片图像中的位置拼接各个第一特征图形成拼接特征图；Stitching module 33, for splicing each first feature map according to the position of each image block in the pathological slice image to form a splicing feature map;
多个第二神经网络34，分别用于从拼接特征图中提取第二特征图；A plurality of second neural networks 34 are respectively used to extract a second feature map from the spliced feature map;
组合模块35，用于将第二特征图进行叠加形成特征图集；A combination module 35, configured to superimpose the second feature map to form a feature map set;
分类模块36，用于对特征图集进行分类得到分类结果，分类结果用于表示多个图像块分别是否正常。The classification module 36 is configured to classify the feature atlas to obtain classification results, and the classification results are used to indicate whether the multiple image blocks are normal or not.
根据本发明实施例提供的病理切片图像识别装置，首先通过一个神经网络分别从输入的多个病理切片图像块中提取特征图，进而根据各个图像块在病理切片图像中的位置拼接这些特征图，使得到的拼接特征图体现空间信息；然后再使用多个神经网络并行对该拼接特征图进行识别，其中各个神经网络可分别充分利用图像块的空间信息，进而提取多个特征图，最后对多个特征图进行组合，进而根据组合的特征图集得到分类结果。该装置能够使神经网络非常有效的进行空间信息的利用，在给一张病理切片图像进行分类时，会参考多种不同的空间位置信息，得出更准确的结果。According to the pathological slice image recognition device provided in the embodiment of the present invention, first, a neural network is used to extract feature maps from multiple input pathological slice image blocks, and then these feature maps are spliced according to the positions of each image block in the pathological slice image, The stitched feature map obtained reflects spatial information; then multiple neural networks are used to identify the stitched feature map in parallel, and each neural network can make full use of the spatial information of the image block, and then extract multiple feature maps. The feature maps are combined, and then the classification result is obtained according to the combined feature map set. The device can enable the neural network to utilize spatial information very effectively, and when classifying a pathological slice image, it will refer to a variety of different spatial position information to obtain more accurate results.
在一个优选的实施例中，划分模块31采用等分方式得到9个尺寸相等的图像块。9个尺寸相等的图像块作为第一神经网络32的输入数据，第一神经网络32输出9个第一特征图，与9个尺寸相等的图像块一一对应；第二神经网络34的数量为9个，拼接特征图作为9个第二神经网络的输入数据，9个第二神经网络34分别输出1个第二特征图。In a preferred embodiment, the division module 31 obtains 9 image blocks of equal size by means of equal division. 9 image blocks of equal size are used as the input data of the first neural network 32, and the first feature map of the first neural network 32 outputs 9, corresponding to the image blocks of 9 equal sizes; the quantity of the second neural network 34 is Nine, spliced feature maps are used as the input data of the nine second neural networks, and the nine second neural networks 34 respectively output one second feature map.
在一个优选的实施例中，第一特征图、第二特征图是未经全局平均池化处理的特征图。In a preferred embodiment, the first feature map and the second feature map are feature maps that have not been processed by global average pooling.
在一个优选的实施例中，分类模块36包括：In a preferred embodiment, the classification module 36 includes:
全局平均池化模块，用于对特征图集进行全局平均池化处理；The global average pooling module is used to perform global average pooling processing on the feature atlas;
分类器，用于对全局平均池化处理结果进行分类得到分类结果。A classifier is used to classify the global average pooling processing results to obtain classification results.
本领域内的技术人员应明白，本发明的实施例可提供为方法、系统、或计算机程序产品。因此，本发明可采用完全硬件实施例、完全软件实施例、或结合软件和硬件方面的实施例的形式。而且，本发明可采用在一个或多个其中包含有计算机可用程序代码的计算机可用存储介质(包括但不限于磁盘存储器、CD-ROM、光学存储器等)上实施的计算机程序产品的形式。Those skilled in the art should understand that the embodiments of the present invention may be provided as methods, systems, or computer program products. Accordingly, the present invention can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.
本发明是参照根据本发明实施例的方法、设备(系统)、和计算机程序产品的流程图和/或方框图来描述的。应理解可由计算机程序指令实现流程图和/或方框图中的每一流程和/或方框、以及流程图和/或方框图中的流程和/或方框的结合。可提供这些计算机程序指令到通用计算机、专用计算机、嵌入式处理机或其他可编程数据处理设备的处理器以产生一个机器，使得通过计算机或其他可编程数据处理设备的处理器执行的指令产生用于实现在流程图一个流程或多个流程和/或方框图一个方框或多个方框中指定的功能的装置。The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each process and/or block in the flowchart illustrations and/or block diagrams, and combinations of processes and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a general purpose computer, special purpose computer, embedded processor, or processor of other programmable data processing equipment to produce a machine such that the instructions executed by the processor of the computer or other programmable data processing equipment produce a An apparatus for realizing the functions specified in one or more procedures of the flowchart and/or one or more blocks of the block diagram.
这些计算机程序指令也可存储在能引导计算机或其他可编程数据处理设备以特定方式工作的计算机可读存储器中，使得存储在该计算机可读存储器中的指令产生包括指令装置的制造品，该指令装置实现在流程图一个流程或多个流程和/或方框图一个方框或多个方框中指定的功能。These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing apparatus to operate in a specific manner, such that the instructions stored in the computer-readable memory produce an article of manufacture comprising instruction means, the instructions The device realizes the function specified in one or more procedures of the flowchart and/or one or more blocks of the block diagram.
这些计算机程序指令也可装载到计算机或其他可编程数据处理设备上，使得在计算机或其他可编程设备上执行一系列操作步骤以产生计算机实现的处理，从而在计算机或其他可编程设备上执行的指令提供用于实现在流程图一个流程或多个流程和/或方框图一个方框或多个方框中指定的功能的步骤。These computer program instructions can also be loaded onto a computer or other programmable data processing device, causing a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, thereby The instructions provide steps for implementing the functions specified in the flow chart or blocks of the flowchart and/or the block or blocks of the block diagrams.
显然，上述实施例仅仅是为清楚地说明所作的举例，而并非对实施方式的限定。对于所属领域的普通技术人员来说，在上述说明的基础上还可以做出其它不同形式的变化或变动。这里无需也无法对所有的实施方式予以穷举。而由此所引伸出的显而易见的变化或变动仍处于本发明创造的保护范围之中。Apparently, the above-mentioned embodiments are only examples for clear description, rather than limiting the implementation. For those of ordinary skill in the art, changes or modifications in other different forms can also be made on the basis of the above description. There is no need and cannot be exhaustive of all implementations here. And the obvious changes or changes derived from this are still within the protection scope of the present invention.
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910224185.7ACN110084270A (en) | 2019-03-22 | 2019-03-22 | Pathological slice image recognition method and device |
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910224185.7ACN110084270A (en) | 2019-03-22 | 2019-03-22 | Pathological slice image recognition method and device |
| Publication Number | Publication Date |
|---|---|
| CN110084270Atrue CN110084270A (en) | 2019-08-02 |
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910224185.7APendingCN110084270A (en) | 2019-03-22 | 2019-03-22 | Pathological slice image recognition method and device |
| Country | Link |
|---|---|
| CN (1) | CN110084270A (en) |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110600106A (en)* | 2019-08-28 | 2019-12-20 | 上海联影智能医疗科技有限公司 | Pathological section processing method, computer device and storage medium |
| CN111210006A (en)* | 2020-01-02 | 2020-05-29 | 北京科技大学 | Heterogeneous deep neural network structure combination method and system for multi-feature map |
| CN115063369A (en)* | 2019-09-30 | 2022-09-16 | 上海联影智能医疗科技有限公司 | Brain image detection method, computer equipment and storage medium |
| CN118334518A (en)* | 2024-04-12 | 2024-07-12 | 亿海蓝（北京）数据技术股份公司 | Identification method, identification device, electronic apparatus, and readable storage medium |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106780475A (en)* | 2016-12-27 | 2017-05-31 | 北京市计算中心 | A kind of image processing method and device based on histopathologic slide's image organizational region |
| CN106934798A (en)* | 2017-02-20 | 2017-07-07 | 苏州体素信息科技有限公司 | Diabetic retinopathy classification stage division based on deep learning |
| CN107492099A (en)* | 2017-08-28 | 2017-12-19 | 京东方科技集团股份有限公司 | Medical image analysis method, medical image analysis system and storage medium |
| CN108197606A (en)* | 2018-01-31 | 2018-06-22 | 浙江大学 | The recognition methods of abnormal cell in a kind of pathological section based on multiple dimensioned expansion convolution |
| CN108734211A (en)* | 2018-05-17 | 2018-11-02 | 腾讯科技（深圳）有限公司 | The method and apparatus of image procossing |
| CN109376777A (en)* | 2018-10-18 | 2019-02-22 | 四川木牛流马智能科技有限公司 | Cervical cancer tissues pathological image analysis method and equipment based on deep learning |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106780475A (en)* | 2016-12-27 | 2017-05-31 | 北京市计算中心 | A kind of image processing method and device based on histopathologic slide's image organizational region |
| CN106934798A (en)* | 2017-02-20 | 2017-07-07 | 苏州体素信息科技有限公司 | Diabetic retinopathy classification stage division based on deep learning |
| CN107492099A (en)* | 2017-08-28 | 2017-12-19 | 京东方科技集团股份有限公司 | Medical image analysis method, medical image analysis system and storage medium |
| CN108197606A (en)* | 2018-01-31 | 2018-06-22 | 浙江大学 | The recognition methods of abnormal cell in a kind of pathological section based on multiple dimensioned expansion convolution |
| CN108734211A (en)* | 2018-05-17 | 2018-11-02 | 腾讯科技（深圳）有限公司 | The method and apparatus of image procossing |
| CN109376777A (en)* | 2018-10-18 | 2019-02-22 | 四川木牛流马智能科技有限公司 | Cervical cancer tissues pathological image analysis method and equipment based on deep learning |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110600106A (en)* | 2019-08-28 | 2019-12-20 | 上海联影智能医疗科技有限公司 | Pathological section processing method, computer device and storage medium |
| CN110600106B (en)* | 2019-08-28 | 2022-07-05 | 上海联影智能医疗科技有限公司 | Pathological section processing method, computer device and storage medium |
| CN115083571A (en)* | 2019-08-28 | 2022-09-20 | 上海联影智能医疗科技有限公司 | Pathological slice processing method, computer equipment and storage medium |
| CN115063369A (en)* | 2019-09-30 | 2022-09-16 | 上海联影智能医疗科技有限公司 | Brain image detection method, computer equipment and storage medium |
| CN115063369B (en)* | 2019-09-30 | 2025-04-25 | 上海联影智能医疗科技有限公司 | Brain image detection method, computer device and storage medium |
| CN111210006A (en)* | 2020-01-02 | 2020-05-29 | 北京科技大学 | Heterogeneous deep neural network structure combination method and system for multi-feature map |
| CN111210006B (en)* | 2020-01-02 | 2023-07-28 | 北京科技大学 | Multi-feature map-oriented heterogeneous deep neural network structure combination method and system |
| CN118334518A (en)* | 2024-04-12 | 2024-07-12 | 亿海蓝（北京）数据技术股份公司 | Identification method, identification device, electronic apparatus, and readable storage medium |
| Publication | Publication Date | Title |
|---|---|---|
| US11069062B2 (en) | Automated screening of histopathology tissue samples via analysis of a normal model | |
| CN111951221B (en) | A glomerular cell image recognition method based on deep neural network | |
| CN110889853B (en) | Tumor segmentation method based on residual error-attention deep neural network | |
| CN108986115B (en) | Medical image segmentation method and device and intelligent terminal | |
| CN105144239B (en) | Image processing apparatus, image processing method | |
| CN109409416A (en) | Feature vector dimension reduction method and medical image recognition method, apparatus and storage medium | |
| CN110084270A (en) | Pathological slice image recognition method and device | |
| US12315142B2 (en) | Automated clustering of anomalous histopathology tissue samples | |
| CN115136189A (en) | Automated detection of tumors based on image processing | |
| JP2017107543A (en) | Method and system for automated analysis of cell images | |
| CN110647889B (en) | Medical image recognition method, medical image recognition apparatus, terminal device, and medium | |
| CN112132827A (en) | Pathological image processing method and device, electronic equipment and readable storage medium | |
| CN112581458A (en) | Image processing method and device | |
| CN112581438A (en) | Slice image recognition method and device, storage medium and electronic equipment | |
| CN114511523B (en) | Gastric cancer molecular subtype classification method and device based on self-supervised learning | |
| CN111583385A (en) | A personalized deformation method and system for a deformable digital human anatomy model | |
| CN112215217B (en) | Digital image recognition method and device for simulating doctor to read film | |
| US20180047158A1 (en) | Chest radiograph (cxr) image analysis | |
| Cerrolaza et al. | Fetal skull segmentation in 3D ultrasound via structured geodesic random forest | |
| CN114511702B (en) | A remote sensing image segmentation method and system based on multi-scale decentralized attention | |
| CN113379770B (en) | Construction method of nasopharyngeal carcinoma MR image segmentation network, image segmentation method and device | |
| CN111951271B (en) | Method and device for identifying cancer cells in pathological image | |
| CN114287042A (en) | Analysis method and related device of brain image based on VRDS AI | |
| CN110992310A (en) | Method and device for determining partition where mediastinal lymph node is located | |
| CN112990367A (en) | Image processing method, device, equipment and storage medium |
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | Application publication date:20190802 | |
| RJ01 | Rejection of invention patent application after publication |