WO2019001209A1 - Classification algorithm for retinal oct image based on three-dimensional convolutional neural network - Google Patents
Classification algorithm for retinal oct image based on three-dimensional convolutional neural networkDownload PDFInfo
- Publication number
- WO2019001209A1 WO2019001209A1PCT/CN2018/089072CN2018089072WWO2019001209A1WO 2019001209 A1WO2019001209 A1WO 2019001209A1CN 2018089072 WCN2018089072 WCN 2018089072WWO 2019001209 A1WO2019001209 A1WO 2019001209A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- dimensional
- convolutional neural
- neural network
- layer
- retinal
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Ceased
Links
Images
Classifications
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/09—Supervised learning
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/0464—Convolutional networks [CNN, ConvNet]
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/096—Transfer learning
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0012—Biomedical image inspection
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10072—Tomographic images
- G06T2207/10101—Optical tomography; Optical coherence tomography [OCT]
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30004—Biomedical image processing
- G06T2207/30041—Eye; Retina; Ophthalmic
Definitions
- the inventionrelates to a classification algorithm of a retinal OCT image based on a three-dimensional convolutional neural network, and belongs to the technical field of retinal image classification.
- the existing automatic retinal classification technologyis mostly based on retinal OCT of the fundus or small field of view, that is, the retinal OCT image centered on the macula or centered on the optic nerve head.
- OCT imagesCompared with the fundus color photography, OCT images have the advantages of non-invasive, high-speed, high-resolution, three-dimensional imaging, etc.
- the retinal classification in OCT imagesstill faces many challenges: the difference between the various types of images is not obvious, and the image itself has a large number of speckle noise. These problems make it difficult for traditional methods to achieve more accurate classification results.
- Convolutional neural networkshave powerful learning capabilities, and have achieved great success in medical image classification (for example, classification of mammography images, classification of CT interstitial images, classification of diabetic retinal fundus, etc.). So consider using this framework for the task of retinal OCT image classification.
- the above medical imagesare all two-dimensional images, and both use a two-dimensional convolutional neural network to achieve the classification effect.
- the OCT imageis a three-dimensional image, it is insufficient to directly classify the image using the traditional two-dimensional convolutional neural network: 1) The two-dimensional network structure can not use the three-dimensional spatial information of the image, which will lose a lot of useful information, thus limiting the classification performance of the model; (2) The convolutional neural network is a multi-layer learning network, and the traditional method is only the last one of the network. The layer is supervised, ignoring the influence of the middle layer supervision on the classification effect of the model.
- the technical problem to be solved by the present inventionis to provide a classification algorithm for retinal OCT images based on a three-dimensional convolutional neural network that can classify three-dimensional retinal OCT images and improve classification accuracy.
- the technical solution adopted by the present inventionis:
- a classification algorithm for retinal OCT images based on a three-dimensional convolutional neural networkcomprising the following steps:
- S01Collecting three kinds of retinal OCT images, which are a retinal OCT image centered on the macula, a retinal OCT image centered on the optic nerve nipple, and a retinal OCT image centered on a large field of view, and each of the three retinas is labeled.
- retinal OCT imageswhich are a retinal OCT image centered on the macula, a retinal OCT image centered on the optic nerve nipple, and a retinal OCT image centered on a large field of view, and each of the three retinas is labeled.
- 2 categoriesrespectively, normal retinal images and abnormal retinal images;
- S02data preprocessing, downsampling the three-dimensional OCT image data, and obtaining a three-dimensional image of uniform size for inputting a three-dimensional convolutional neural network;
- S03Pre-training a three-dimensional convolutional neural network model with a plurality of marked natural images according to the migration learning idea, the three-dimensional convolutional neural network model comprising an input layer, a plurality of convolutional layers, a plurality of pooling layers, and a plurality of fully connected layers And an output layer;
- S04Fine-tuning the trained three-dimensional convolutional neural network model with the pre-processed retinal OCT image, and adding a branch network after the convolutional layer in the middle of the mainstream network, the branch network further extracting the image part by using the multi-layer perceptual convolution Information, the branch network is also composed of a convolution layer, a pooling layer, a full connection layer, and an output layer, and the mainstream network and the output layer of the branch network are merged;
- test imageis preprocessed according to the step S02, and the three-dimensional convolutional neural network model after fine adjustment in the step S04 is used for testing, and the classification result is output.
- All three-dimensional OCT imageswere downsampled by three-dimensional linear interpolation to obtain three-dimensional images of 96 ⁇ 96 ⁇ 16 size.
- the trained mainstream three-dimensional convolutional neural network model networkincludes one input layer, eight convolutional layers, five pooling layers, two fully connected layers, and one output layer, from 1 to 8 convolutional layers.
- the filter numberis 64, 128, 256, 256, 512, 512, 512, 512
- the convolution kernel sizeis 3 ⁇ 3 ⁇ 3
- each fully connected layerhas 4096 outputs
- the output layerhas 6 outputs
- the pooling layeruses the maximum pooling method.
- the branch networkis added after the fourth convolutional layer.
- the branch network of the branch networkuses an average pooling method to downsample the fourth volume layer, including one pooling layer, two convolution layers, two fully connected layers, and one output layer.
- the output layeruses a Softmax classifier.
- the fusion in step S04employs a maximum probability fusion method.
- the beneficial effects achieved by the inventionare that the OCT image of the small field of view can only obtain the structural information of the retina, and the OCT image of the large field of view can not only obtain the retinal information centered on the macula, but also acquire the retinal information centered on the optic nerve nipple, and combine
- the three kinds of OCT images of small field of view image and large field of view imagecan obtain more comprehensive retinal information.
- a classification methodis proposed, which can comprehensively analyze retinal information, thereby improving the robustness of the retinal classification algorithm.
- the three-dimensional convolutional neural networkcan be used to extract three-dimensional image information, directly process the three-dimensional OCT image, and use the mainstream network to combine the branch network, which can make good use of local information, improve the accuracy of the classification result, and improve the recognition accuracy of the convolutional neural network.
- the branch networkis added from the middle, avoiding the fact that only a small amount of local information can be used too far, and only the overall information can be used too late. After many experiments, it is best to add the layer 4 layer.
- 1is a three-dimensional convolutional neural network model constructed by the present invention
- Figure 2shows three retinal OCT images based on three different scanning positions, (a) for fundus color photography, (b) retinal OCT image centered on large field of view, and (c) retinal OCT centered on optic nerve head.
- the image, (d)is a retinal OCT image centered on the macula.
- a classification algorithm for retinal OCT images based on a three-dimensional convolutional neural networkcomprising the following steps:
- S01collecting three retinal OCT images, which are a retinal OCT image centered on the macula, a retinal OCT image centered on the optic nerve head, and a retinal OCT image centered on the large field of view, as shown in FIG.
- Each type of retinais labeled as 2 types, which are normal retinal images and abnormal retinal images.
- Theseare: large field of view abnormal retinal OCT image (ANW), large visual field normal retinal OCT image (NW), centered on macula Abnormal retinal OCT image (ANM), normal retinal OCT image (NM) centered on the macula, abnormal retinal OCT image (ANO) centered on the optic nerve head, and normal retinal OCT image centered on the optic nerve head (NO );
- S02Data pre-processing, down-sampling the three-dimensional OCT image data, using three-dimensional linear interpolation method, downsampling all three-dimensional OCT images, and obtaining a unified three-dimensional image of 96 ⁇ 96 ⁇ 16 size for inputting three-dimensional convolutional nerves The internet;
- the trained mainstream three-dimensional convolutional neural network model networkincludes one input layer, eight convolutional layers, five pooling layers, two fully connected layers, and one output layer, from 1 to 8
- the number of filters in the convolutional layeris 64, 128, 256, 256, 512, 512, 512, 512
- the convolution kernel sizeis 3 ⁇ 3 ⁇ 3
- each fully connected layerhas 4096 outputs
- the output layerhas 6 outputs
- the pooling layeruses maximum pooling. law.
- the output layeruses a Softmax classifier to output a probability that each input image belongs to each class, selects the category with the highest probability as the input picture, and uses the stochastic gradient descent method to solve the weight of the con
- S04Fine-tuning the trained three-dimensional convolutional neural network model with the pre-processed retinal OCT image, and adding a branch network after the convolutional layer in the middle of the mainstream network, the branch network further extracting the image part by using the multi-layer perceptual convolution Information, the branch network is also composed of a convolution layer, a pooling layer, a fully connected layer, and an output layer, including one pooling layer, two convolution layers, two fully connected layers, and one output layer.
- the fourth volume layeris downsampled by the average pooling method, and the average pooling method is used for downsampling, and the mainstream network and the output layer of the branch network are merged.
- the maximum probability fusion methodthe output belongs to the category with the highest probability;
- test imageis preprocessed according to the step S02, and the three-dimensional convolutional neural network model after fine adjustment in S04 is used for testing, and the classification result is output.
- the accuracy of classification of the three retinal OCT imagesis higher than that of most traditional three-dimensional convolutional neural networks.
- the present inventionfirst proposes to classify retinal OCT images into six categories and utilize three-dimensional
- the improved convolutional neural network modelrealizes the automatic classification of 6 types of OCT images for the first time.
- This network structure modelcan be applied to the automatic classification of other 3D medical images through different data training.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- General Health & Medical Sciences (AREA)
- Health & Medical Sciences (AREA)
- Evolutionary Computation (AREA)
- General Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Mathematical Physics (AREA)
- Computing Systems (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Molecular Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Computational Biology (AREA)
- Evolutionary Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Medical Informatics (AREA)
- Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
- Radiology & Medical Imaging (AREA)
- Quality & Reliability (AREA)
- Image Analysis (AREA)
- Eye Examination Apparatus (AREA)
Abstract
Translated fromChineseDescription
Translated fromChinese本发明涉及一种基于三维卷积神经网络的视网膜OCT图像的分类算法，属于视网膜图像分类技术领域。The invention relates to a classification algorithm of a retinal OCT image based on a three-dimensional convolutional neural network, and belongs to the technical field of retinal image classification.
现有的视网膜自动分类技术大部分基于眼底彩照或者小视野的视网膜OCT，即以黄斑为中心或者以视神经乳头为中心的视网膜OCT图像。相比较眼底彩照，OCT图像具有无创、高速、高分辨率、三维成像等优点，不过OCT图像中视网膜分类目前仍面临诸多挑战：图像各类之间差异不明显，图像本身存在大量斑噪声等。这些问题使得传统的方法很难取得较为精确的分类效果。卷积神经网络具有强大的学习能力，其在医学图像分类(例如，乳腺X线肿瘤图像的分类、CT肺间质图像的分类、糖尿病性视网膜眼底彩照分类等)中已取得了巨大的成功。所以考虑将该框架用于视网膜OCT图像分类的任务中。然而，以上的医学图像都是二维图像，都使用二维卷积神经网络来达到分类效果，由于OCT图像为三维图像，直接使用传统的二维卷积神经网络对图像进行分类存在不足：(1)二维网络结构不能利用图像的三维空间信息，会损失很多有用信息，从而限制了模型的分类性能；(2)卷积神经网络为多层学习网络，传统的方法只是对网络的最后一层进行监督，忽略了中间层监督对模型分类效果的影响。The existing automatic retinal classification technology is mostly based on retinal OCT of the fundus or small field of view, that is, the retinal OCT image centered on the macula or centered on the optic nerve head. Compared with the fundus color photography, OCT images have the advantages of non-invasive, high-speed, high-resolution, three-dimensional imaging, etc. However, the retinal classification in OCT images still faces many challenges: the difference between the various types of images is not obvious, and the image itself has a large number of speckle noise. These problems make it difficult for traditional methods to achieve more accurate classification results. Convolutional neural networks have powerful learning capabilities, and have achieved great success in medical image classification (for example, classification of mammography images, classification of CT interstitial images, classification of diabetic retinal fundus, etc.). So consider using this framework for the task of retinal OCT image classification. However, the above medical images are all two-dimensional images, and both use a two-dimensional convolutional neural network to achieve the classification effect. Since the OCT image is a three-dimensional image, it is insufficient to directly classify the image using the traditional two-dimensional convolutional neural network: 1) The two-dimensional network structure can not use the three-dimensional spatial information of the image, which will lose a lot of useful information, thus limiting the classification performance of the model; (2) The convolutional neural network is a multi-layer learning network, and the traditional method is only the last one of the network. The layer is supervised, ignoring the influence of the middle layer supervision on the classification effect of the model.
发明内容Summary of the invention
本发明所要解决的技术问题是，提供一种可以对三维视网膜OCT图像进行分类，提高分类准确性的基于三维卷积神经网络的视网膜OCT图像的分类算法。The technical problem to be solved by the present invention is to provide a classification algorithm for retinal OCT images based on a three-dimensional convolutional neural network that can classify three-dimensional retinal OCT images and improve classification accuracy.
为解决上述技术问题，本发明采用的技术方案为：In order to solve the above technical problems, the technical solution adopted by the present invention is:
基于三维卷积神经网络的视网膜OCT图像的分类算法，包括以下步骤：A classification algorithm for retinal OCT images based on a three-dimensional convolutional neural network, comprising the following steps:
S01：采集三种视网膜OCT图像，分别是以黄斑为中心的视网膜OCT图像、以视神经乳头为中心的视网膜OCT图像和以大视野为中心的视网膜OCT图像，将所述三种视网膜每种分类标记为2类，分别是正常视网膜图像和异常视网膜 图像；S01: Collecting three kinds of retinal OCT images, which are a retinal OCT image centered on the macula, a retinal OCT image centered on the optic nerve nipple, and a retinal OCT image centered on a large field of view, and each of the three retinas is labeled. For 2 categories, respectively, normal retinal images and abnormal retinal images;
S02：数据预处理，对三维OCT图像数据进行降采样，得到统一大小的三维图像用于输入三维卷积神经网络；S02: data preprocessing, downsampling the three-dimensional OCT image data, and obtaining a three-dimensional image of uniform size for inputting a three-dimensional convolutional neural network;
S03：根据迁移学习思想，用大量标记好的自然图像预训练一个三维卷积神经网络模型，所述三维卷积神经网络模型包括输入层、若干卷积层、若干池化层、若干全连接层以及输出层；S03: Pre-training a three-dimensional convolutional neural network model with a plurality of marked natural images according to the migration learning idea, the three-dimensional convolutional neural network model comprising an input layer, a plurality of convolutional layers, a plurality of pooling layers, and a plurality of fully connected layers And an output layer;
S04：用预处理好的视网膜OCT图像对训练好的三维卷积神经网络模型进行微调，在主流网络中间的卷积层后加入分支网络，所述分支网络利用多层感知卷积进一步提取图像局部信息，所述分支网络同样由卷积层、池化层、全连接层和输出层组成，将主流网络和所述分支网络的输出层进行融合；S04: Fine-tuning the trained three-dimensional convolutional neural network model with the pre-processed retinal OCT image, and adding a branch network after the convolutional layer in the middle of the mainstream network, the branch network further extracting the image part by using the multi-layer perceptual convolution Information, the branch network is also composed of a convolution layer, a pooling layer, a full connection layer, and an output layer, and the mainstream network and the output layer of the branch network are merged;
S05：将测试图像按照S02步骤进行预处理，利用S04步骤中微调后的所述三维卷积神经网络模型进行测试，输出分类结果。S05: The test image is preprocessed according to the step S02, and the three-dimensional convolutional neural network model after fine adjustment in the step S04 is used for testing, and the classification result is output.
采用三维线性插值的方法，对所有三维OCT图像进行降采样，得到96×96×16尺寸大小的三维图像。All three-dimensional OCT images were downsampled by three-dimensional linear interpolation to obtain three-dimensional images of 96×96×16 size.
所述训练好的三维卷积神经网络模型主流网络包括1个输入层、8个卷积层、5个池化层、2个全连接层和1个输出层，从1到8卷积层的滤波器个数分是64,128,256,256,512,512,512,512，卷积核大小是3×3×3，每个全连接层有4096个输出，输出层有6个输出，所述池化层均采用最大池化法。The trained mainstream three-dimensional convolutional neural network model network includes one input layer, eight convolutional layers, five pooling layers, two fully connected layers, and one output layer, from 1 to 8 convolutional layers. The filter number is 64, 128, 256, 256, 512, 512, 512, 512, the convolution kernel size is 3 × 3 × 3, each fully connected layer has 4096 outputs, and the output layer has 6 outputs, and the pooling layer uses the maximum pooling method.
所述分支网络加在第4个卷积层后。The branch network is added after the fourth convolutional layer.
所述分支网络所述分支网络采用平均池化方法对第4卷积层进行降采样，包括1个池化层、2个卷积层、2个全连接层和1个输出层。The branch network of the branch network uses an average pooling method to downsample the fourth volume layer, including one pooling layer, two convolution layers, two fully connected layers, and one output layer.
所述输出层采用Softmax分类器。The output layer uses a Softmax classifier.
步骤S04中所述融合采用最大概率融合方法。The fusion in step S04 employs a maximum probability fusion method.
本发明所达到的有益效果：小视野的OCT图像只能获取视网膜部分结构信息，大视野的OCT图像不仅能获取以黄斑为中心的视网膜信息，而且能获取以视神经乳头为中心的视网膜信息，结合两种小视野图像和大视野图像这三种OCT图像，能够获取更加全面的视网膜信息，本发明据此提出来一种分类方法，可以更全面分析视网膜信息，从而可以提高视网膜分类算法的鲁棒性；由于医 学图像数据量较少，根据迁移学习的思想，因为自然图像与医学图像的底部特征是相似的，可以利用大量的自然图像来预训练三维卷积神经网络，再用OCT图像进行微调，利用三维卷积神经网络可以提取图像三维信息，直接对三维OCT图像进行处理，采用主流网络结合分支网络，可以利用好局部信息，提高分类结果的准确性，提高卷积神经网络的识别精度，还可以将其应用于数据量较少的领域，采用从中间加入分支网络，避免了太靠前只能利用少量局部信息以及太靠后只能利用整体信息，通过多次试验，加在第4卷积层后效果最好。The beneficial effects achieved by the invention are that the OCT image of the small field of view can only obtain the structural information of the retina, and the OCT image of the large field of view can not only obtain the retinal information centered on the macula, but also acquire the retinal information centered on the optic nerve nipple, and combine The three kinds of OCT images of small field of view image and large field of view image can obtain more comprehensive retinal information. According to the present invention, a classification method is proposed, which can comprehensively analyze retinal information, thereby improving the robustness of the retinal classification algorithm. Because of the small amount of medical image data, according to the idea of migration learning, because the bottom features of natural images and medical images are similar, a large number of natural images can be used to pre-train the three-dimensional convolutional neural network, and then fine-tuned with OCT images. The three-dimensional convolutional neural network can be used to extract three-dimensional image information, directly process the three-dimensional OCT image, and use the mainstream network to combine the branch network, which can make good use of local information, improve the accuracy of the classification result, and improve the recognition accuracy of the convolutional neural network. Can also be applied to the amount of data In a few areas, the branch network is added from the middle, avoiding the fact that only a small amount of local information can be used too far, and only the overall information can be used too late. After many experiments, it is best to add the layer 4 layer.
图1是本发明构建的三维卷积神经网络模型；1 is a three-dimensional convolutional neural network model constructed by the present invention;
图2是根据三种不同的扫描位置得到三种视网膜OCT图像，(a)为眼底彩照，(b)为以大视野为中心的视网膜OCT图像，(c)为以视神经乳头为中心的视网膜OCT图像，(d)为以黄斑为中心的视网膜OCT图像。Figure 2 shows three retinal OCT images based on three different scanning positions, (a) for fundus color photography, (b) retinal OCT image centered on large field of view, and (c) retinal OCT centered on optic nerve head. The image, (d) is a retinal OCT image centered on the macula.
下面结合附图对本发明作进一步描述。以下实施例仅用于更加清楚地说明本发明的技术方案，而不能以此来限制本发明的保护范围。The invention is further described below in conjunction with the drawings. The following examples are only intended to more clearly illustrate the technical solutions of the present invention, and are not intended to limit the scope of the present invention.
实施例1Example 1
基于三维卷积神经网络的视网膜OCT图像的分类算法，包括以下步骤：A classification algorithm for retinal OCT images based on a three-dimensional convolutional neural network, comprising the following steps:
S01：采集三种视网膜OCT图像，分别是以黄斑为中心的视网膜OCT图像、以视神经乳头为中心的视网膜OCT图像和以大视野为中心的视网膜OCT图像，如图1所示，将所述三种视网膜每种分类标记为2类，分别是正常视网膜图像和异常视网膜图像，这6类为：大视野不正常视网膜OCT图像(ANW)、大视野正常视网膜OCT图像(NW)、以黄斑为中心的不正常视网膜OCT图像(ANM)、以黄斑为中心的正常视网膜OCT图像(NM)、以视神经乳头为中心的不正常视网膜OCT图像(ANO)和以视神经乳头为中心的正常视网膜OCT图像(NO)；S01: collecting three retinal OCT images, which are a retinal OCT image centered on the macula, a retinal OCT image centered on the optic nerve head, and a retinal OCT image centered on the large field of view, as shown in FIG. Each type of retina is labeled as 2 types, which are normal retinal images and abnormal retinal images. These are: large field of view abnormal retinal OCT image (ANW), large visual field normal retinal OCT image (NW), centered on macula Abnormal retinal OCT image (ANM), normal retinal OCT image (NM) centered on the macula, abnormal retinal OCT image (ANO) centered on the optic nerve head, and normal retinal OCT image centered on the optic nerve head (NO );
S02：数据预处理，对三维OCT图像数据进行降采样，采用三维线性插值的方法，对所有三维OCT图像进行降采样，得到统一96×96×16尺寸大小的三维图像用于输入三维卷积神经网络；S02: Data pre-processing, down-sampling the three-dimensional OCT image data, using three-dimensional linear interpolation method, downsampling all three-dimensional OCT images, and obtaining a unified three-dimensional image of 96×96×16 size for inputting three-dimensional convolutional nerves The internet;
S03：根据迁移学习思想，用大量标记好的自然图像预训练一个三维卷积神 经网络模型，所述三维卷积神经网络模型包括输入层、若干卷积层、若干池化层、若干全连接层以及输出层，所述训练好的三维卷积神经网络模型主流网络包括1个输入层、8个卷积层、5个池化层、2个全连接层和1个输出层，从1到8卷积层的滤波器个数分是64,128,256,256,512,512,512,512，卷积核大小是3×3×3，每个全连接层有4096个输出，输出层有6个输出，所述池化层均采用最大池化法。所述输出层采用Softmax分类器输出每个输入图像属于每个类的概率，选择概率最大的作为该输入图片的类别，使用随机梯度下降法对卷积神经网络的权重进行求解；S03: Pre-training a three-dimensional convolutional neural network model with a plurality of marked natural images according to the migration learning idea, the three-dimensional convolutional neural network model comprising an input layer, a plurality of convolutional layers, a plurality of pooling layers, and a plurality of fully connected layers And an output layer, the trained mainstream three-dimensional convolutional neural network model network includes one input layer, eight convolutional layers, five pooling layers, two fully connected layers, and one output layer, from 1 to 8 The number of filters in the convolutional layer is 64, 128, 256, 256, 512, 512, 512, 512, the convolution kernel size is 3 × 3 × 3, each fully connected layer has 4096 outputs, and the output layer has 6 outputs, and the pooling layer uses maximum pooling. law. The output layer uses a Softmax classifier to output a probability that each input image belongs to each class, selects the category with the highest probability as the input picture, and uses the stochastic gradient descent method to solve the weight of the convolutional neural network;
S04：用预处理好的视网膜OCT图像对训练好的三维卷积神经网络模型进行微调，在主流网络中间的卷积层后加入分支网络，所述分支网络利用多层感知卷积进一步提取图像局部信息，所述分支网络同样由卷积层、池化层、全连接层和输出层组成，包括1个池化层、2个卷积层、2个全连接层和1个输出层，所述分支网络加在第4个卷积层后，采用平均池化方法对第4卷积层进行降采样，采用平均池化方法进行降采样，将主流网络和所述分支网络的输出层进行融合，采用最大概率融合方法，输出属于概率最大的类别；S04: Fine-tuning the trained three-dimensional convolutional neural network model with the pre-processed retinal OCT image, and adding a branch network after the convolutional layer in the middle of the mainstream network, the branch network further extracting the image part by using the multi-layer perceptual convolution Information, the branch network is also composed of a convolution layer, a pooling layer, a fully connected layer, and an output layer, including one pooling layer, two convolution layers, two fully connected layers, and one output layer. After the branch network is added to the fourth convolutional layer, the fourth volume layer is downsampled by the average pooling method, and the average pooling method is used for downsampling, and the mainstream network and the output layer of the branch network are merged. Using the maximum probability fusion method, the output belongs to the category with the highest probability;
S05：将测试图像按照S02步骤进行预处理，利用S04中微调后的所述三维卷积神经网络模型进行测试，输出分类结果。S05: The test image is preprocessed according to the step S02, and the three-dimensional convolutional neural network model after fine adjustment in S04 is used for testing, and the classification result is output.
实验结果Experimental result
我们在873个数据上进行了本发明方法的测试。采用了三折交叉验证方法来检验本方法的可行性和有效性，采用F值(F-score)、正确率和AUC(Area under Curve)值作为评估方法的客观标准，结果见表1以及表2，表1比较了传统的三维卷积神经网络和改进的三维卷积神经网络的关于每一类F值和平均正确率；表2比较了传统的三维卷积神经网络和改进的三维卷积神经网络在每折实验中的每一类的AUC值。We tested the method of the invention on 873 data. The three-fold cross-validation method was used to test the feasibility and effectiveness of the method. The F-value (F-score), correctness rate and AUC (Area under Curve) values were used as objective criteria for the evaluation method. The results are shown in Table 1 and Table. 2, Table 1 compares the traditional three-dimensional convolutional neural network and the improved three-dimensional convolutional neural network for each type of F value and average correct rate; Table 2 compares the traditional three-dimensional convolutional neural network and improved three-dimensional convolution The AUC value of each type of neural network in each fold experiment.
表1 两种方法F值和平均正确率的比较Table 1 Comparison of F value and average correct rate of two methods
表2 两种方法AUC值的比较Table 2 Comparison of AUC values between the two methods
结果分析：本发明对三种视网膜OCT图像进行分类的准确性比大部分传统的三维卷积神经网络的准确性高，本发明首次提出将视网膜OCT图像分为6类的分类方案，并利用三维改进的卷积神经网络模型，首次实现了6类OCT图像的自动分类，这个网络结构模型通过不同数据的训练可适用于其他三维医学图像的自动分类。Analysis of results: The accuracy of classification of the three retinal OCT images is higher than that of most traditional three-dimensional convolutional neural networks. The present invention first proposes to classify retinal OCT images into six categories and utilize three-dimensional The improved convolutional neural network model realizes the automatic classification of 6 types of OCT images for the first time. This network structure model can be applied to the automatic classification of other 3D medical images through different data training.
以上所述仅是本发明的优选实施方式，应当指出，对于本技术领域的普通技术人员来说，在不脱离本发明技术原理的前提下，还可以做出若干改进和变 形，这些改进和变形也应视为本发明的保护范围。The above is only a preferred embodiment of the present invention, and it should be noted that those skilled in the art can make several improvements and modifications without departing from the technical principles of the present invention. It should also be considered as the scope of protection of the present invention.
Claims (7)
Translated fromChineseApplications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710506132.5 | 2017-06-28 | ||
| CN201710506132.5ACN107437092B (en) | 2017-06-28 | 2017-06-28 | Classification of retinal OCT images based on three-dimensional convolutional neural network |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| WO2019001209A1true WO2019001209A1 (en) | 2019-01-03 |
Family
ID=60459558
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| PCT/CN2018/089072CeasedWO2019001209A1 (en) | 2017-06-28 | 2018-05-30 | Classification algorithm for retinal oct image based on three-dimensional convolutional neural network |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN107437092B (en) |
| WO (1) | WO2019001209A1 (en) |
Cited By (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108395986A (en)* | 2018-03-23 | 2018-08-14 | 余晖 | Human papilloma virus automatic parting direction detection device based on deep learning |
| CN109816714A (en)* | 2019-01-15 | 2019-05-28 | 西北大学 | A point cloud object type recognition method based on 3D convolutional neural network |
| CN110222716A (en)* | 2019-05-08 | 2019-09-10 | 天津大学 | Image classification method based on full resolution depth convolutional neural networks |
| CN110443286A (en)* | 2019-07-18 | 2019-11-12 | 广州华多网络科技有限公司 | Training method, image-recognizing method and the device of neural network model |
| CN110942106A (en)* | 2019-12-13 | 2020-03-31 | 东华大学 | A pooled convolutional neural network image classification method based on square mean |
| CN111723738A (en)* | 2020-06-19 | 2020-09-29 | 安徽工业大学 | A method and system for classifying microscopic images of coal rock crustoid based on transfer learning |
| CN111860064A (en)* | 2019-04-30 | 2020-10-30 | 杭州海康威视数字技术股份有限公司 | Target detection method, device and equipment based on video and storage medium |
| CN112446860A (en)* | 2020-11-23 | 2021-03-05 | 中山大学中山眼科中心 | Automatic screening method for diabetic macular edema based on transfer learning |
| CN113034475A (en)* | 2021-03-30 | 2021-06-25 | 浙江工业大学 | Finger OCT (optical coherence tomography) volume data denoising method based on lightweight three-dimensional convolutional neural network |
| CN114037660A (en)* | 2021-10-26 | 2022-02-11 | 国药集团基因科技有限公司 | OCT retinopathy image recognition method and system |
| CN114821205A (en)* | 2022-06-30 | 2022-07-29 | 澄影科技(北京)有限公司 | A kind of image processing method, device and equipment based on multi-dimensional feature |
| CN115219810A (en)* | 2022-05-18 | 2022-10-21 | 四川大学 | Line trip prediction method based on lightning positioning system |
| CN116188719A (en)* | 2023-04-27 | 2023-05-30 | 中地云智慧科技有限公司 | Fragment removing method for live-action three-dimensional data |
| CN119399546A (en)* | 2024-11-06 | 2025-02-07 | 江苏富翰医疗产业发展有限公司 | Fundus image classification method and system |
Families Citing this family (26)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107437092B (en)* | 2017-06-28 | 2019-11-15 | 苏州比格威医疗科技有限公司 | Classification of retinal OCT images based on three-dimensional convolutional neural network |
| CN107945176B (en)* | 2017-12-15 | 2021-05-11 | 西安中科微光影像技术有限公司 | A Color IVOCT Imaging Method |
| CN108510004B (en)* | 2018-04-04 | 2022-04-08 | 深圳大学 | A cell classification method and system based on deep residual network |
| US10685265B2 (en)* | 2018-04-11 | 2020-06-16 | International Business Machines Corporation | Cognitive analysis and classification of apparel images |
| CN111656357B (en)* | 2018-04-17 | 2024-05-10 | 深圳华大生命科学研究院 | Modeling method, device and system for ophthalmic disease classification model |
| CN110415252B (en)* | 2018-04-26 | 2022-08-05 | 北京连心医疗科技有限公司 | CNN-based periocular organ segmentation method, CNN-based periocular organ segmentation equipment and CNN-based periocular organ segmentation storage medium |
| US11538083B2 (en) | 2018-05-17 | 2022-12-27 | International Business Machines Corporation | Cognitive fashion product recommendation system, computer program product, and method |
| CN109325398B (en)* | 2018-06-30 | 2020-10-09 | 东南大学 | Human face attribute analysis method based on transfer learning |
| JP7229881B2 (en)* | 2018-08-14 | 2023-02-28 | キヤノン株式会社 | MEDICAL IMAGE PROCESSING APPARATUS, TRAINED MODEL, MEDICAL IMAGE PROCESSING METHOD AND PROGRAM |
| CN109376767B (en)* | 2018-09-20 | 2021-07-13 | 中国科学技术大学 | Classification of retinal OCT images based on deep learning |
| CN109583297B (en)* | 2018-10-25 | 2020-10-02 | 清华大学 | Retina OCT volume data recognition method and device |
| CN109726743B (en)* | 2018-12-12 | 2023-03-24 | 苏州大学 | Retina OCT image classification method based on three-dimensional convolutional neural network |
| US10963757B2 (en) | 2018-12-14 | 2021-03-30 | Industrial Technology Research Institute | Neural network model fusion method and electronic device using the same |
| CN109670489B (en)* | 2019-02-18 | 2023-06-27 | 广州视源电子科技股份有限公司 | Weakly supervised early age-related macular degeneration classification method based on multi-instance learning |
| CN110147715A (en)* | 2019-04-01 | 2019-08-20 | 江西比格威医疗科技有限公司 | A kind of retina OCT image Bruch film angle of release automatic testing method |
| CN110110600B (en)* | 2019-04-04 | 2024-05-24 | 平安科技（深圳）有限公司 | Eye OCT image focus identification method, device and storage medium |
| CN110009626A (en)* | 2019-04-11 | 2019-07-12 | 北京百度网讯科技有限公司 | Method and apparatus for generating images |
| CN110427954A (en)* | 2019-07-26 | 2019-11-08 | 中国科学院自动化研究所 | The image group feature extracting method of multizone based on tumor imaging |
| CN110415244A (en)* | 2019-08-02 | 2019-11-05 | 陈晋音 | Judge the method and its model training method, device of skeletal status |
| CN110910371A (en)* | 2019-11-22 | 2020-03-24 | 北京理工大学 | Automatic classification method and device for liver tumors based on physiological indicators and image fusion |
| CN111144296B (en)* | 2019-12-26 | 2023-04-18 | 湖南大学 | Retina fundus picture classification method based on improved CNN model |
| CN111402217B (en)* | 2020-03-10 | 2023-10-31 | 广州视源电子科技股份有限公司 | Image grading method, device, equipment and storage medium |
| CN114596630A (en)* | 2022-02-17 | 2022-06-07 | 复旦大学 | Identification method for goods taken from intelligent unmanned supermarket |
| CN115170503B (en)* | 2022-07-01 | 2023-12-19 | 上海市第一人民医院 | Fundus image visual field classification method and device based on decision rule and deep neural network |
| CN117237711B (en)* | 2023-09-11 | 2024-12-06 | 杭州电子科技大学 | A dual-modal fundus image classification method based on adversarial learning |
| CN120352923B (en)* | 2025-06-23 | 2025-09-23 | 中铁科学研究院集团有限公司 | A digital detection method for strong earthquake faults using convolutional neural networks |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105825509A (en)* | 2016-03-17 | 2016-08-03 | 电子科技大学 | Cerebral vessel segmentation method based on 3D convolutional neural network |
| CN106875386A (en)* | 2017-02-13 | 2017-06-20 | 苏州江奥光电科技有限公司 | A kind of method for carrying out dental health detection automatically using deep learning |
| CN107437092A (en)* | 2017-06-28 | 2017-12-05 | 苏州比格威医疗科技有限公司 | The sorting algorithm of retina OCT image based on Three dimensional convolution neutral net |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104281858B (en)* | 2014-09-15 | 2018-07-10 | 中安消技术有限公司 | Three dimensional convolution neural network training method, video accident detection method and device |
| CN105975931B (en)* | 2016-05-04 | 2019-06-14 | 浙江大学 | A Convolutional Neural Network Face Recognition Method Based on Multi-scale Pooling |
| CN106326939A (en)* | 2016-08-31 | 2017-01-11 | 深圳市诺比邻科技有限公司 | Parameter optimization method and system of convolutional neural network |
- 2017
- 2017-06-28CNCN201710506132.5Apatent/CN107437092B/enactiveActive
- 2018
- 2018-05-30WOPCT/CN2018/089072patent/WO2019001209A1/ennot_activeCeased
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105825509A (en)* | 2016-03-17 | 2016-08-03 | 电子科技大学 | Cerebral vessel segmentation method based on 3D convolutional neural network |
| CN106875386A (en)* | 2017-02-13 | 2017-06-20 | 苏州江奥光电科技有限公司 | A kind of method for carrying out dental health detection automatically using deep learning |
| CN107437092A (en)* | 2017-06-28 | 2017-12-05 | 苏州比格威医疗科技有限公司 | The sorting algorithm of retina OCT image based on Three dimensional convolution neutral net |
Cited By (22)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108395986A (en)* | 2018-03-23 | 2018-08-14 | 余晖 | Human papilloma virus automatic parting direction detection device based on deep learning |
| CN109816714A (en)* | 2019-01-15 | 2019-05-28 | 西北大学 | A point cloud object type recognition method based on 3D convolutional neural network |
| CN111860064A (en)* | 2019-04-30 | 2020-10-30 | 杭州海康威视数字技术股份有限公司 | Target detection method, device and equipment based on video and storage medium |
| CN111860064B (en)* | 2019-04-30 | 2023-10-20 | 杭州海康威视数字技术股份有限公司 | Video-based target detection method, device, equipment and storage medium |
| CN110222716B (en)* | 2019-05-08 | 2023-07-25 | 天津大学 | Image classification method based on full-resolution depth convolution neural network |
| CN110222716A (en)* | 2019-05-08 | 2019-09-10 | 天津大学 | Image classification method based on full resolution depth convolutional neural networks |
| CN110443286A (en)* | 2019-07-18 | 2019-11-12 | 广州华多网络科技有限公司 | Training method, image-recognizing method and the device of neural network model |
| CN110443286B (en)* | 2019-07-18 | 2024-06-04 | 广州方硅信息技术有限公司 | Training method of neural network model, image recognition method and device |
| CN110942106A (en)* | 2019-12-13 | 2020-03-31 | 东华大学 | A pooled convolutional neural network image classification method based on square mean |
| CN110942106B (en)* | 2019-12-13 | 2023-11-07 | 东华大学 | A pooled convolutional neural network image classification method based on squared average |
| CN111723738A (en)* | 2020-06-19 | 2020-09-29 | 安徽工业大学 | A method and system for classifying microscopic images of coal rock crustoid based on transfer learning |
| CN112446860B (en)* | 2020-11-23 | 2024-04-16 | 中山大学中山眼科中心 | Automatic screening method for diabetic macular edema based on transfer learning |
| CN112446860A (en)* | 2020-11-23 | 2021-03-05 | 中山大学中山眼科中心 | Automatic screening method for diabetic macular edema based on transfer learning |
| CN113034475A (en)* | 2021-03-30 | 2021-06-25 | 浙江工业大学 | Finger OCT (optical coherence tomography) volume data denoising method based on lightweight three-dimensional convolutional neural network |
| CN113034475B (en)* | 2021-03-30 | 2024-04-19 | 浙江工业大学 | Finger OCT volume data denoising method based on lightweight 3D convolutional neural network |
| CN114037660A (en)* | 2021-10-26 | 2022-02-11 | 国药集团基因科技有限公司 | OCT retinopathy image recognition method and system |
| CN115219810A (en)* | 2022-05-18 | 2022-10-21 | 四川大学 | Line trip prediction method based on lightning positioning system |
| CN115219810B (en)* | 2022-05-18 | 2023-06-20 | 四川大学 | A Line Trip Prediction Method Based on Lightning Location System |
| CN114821205A (en)* | 2022-06-30 | 2022-07-29 | 澄影科技(北京)有限公司 | A kind of image processing method, device and equipment based on multi-dimensional feature |
| CN116188719A (en)* | 2023-04-27 | 2023-05-30 | 中地云智慧科技有限公司 | Fragment removing method for live-action three-dimensional data |
| CN116188719B (en)* | 2023-04-27 | 2023-11-17 | 中地云智慧科技有限公司 | Fragment removing method for live-action three-dimensional data |
| CN119399546A (en)* | 2024-11-06 | 2025-02-07 | 江苏富翰医疗产业发展有限公司 | Fundus image classification method and system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107437092B (en) | 2019-11-15 |
| CN107437092A (en) | 2017-12-05 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| WO2019001209A1 (en) | Classification algorithm for retinal oct image based on three-dimensional convolutional neural network | |
| Tabassum et al. | CDED-Net: Joint segmentation of optic disc and optic cup for glaucoma screening | |
| Li et al. | A large-scale database and a CNN model for attention-based glaucoma detection | |
| Sevastopolsky | Optic disc and cup segmentation methods for glaucoma detection with modification of U-Net convolutional neural network | |
| Abdullah et al. | A review on glaucoma disease detection using computerized techniques | |
| CN109376636B (en) | Capsule network-based eye fundus retina image classification method | |
| Pachade et al. | NENet: Nested EfficientNet and adversarial learning for joint optic disc and cup segmentation | |
| CN109376767A (en) | Classification of retinal OCT images based on deep learning | |
| CN114287878A (en) | Diabetic retinopathy focus image identification method based on attention model | |
| CN109300121A (en) | Method and system for constructing a diagnostic model of cardiovascular disease and the diagnostic model | |
| CN106920227A (en) | Based on the Segmentation Method of Retinal Blood Vessels that deep learning is combined with conventional method | |
| CN112435281B (en) | A method and system for multispectral fundus image analysis based on adversarial learning | |
| CN110837803A (en) | Diabetic retinopathy grading method based on depth map network | |
| JP2019192215A (en) | 3d quantitative analysis of retinal layers with deep learning | |
| CN111833334A (en) | A method of fundus image feature processing and analysis based on twin network architecture | |
| CN108764342B (en) | A Semantic Segmentation Method for Optic Disc and Optic Cup in Fundus Map | |
| CN115004222A (en) | Neural network processing of OCT data to generate predictions of geographic atrophic growth rates | |
| CN109671049A (en) | A kind of medical image processing method, system, equipment, storage medium | |
| Aydin et al. | Domain modelling for a lightweight convolutional network focused on automated exudate detection in retinal fundus images | |
| CN110599480A (en) | Multi-source input fundus image classification method and device | |
| Al-Bander et al. | A novel choroid segmentation method for retinal diagnosis using deep learning | |
| CN117636449A (en) | A fundus image classification method | |
| Tian et al. | Learning discriminative representations for fine-grained diabetic retinopathy grading | |
| Pappu et al. | EANet: Multiscale autoencoder based edge attention network for fluid segmentation from SD‐OCT images | |
| CN108665474B (en) | A B-COSFIRE-based method for retinal blood vessel segmentation in fundus images |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| 121 | Ep: the epo has been informed by wipo that ep was designated in this application | Ref document number:18825061 Country of ref document:EP Kind code of ref document:A1 | |
| NENP | Non-entry into the national phase | Ref country code:DE | |
| 122 | Ep: pct application non-entry in european phase | Ref document number:18825061 Country of ref document:EP Kind code of ref document:A1 | |
| 32PN | Ep: public notification in the ep bulletin as address of the adressee cannot be established | Free format text:NOTING OF LOSS OF RIGHTS (EPO FORM 1205A DATED 18.02.2020) | |
| 122 | Ep: pct application non-entry in european phase | Ref document number:18825061 Country of ref document:EP Kind code of ref document:A1 |