CN111047589A - An attention-enhanced brain tumor-assisted intelligent detection and recognition method - Google Patents

Abstract

Translated fromChinese

本发明实现了一套注意力增强的脑肿瘤辅助智能检测识别方法，技术方案在U‑Net模型的基础上进行改进，提出使用对分割任务的训练作为分类任务的注意力增强机制，通过对分割任务、病灶区域和边缘信息的关注提高分类任务的准确率，并通过多任务的损失衡量和训练方法，同时对分割任务与分类任务进行优化，达到分割任务与分类任务两个任务上的预期效果，实现设计目的与应用目标。

The invention realizes a set of attention-enhanced brain tumor auxiliary intelligent detection and identification method, the technical scheme is improved on the basis of the U-Net model, and proposes to use the training of the segmentation task as the attention enhancement mechanism of the classification task. The focus on tasks, lesion areas and edge information improves the accuracy of classification tasks, and through multi-task loss measurement and training methods, the segmentation task and the classification task are optimized at the same time to achieve the expected results of the segmentation task and the classification task. , to achieve design goals and application goals.

Description

Attention-enhanced brain tumor auxiliary intelligent detection and identification method

Technical Field

The invention relates to the field of image processing, in particular to an attention-enhanced brain tumor auxiliary intelligent detection and identification method in the fields of medical imaging and computer-aided diagnosis.

Background

Tumors that grow in the cranium are collectively referred to as brain tumors, and refer to tumors of the nervous system that occur in the cranial cavity, including tumors that originate in the neuroepithelium, peripheral nerves, meninges, and germ cells, tumors of lymphoid and hematopoietic tissues, craniopharyngiomas and granulocytic tumors in the sphenoid saddle region, and metastatic tumors. Tumors arise from the parenchyma of the brain and are called primary intracranial tumors, and metastasis from malignant tumors of other visceral tissues of the body to the cranium is called secondary intracranial tumors. Intracranial tumors can occur at any age, most commonly between 20 and 50 years of age. With the development of neuroimaging techniques and functional examination techniques in recent years, auxiliary examination has become a main means for diagnosing intracranial tumors.

Brain glioma is the most common primary intracranial malignant tumor, accounting for over 75%. Brain glioma is divided into localized glioma and diffuse glioma, which can be divided into WHO I-IV grade according to the malignancy degree of tumor, and the malignancy degree is increased along with the increase of grade. Brain glioma can be divided into low-grade glioma (LGG, WHO I-II grade) and high-grade glioma (HGG, WHO III-IV grade), and can be divided into different subtypes according to gene mutation, chromosome change and the like, and the treatment modes and prognosis of the brain glioma with different grades and different gene mutations are different. Therefore, if the tumor region and the tumor grade can be accurately segmented and judged before the surgical treatment, the method is helpful for guiding the selection of the treatment scheme and the surgical resection region, and has important values for improving the treatment effect and the prognosis of the patient.

Magnetic Resonance Imaging (MRI) is a medical imaging technique that utilizes the hydrogen nuclei, i.e., hydrogen protons, of the human body in a strong external magnetic field to generate magnetic resonance under the action of specific radio frequency pulses. MRI is the main imaging examination technique for various intracranial diseases, can be used as the first examination method for some diseases, and is also an important supplement for CT examination. The MRI examination has the advantages of high tissue resolution, multiple sequences, multiple parameters, multiple directions, multiple kinds of fMRI examination and the like, and can more sensitively discover the lesion and display the lesion characteristics, thereby being beneficial to early detection and accurate diagnosis of the disease. Common MRI images for brain glioma diagnosis include images of three planes of axial (Axis), Sagittal (Sagittal) and Coronal (Coronal) planes, and T1, T1 enhanced, T2 and T2 water suppression four modalities, and the location, range and grade of the tumor are clinically and frequently determined in combination with information of the three planes and the four modalities. However, due to the diversity of the appearance and shape of brain tumors, the separation of brain tumors in multi-modality MRI images is one of the most challenging and difficult persons in medical image processing, and the classification of brain gliomas based on non-invasive examination such as brain MRI, and even the judgment and classification of genotypes are the research directions of great clinical attention.

The localized glioma is mostly seen in children, adults are relatively few, most patients can be cured by operation, the malignancy degree is low, and the research focus of the patent is not taken as the research focus of the patent.

The application of the current deep learning on brain MRI images, particularly other related patents, is concentrated in the field of brain tumor segmentation, and the fields directly related to diagnosis and treatment, such as classification and grading of brain tumors, are less involved, which is more clinically concerned and difficult to achieve by the current human eyes in noninvasive image examination. The method for obtaining a better effect on a brain tumor segmentation task mostly adopts U-Net as a basic frame, and is improved on the basis, the technical scheme of the application also takes U-Net as a basis, uses a three-dimensional model to operate a three-dimensional image, optimizes the three-dimensional image according to the latest progress of current machine learning and deep learning in the computer vision field on the basis of the original U-Net to achieve a better effect, focuses more on a tumor grading and classification diagnosis layer compared with other works, and focuses on an abnormal area in a brain MRI image to realize a tumor diagnosis task by taking the classification task as an attention enhancement mode.

Disclosure of Invention

At present, most of the applications of U-Net to MRI image processing of brain tumors are image segmentation tasks, the applications only segment the region of the tumor, and do not further use the segmented images to obtain valuable information at the medical level, which is a problem that in the process of communicating with clinicians, the clinician pays more attention to the images and needs to research heavily.

In order to achieve the purpose, the invention adopts the following technical scheme:

an attention-enhanced brain tumor auxiliary intelligent detection and identification method is characterized by comprising the following steps: comprises the following steps of;

the method comprises the following steps: establishing a three-dimensional convolution network model of a multitask neural network based on U-Net and suitable for segmentation and diagnosis of a brain glioma lesion region in a brain MRI image;

step two: a multi-task joint training objective;

step three: measuring the loss of multiple tasks and optimizing the result;

step four: and (4) model training, result evaluation and output.

The step of establishing a three-dimensional convolution network model of a multitask neural network based on U-Net and suitable for segmentation and diagnosis of a brain glioma lesion region in a brain MRI image comprises the following steps:

building a three-dimensional model of an attention area based on a tumor area as the model based on an original U-Net network by using a three-dimensional convolution processing method, wherein the framework of the model comprises a down-sampling data path and an up-sampling data path, each layer on the down-sampling path comprises two convolution layers of 3 x 3, and each convolution layer adopts a dropout mode to prevent overfitting and uses an activation function of a ReLU. After two convolutional layers, using a maximum pooling layer with a step size of 2 and a size of 3 × 3 × 3 to perform downsampling operation; the up-sampling operation is carried out between two layers of the up-sampling data path in a deconvolution mode, the features after up-sampling are spliced with the features of the left down-sampling layer, the spliced features are subjected to convolution operation twice as same as that of the left up-sampling layer, after the up-sampling path finally obtains the features fused with the deep layer and the shallow layer information, convolution with the convolution kernel size of 3 x 3 is carried out twice, and after the features after convolution are obtained, the model is divided into two branches: one continues to carry out convolution operation once, the output channel is the same as the classification in the semantic segmentation result, and then the output result containing the background, the edema, the tumor parenchyma, the necrosis and the enhancement kernel is obtained through softmax calculation; after the other branch is subjected to convolution operation for one time, Global Average Pooling (Global Average Pooling) is used, then two full-connection layers are followed, and the output of the second full-connection layer is the same as the classification number of pathological diagnosis;

and then traversing all brain MRI images imported from a case, counting average value and variance information, reserving the brain MRI images for use in a standardized operation in a training and predicting process, receiving input brain MRI image sequences of four modes of T1, T1 enhancement, T2 and T2-Flair, taking each mode as a channel, splicing all slice images in the scanning sequence and a segmentation result labeling mask respectively to form a three-dimensional picture and a labeling sequence, binding the two sequences and a diagnosis result corresponding to the sequences to be used as a sample, and processing all the images.

The multi-task joint training target step comprises the following steps:

on the full convolution model, a classification branch is added after shallow information and deep information are fused, a single task segmentation and classification model is used, and semantic segmentation and classification results are obtained simultaneously, so that a segmentation task and a classification task of brain tumors can be executed simultaneously and shallow features are shared;

after the features of the deep layer information and the shallow layer information are fused, convolution with convolution kernel size of 3 multiplied by 3 is carried out twice, and after the features after convolution are obtained, the model is divided into two branches: one continues to carry out convolution operation once, the output channel is the same as the classification in the semantic segmentation result, and then the output result containing the background, the edema, the tumor parenchyma, the necrosis and the enhancement kernel is obtained through softmax calculation; the other branch was subjected to a convolution operation, followed by Global Average Pooling (Global Average Pooling), followed by two fully-connected layers, the output of the second fully-connected layer being the same as the pathological diagnosis classification number, including the following categories: oligodendroglioma, anaplastic oligodendroglioma, astrocytoma, anaplastic astrocytoma, glioblastoma.

The measuring the loss of the multiple tasks, and the optimizing the result step uses a loss function of the multiple task combination to measure the segmentation result and the classification result and optimize the segmentation and classification result, wherein:

the loss function of the image segmentation model adopts a Dice loss function;

the loss function of the tumor classification module selects a cross-entropy function.

The model training and result evaluation and output step comprises the following steps:

a model training step, namely setting a Dice pass of an image segmentation model or a cross entropy loss function of a tumor classification model to carry out back propagation for iterative training, and carrying out back propagation for iterative training after combining a pass value of the image segmentation model and a pass value of the tumor classification model in a certain proportion;

after the training effect is converged and a more ideal result is obtained, evaluating the effect on a training set;

and inputting a new case image to the evaluated model, and outputting a detection and identification result.

Compared with the prior art, the invention has the advantages that:

the design scheme fully utilizes the segmented information, and expands the classification task module aiming at the medical information hidden in the segmented image, thereby playing the roles of providing suggestions for an auxiliary medical system of a doctor in the diagnosis process, improving the diagnosis capability of a medical institution on related diseases, and feeding pathological research on the brain tumor in the medical field in the aspects of classification, classification and the like of the brain tumor.

At present, most of U-Net is applied to brain tumor MRI image processing to perform image segmentation tasks, the applications only segment the region of a tumor, valuable medical information is not obtained by further using the segmented image, and segmentation and diagnosis results of a focus region are obtained simultaneously through processing and analyzing the image.

Drawings

FIG. 1 is a design framework of a brain tumor auxiliary detection and identification system;

FIG. 2 is a multi-task learning model of brain tumor segmentation and classification diagnosis tasks;

Detailed Description

Referring to the attached drawings 1-2 in the specification, the invention provides a method for analyzing and processing a three-dimensional brain nuclear magnetic resonance image by combining a medical image with a deep learning and computer vision method and using a computer vision analysis method, and carries out segmentation of a glioma lesion region on the brain nuclear magnetic resonance image and a classification diagnosis task based on the image. Aiming at the problems that the data volume of a medical image data set is small, the category is unbalanced and serious, and the existing method focuses on the segmentation of a focus region and omits a classification diagnosis task, an improved 3D U-Net-based convolutional neural network is provided, classification diagnosis branches are added, and segmentation and classification results are obtained simultaneously in a multi-task combined training mode. FIG. 1 is an algorithm design flow proposed by the present invention, which first preprocesses an MRI image, a segmentation result of an artificial label corresponding to the MRI image, and classification diagnostic information obtained from pathological information to obtain a three-dimensional image sequence including four modalities, divides the processed image into a training set and a test set in proportion, trains on the training set, optimizes the performance of a model on two tasks of segmentation and classification, and finally evaluates the effect on the training set after the training effect is converged and a more ideal result is obtained.

Before a model is established, brain MRI image data used by a user directly comes from a medical record system of a hospital, image preprocessing operations of denoising, brightness contrast adjustment and the like for enhancing visibility are completed, image data are labeled manually, different parts of a tumor, including four parts of edema, tumor parenchyma, enhanced nucleus and necrosis, are labeled, and diagnosis information of tumor types, stages and the like is obtained from medical record diagnosis of cases and postoperative pathological information.

And traversing all MRI images, counting information such as the average value, the variance and the like, and reserving the information for standardized operation in the training and predicting process. In addition, four modal images at the same slice position of the same scanning sequence are stacked, each mode is used as a channel, then all slice images under the scanning sequence and a segmentation result marking mask (mask) are respectively spliced to form a three-dimensional image and a marking sequence, and then the two sequences and a diagnosis result corresponding to the sequences are bound to form a sample.

After all samples are processed, the samples are divided into a training set and a testing set according to a determined proportion for later use.

And then, building a three-dimensional model by using a three-dimensional convolution processing method based on the original U-Net network.

Fig. 2 shows the model architecture used in the present invention, comprising a left-hand, one-down-sampled data path and a right-hand, one-up-sampled data path. Each layer on the left downsampled path contains two 3 x 3 convolutional layers, and each convolutional layer adopts dropout mode to prevent overfitting, and the activation function of the ReLU is used. After two convolutional layers, using a maximum pooling layer with a step size of 2 and a size of 3 × 3 × 3 to perform downsampling operation; and performing up-sampling operation between two layers of the right data path in a deconvolution mode, splicing the up-sampled features with the features of the left down-sampling layer, and performing convolution operation on the spliced features twice, wherein the convolution operation is the same as that of the left up-sampling layer.

The downsampling layer improves the receptive field of each pixel in the finally obtained deepest layer features by continuously reducing the resolution of the features, obtains a high-level representation with more abstract features, namely deep information, and has higher representation capability for picture classification and pixel class judgment in pictures, but the accuracy of pixel-level classification and the resolution of obtained classification results are greatly reduced due to the loss of the resolution. The middle cross-layer connection is shallow information, and the problem that the resolution ratio of the result is reduced compared with the input result is solved by fusing the middle cross-layer connection with the up-sampling path characteristics on the right side, so that the fused result is better in expression result.

After the characteristics of the fused deep-layer and shallow-layer information are finally obtained from the up-sampling channel, convolution with convolution kernel size of 3 × 3 × 3 is performed twice, and after the characteristics of the convolution are obtained, the model is divided into two branches: one continues to carry out convolution operation once, the output channel is the same as the classification in the semantic segmentation result, and then the output result containing the background, the edema, the tumor parenchyma, the necrosis and the enhancement kernel is obtained through softmax calculation; the other branch was subjected to a convolution operation, followed by Global Average Pooling (Global Average Pooling), followed by two fully-connected layers, the output of the second fully-connected layer being the same as the pathological diagnosis classification number, including the following categories: oligodendroglioma, anaplastic oligodendroglioma, astrocytoma, anaplastic astrocytoma, glioblastoma.

The above process can be regarded as a process of accurately identifying the brain tumor category by combining the background based on the tumor region as the attention region of the model. The process uses a multi-task learning method, so that the segmentation task and the classification task of the brain tumor can be simultaneously executed, and the segmentation task and the classification task can be jointly promoted under the mutual action due to the sharing of the shallow features.

The multi-task learning method is measured by the loss function and the model training method is used for optimizing the performance of the model.

1. Loss function

1) Selection of a loss function for brain tumor segmentation task model:

one challenge in medical image segmentation is the problem of class imbalance in the data, for example in brain tumor MRI images, the proportion of the entire data that is the target object to be segmented is particularly small, resulting in severe class imbalance. In this case, the training is hindered by using the traditional classification cross entropy loss function, and the Dice loss function can effectively deal with the class imbalance problem, so the invention adopts the loss function as the loss function of the segmentation model, which is specifically expressed as follows:

where u is the segmentation result output for the network, v is the segmentation of the labels, and i is the number of pixels in the training block

2) Selection of loss function for brain tumor classification task:

the problem is a classification problem, and a cross entropy function widely used by the classification problem is adopted as a loss function of the module, which is specifically expressed as follows:

where N is the total number of samples, K is the total number of classes, y_i，jIs a tag value, p_i，jIs a predicted value

2. The model training method comprises the following steps:

1) and setting a cross entropy loss function which only allows an image segmentation model or a tumor classification model to carry out back propagation for iterative training, wherein the overall loss function is expressed as:

2) combining the loss value of the image segmentation module and the loss value of the tumor classification model according to a certain proportion, and then carrying out back propagation for iterative training, namely expressing the total loss function as:

Loss₂＝Loss_Dice+αLoss_cross

wherein α is an adjustable scaling factor.

Claims (5)

1. An attention-enhanced brain tumor auxiliary intelligent detection and identification method is characterized by comprising the following steps: comprises the following steps of;

the method comprises the following steps: establishing a three-dimensional convolution network model of a multitask neural network based on U-Net and suitable for segmentation and diagnosis of a brain glioma lesion region in a brain MRI image;

step two: a multi-task joint training objective;

step three: measuring the loss of multiple tasks and optimizing the result;

step four: and (4) model training, result evaluation and output.

2. The method for aided intelligent detection and identification of brain tumors according to claim 1, wherein the method comprises the following steps: the step of establishing a three-dimensional convolution network model of a multitask neural network based on U-Net and suitable for segmentation and diagnosis of a brain glioma lesion region in a brain MRI image comprises the following steps:

building a three-dimensional model of an attention area based on a tumor area as the model based on an original U-Net network by using a three-dimensional convolution processing method, wherein the framework of the model comprises a down-sampling data path and an up-sampling data path, each layer on the down-sampling path comprises two convolution layers of 3 x 3, and each convolution layer adopts a dropout mode to prevent overfitting and uses an activation function of a ReLU. After two convolutional layers, using a maximum pooling layer with a step size of 2 and a size of 3 × 3 × 3 to perform downsampling operation; the up-sampling operation is carried out between two layers of the up-sampling data path in a deconvolution mode, the features after up-sampling are spliced with the features of the left down-sampling layer, the spliced features are subjected to convolution operation twice as same as that of the left up-sampling layer, after the up-sampling path finally obtains the features fused with the deep layer and the shallow layer information, convolution with the convolution kernel size of 3 x 3 is carried out twice, and after the features after convolution are obtained, the model is divided into two branches: one continues to carry out convolution operation once, the output channel is the same as the classification in the semantic segmentation result, and then the output result containing the background, the edema, the tumor parenchyma, the necrosis and the enhancement kernel is obtained through softmax calculation; after the other branch is subjected to convolution operation for one time, Global Average Pooling (Global Average Pooling) is used, then two full-connection layers are followed, and the output of the second full-connection layer is the same as the classification number of pathological diagnosis;

and then traversing all brain MRI images imported from a case, counting average value and variance information, reserving the brain MRI images for use in a standardized operation in a training and predicting process, receiving input brain MRI image sequences of four modes of T1, T1 enhancement, T2 and T2-Flair, taking each mode as a channel, splicing all slice images in the scanning sequence and a segmentation result labeling mask respectively to form a three-dimensional picture and a labeling sequence, binding the two sequences and a diagnosis result corresponding to the sequences to be used as a sample, and processing all the images.

3. The method for aided intelligent detection and identification of brain tumors according to claim 2, wherein the method comprises the following steps: the multi-task joint training target step comprises the following steps:

on the full convolution model, a classification branch is added after shallow information and deep information are fused, a single task segmentation and classification model is used, and semantic segmentation and classification results are obtained simultaneously, so that a segmentation task and a classification task of brain tumors can be executed simultaneously and shallow features are shared;

after the features of the deep layer information and the shallow layer information are fused, convolution with convolution kernel size of 3 multiplied by 3 is carried out twice, and after the features after convolution are obtained, the model is divided into two branches: one continues to carry out convolution operation once, the output channel is the same as the classification in the semantic segmentation result, and then the output result containing the background, the edema, the tumor parenchyma, the necrosis and the enhancement kernel is obtained through softmax calculation; the other branch was subjected to a convolution operation, followed by Global Average Pooling (Global Average Pooling), followed by two fully-connected layers, the output of the second fully-connected layer being the same as the pathological diagnosis classification number, including the following categories: oligodendroglioma, anaplastic oligodendroglioma, astrocytoma, anaplastic astrocytoma, glioblastoma.

4. The method for aided intelligent detection and identification of brain tumors with enhanced attention according to claim 3, wherein the method comprises the following steps: the measuring the loss of the multiple tasks, and the optimizing the result step uses a loss function of the multiple task combination to measure the segmentation result and the classification result and optimize the segmentation and classification result, wherein:

the loss function of the image segmentation model adopts a Dice loss function;

the loss function of the tumor classification module selects a cross-entropy function.

5. The method for aided intelligent detection and identification of brain tumors according to claim 4, wherein the method comprises the following steps: the model training and result evaluation and output step comprises the following steps:

a model training step, namely setting a Dice pass of an image segmentation model or a cross entropy loss function of a tumor classification model to carry out back propagation for iterative training, and carrying out back propagation for iterative training after combining a pass value of the image segmentation model and a pass value of the tumor classification model in a certain proportion;

after the training effect is converged and a more ideal result is obtained, evaluating the effect on a training set;

and inputting a new case image to the evaluated model, and outputting a detection and identification result.

Images