CN116402997A - Focus region determination method, device and storage medium based on fusion attention - Google Patents
Focus region determination method, device and storage medium based on fusion attentionDownload PDFInfo
- Publication number
- CN116402997A CN116402997ACN202310326310.1ACN202310326310ACN116402997ACN 116402997 ACN116402997 ACN 116402997ACN 202310326310 ACN202310326310 ACN 202310326310ACN 116402997 ACN116402997 ACN 116402997A
- Authority
- CN
- China
- Prior art keywords
- image
- attention
- module
- feature
- region
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
- G06V10/267—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion by performing operations on regions, e.g. growing, shrinking or watersheds
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
- G06N3/0455—Auto-encoder networks; Encoder-decoder networks
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/0464—Convolutional networks [CNN, ConvNet]
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/11—Region-based segmentation
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
- G06V10/806—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of extracted features
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20112—Image segmentation details
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30004—Biomedical image processing
- G06T2207/30096—Tumor; Lesion
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02A—TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE
- Y02A90/00—Technologies having an indirect contribution to adaptation to climate change
- Y02A90/10—Information and communication technologies [ICT] supporting adaptation to climate change, e.g. for weather forecasting or climate simulation
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Computation (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Computing Systems (AREA)
- General Health & Medical Sciences (AREA)
- Multimedia (AREA)
- Computational Linguistics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Data Mining & Analysis (AREA)
- Molecular Biology (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Medical Informatics (AREA)
- Databases & Information Systems (AREA)
- Quality & Reliability (AREA)
- Image Processing (AREA)
Abstract
Description
Translated fromChinese技术领域technical field
本申请涉及但不限于图像处理技术领域，尤其涉及一种基于融合注意力的病灶区域确定方法、装置、存储介质。The present application relates to but not limited to the technical field of image processing, and in particular relates to a method, device, and storage medium for determining a lesion area based on fusion attention.
背景技术Background technique
现阶段，放射治疗是治疗鼻咽癌的主要方法。通过获取CT或MRI图像等医学图像，并对医学图像进行准确分割鼻咽癌原发灶，能够很好地辅助放射治疗，为放射治疗提供有效的数据基础。但目前，该任务通常由经验丰富的放射科医师通过逐片手工标注的方式来完成，该方式不仅繁琐、耗时耗力，而且还面临着因操作人员自身经验和专业水平的差异，难以保证病灶区域的准确性。At present, radiation therapy is the main method for the treatment of nasopharyngeal carcinoma. By obtaining medical images such as CT or MRI images, and accurately segmenting the primary tumor of nasopharyngeal carcinoma on the medical images, it can well assist radiation therapy and provide an effective data basis for radiation therapy. However, at present, this task is usually completed by experienced radiologists by manually labeling slices one by one. The accuracy of the lesion area.
发明内容Contents of the invention
本申请实施例提供了一种基于融合注意力的病灶区域确定方法、装置、存储介质，能够有效提高病灶区域的检测精度。The embodiments of the present application provide a method, device, and storage medium for determining a lesion area based on fusion attention, which can effectively improve the detection accuracy of the lesion area.
第一方面，本申请实施例提供了一种基于融合注意力的病灶区域确定方法，包括：In the first aspect, the embodiment of the present application provides a method for determining a lesion area based on fusion attention, including:
获取鼻咽区域图像；Obtain an image of the nasopharynx region;
将所述鼻咽区域图像输入至预先训练好的融合注意力网络模型，所述融合注意力网络模型包括编码器和解码器，所述编码器包括依次连接的卷积模块、FA注意力模块和3D交错稀疏自注意力模块；The nasopharyngeal region image is input to the pre-trained fusion attention network model, the fusion attention network model includes an encoder and a decoder, and the encoder includes a sequentially connected convolution module, FA attention module and 3D interleaved sparse self-attention module;
将所述鼻咽区域图像依次输入至所述卷积模块和所述FA注意力模块进行特征提取，得到第一特征图像；The nasopharynx region image is sequentially input to the convolution module and the FA attention module for feature extraction to obtain a first feature image;
对所述第一特征图像进行下采样处理，并将下采样后的第一特征图像输入至所述3D交错稀疏自注意力模块进行图像处理，得到第二特征图像；Carrying out down-sampling processing on the first feature image, and inputting the down-sampled first feature image to the 3D interleaved sparse self-attention module for image processing to obtain a second feature image;
获取第三特征图像，对所述第二特征图像和所述第三特征图像进行融合计算，得到注意力门系数矩阵，所述第三特征图像为参考网络层的上一层网络层输出的特征图像，所述参考网络层为输出所述第二特征图像对应的网络层；Obtain a third feature image, perform fusion calculation on the second feature image and the third feature image, and obtain an attention gate coefficient matrix, and the third feature image is a feature output by the upper network layer of the reference network layer image, the reference network layer is the network layer corresponding to the output of the second feature image;
将所述第二特征图像输入至所述解码器进行解码操作，并将解码后的第二特征图像与所述注意力门系数矩阵进行融合计算，确定病灶区域分割结果。The second feature image is input to the decoder to perform a decoding operation, and the decoded second feature image is fused with the attention gate coefficient matrix to determine a lesion region segmentation result.
在一些实施例中，所述卷积模块包括卷积层、第一BN层和第一激活函数层；所述将所述鼻咽区域图像依次输入至所述第一卷积模块和所述FA注意力模块进行特征提取，得到第一特征图像，包括：In some embodiments, the convolution module includes a convolution layer, a first BN layer, and a first activation function layer; the input of the nasopharyngeal region image to the first convolution module and the FA in sequence The attention module performs feature extraction to obtain the first feature image, including:
将所述鼻咽区域图像输入至所述第一卷积层进行卷积处理，得到第一中间图像；Inputting the image of the nasopharynx region to the first convolutional layer for convolution processing to obtain a first intermediate image;
将所述第一中间图像输入至所述第一BN层进行图像归一化处理，得到第二中间图像；Inputting the first intermediate image to the first BN layer for image normalization processing to obtain a second intermediate image;
将所述第二中间图像输入至所述第一激活函数层进行非线性变换处理，得到第三中间图像；Inputting the second intermediate image to the first activation function layer for nonlinear transformation processing to obtain a third intermediate image;
将所述第三中间图像输入至所述FA注意力模块进行特征提取，得到所述第一特征图像。Inputting the third intermediate image to the FA attention module for feature extraction to obtain the first feature image.
在一些实施例中，所述将下采样后的第一特征图像输入至所述3D交错稀疏自注意力模块进行图像处理，得到第二特征图像，包括：In some embodiments, the first feature image after the downsampling is input to the 3D interleaved sparse self-attention module for image processing to obtain the second feature image, including:
将所述将下采样后的第一特征图像进行图像划分处理，得到预设数量的第一图像子集，各个所述第一图像子集的图像尺寸相等；performing image division processing on the down-sampled first feature image to obtain a preset number of first image subsets, and the image sizes of each of the first image subsets are equal;
依次从各个所述图像子集中获取目标像素点，根据全部的目标像素点构造第二图像子集；Acquiring target pixels from each of the image subsets in sequence, and constructing a second image subset according to all target pixels;
根据所述第二图像子集和预设的自注意力算法计算第一稀疏关联矩阵；calculating a first sparse correlation matrix according to the second image subset and a preset self-attention algorithm;
根据所述第一图像子集和所述自注意力算法计算第二稀疏关联矩阵；calculating a second sparse incidence matrix based on the first subset of images and the self-attention algorithm;
根据所述第一稀疏关联矩阵和所述第二稀疏关联矩阵得到所述第二特征图像。The second feature image is obtained according to the first sparse correlation matrix and the second sparse correlation matrix.
在一些实施例中，所述解码器包括上采样模块，所述上采样模块包括反卷积层、第二BN化层和第二激活函数层，所述将所述第二特征图像输入至所述解码器进行解码操作，包括：In some embodiments, the decoder includes an upsampling module, the upsampling module includes a deconvolution layer, a second BN layer, and a second activation function layer, and the input of the second feature image to the The decoder performs the decoding operation, including:
将所述第二特征图像输入至所述反卷积层进行卷积处理，得到第四中间图像；The second feature image is input to the deconvolution layer for convolution processing to obtain a fourth intermediate image;
将所述第四中间图像输入至所述第二BN层进行图像归一化处理，得到第五中间图像；Inputting the fourth intermediate image to the second BN layer for image normalization processing to obtain a fifth intermediate image;
将所述第五中间图像输入至所述第二激活函数层进行非线性变换处理，得到所述解码后的第二特征图像。The fifth intermediate image is input to the second activation function layer for nonlinear transformation processing to obtain the decoded second feature image.
在一些实施例中，所述FA注意力模块包括通道注意映射模块和空间注意映射模块，所述将所述第三中间图像输入至所述FA注意力模块进行特征提取，得到所述第一特征图像，根据以下公式得到：In some embodiments, the FA attention module includes a channel attention mapping module and a spatial attention mapping module, and the third intermediate image is input to the FA attention module for feature extraction to obtain the first feature image, according to the following formula:
其中，F′为所述第一特征图像，F∈RC×H×W为所述第三中间图像，M(F)为所述FA注意力模块，的表达式为：Wherein, F' is the first feature image, F∈RC×H×W is the third intermediate image, M(F) is the FA attention module, and the expression is:
M(F)＝Sigmoid(MC(F)+MS(F))；M(F)=Sigmoid(MC (F)+MS (F));
其中，Mc(F)为所述通道注意映射模块，Ms(F)为所述空间注意映射模块。Wherein, Mc (F) is the channel attention mapping module, and Ms (F) is the spatial attention mapping module.
在一些实施例中，在所述将所述鼻咽区域图像输入至预先训练好的融合注意力网络模型之前，所述方法还包括：In some embodiments, before the input of the nasopharyngeal region image into the pre-trained fusion attention network model, the method also includes:
根据预设规则对所述鼻咽区域图像进行图像预处理，得到预处理后的目标鼻咽区域图像。Image preprocessing is performed on the nasopharyngeal region image according to a preset rule to obtain a preprocessed target nasopharyngeal region image.
在一些实施例中，所述鼻咽区域图像为CT图像，所述根据预设规则对所述鼻咽区域图像进行图像预处理，得到预处理后的目标鼻咽区域图像，包括：In some embodiments, the image of the nasopharynx region is a CT image, and the image preprocessing is performed on the image of the nasopharynx region according to preset rules to obtain a preprocessed image of the target nasopharynx region, including:
确定所述鼻咽区域图像的Hu值，将所述Hu值截断至预设范围内，得到中间鼻咽区域图像；determining the Hu value of the nasopharyngeal region image, and truncating the Hu value to a preset range to obtain an intermediate nasopharyngeal region image;
对所述中间鼻咽区域图像进行归一化处理，得到所述目标鼻咽区域图像。Perform normalization processing on the middle nasopharyngeal region image to obtain the target nasopharyngeal region image.
第二方面，本申请实施例提供了一种基于融合注意力的病灶区域确定装置，包括：In the second aspect, the embodiment of the present application provides a device for determining a lesion area based on fusion attention, including:
图像获取模块，所述图像获取模块用于获取鼻咽区域图像；An image acquisition module, the image acquisition module is used to acquire nasopharyngeal region images;
第一图像处理模块，所述第一图像处理模块用于将所述鼻咽区域图像输入至预先训练好的融合注意力网络模型，所述融合注意力网络模型包括编码器和解码器，所述编码器包括依次连接的卷积模块、FA注意力模块和3D交错稀疏自注意力模块；A first image processing module, the first image processing module is used to input the image of the nasopharynx region into a pre-trained fusion attention network model, the fusion attention network model includes an encoder and a decoder, the The encoder includes sequentially connected convolution modules, FA attention modules and 3D interleaved sparse self-attention modules;
第二图像处理模块，所述第二图像处理模块用于将所述新的鼻咽区域图像依次输入至所述卷积模块和所述FA注意力模块进行特征提取，得到第一特征图像；A second image processing module, the second image processing module is used to sequentially input the new nasopharyngeal region image to the convolution module and the FA attention module for feature extraction to obtain a first feature image;
第三图像处理模块，所述第三图像处理模块用于对所述第一特征图像进行下采样处理，并将下采样后的第一特征图像输入至所述3D交错稀疏自注意力模块进行图像处理，得到第二特征图像；The third image processing module, the third image processing module is used to perform down-sampling processing on the first feature image, and input the down-sampled first feature image to the 3D interleaved sparse self-attention module for image processing Processing to obtain the second feature image;
第四图像处理模块，所述第四图像处理模块用于获取第三特征图像，对所述第二特征图像和所述第三特征图像进行融合计算，得到注意力门系数矩阵，所述第三特征图像为参考网络层的上一层网络层输出的特征图像，所述参考网络层为输出所述第二特征图像对应的网络层；A fourth image processing module, the fourth image processing module is used to acquire a third feature image, perform fusion calculation on the second feature image and the third feature image to obtain an attention gate coefficient matrix, and the third The feature image is the feature image output by the upper network layer of the reference network layer, and the reference network layer is the network layer corresponding to the output of the second feature image;
病灶区域分割结果确定模块，所述病灶区域分割结果确定模块用于将所述第二特征图像输入至所述解码器进行解码操作，并将解码后的第二特征图像与所述注意力门系数矩阵进行融合计算，确定病灶区域分割结果。A lesion area segmentation result determination module, the lesion area segmentation result determination module is used to input the second feature image to the decoder for decoding operation, and combine the decoded second feature image with the attention gate coefficient The matrix is fused and calculated to determine the segmentation result of the lesion area.
第三方面，本申请实施例提供了一种基于融合注意力的病灶区域确定装置，包括：存储器、处理器及存储在存储器上并可在处理器上运行的计算机程序，所述处理器执行所述计算机程序时实现如第一方面所述的基于融合注意力的病灶区域确定方法。In the third aspect, the embodiment of the present application provides a device for determining a lesion area based on fusion attention, including: a memory, a processor, and a computer program stored in the memory and operable on the processor, and the processor executes the When the computer program is used, the method for determining a lesion area based on fusion attention as described in the first aspect is realized.
第四方面，本申请实施例还提供了一种计算机可读存储介质，存储有计算机可执行指令，所述计算机可执行指令用于执行如第一方面所述的基于融合注意力的病灶区域确定方法。In the fourth aspect, the embodiment of the present application also provides a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions are used to perform the fusion attention-based lesion region determination as described in the first aspect method.
本申请实施例提供了一种基于融合注意力的病灶区域确定方法、装置、存储介质，方法包括：获取鼻咽区域图像；将所述鼻咽区域图像输入至预先训练好的融合注意力网络模型，所述融合注意力网络模型包括编码器和解码器，所述编码器包括依次连接的卷积模块、FA注意力模块和3D交错稀疏自注意力模块；将所述鼻咽区域图像依次输入至所述卷积模块和所述FA注意力模块进行特征提取，得到第一特征图像；对所述第一特征图像进行下采样处理，并将下采样后的第一特征图像输入至所述3D交错稀疏自注意力模块进行图像处理，得到第二特征图像；获取第三特征图像，对所述第二特征图像和所述第三特征图像进行融合计算，得到注意力门系数矩阵，所述第三特征图像为参考网络层的上一层网络层输出的特征图像，所述参考网络层为输出所述第二特征图像对应的网络层；将所述第二特征图像输入至所述解码器进行解码操作，并将解码后的第二特征图像与所述注意力门系数矩阵进行融合计算，确定病灶区域分割结果。本申请实施例能够基于融合注意力网络模型对鼻咽区域图像进行特征提取以及分割鼻咽病灶区域，相较于相关技术中通过医师人工判断划分鼻咽病灶区域的方法，本申请实施例的技术方案能够有效提高病灶区域的检测精度。An embodiment of the present application provides a method, device, and storage medium for determining a lesion area based on fusion attention. The method includes: acquiring an image of the nasopharynx region; inputting the image of the nasopharynx region into a pre-trained fusion attention network model , the fusion attention network model includes an encoder and a decoder, and the encoder includes sequentially connected convolution modules, FA attention modules, and 3D interleaved sparse self-attention modules; the nasopharyngeal region images are sequentially input to The convolution module and the FA attention module perform feature extraction to obtain a first feature image; perform downsampling processing on the first feature image, and input the downsampled first feature image to the 3D interleaved The sparse self-attention module performs image processing to obtain a second feature image; obtains a third feature image, performs fusion calculation on the second feature image and the third feature image, and obtains an attention gate coefficient matrix, and the third feature image The feature image is the feature image output by the upper network layer of the reference network layer, and the reference network layer is the network layer corresponding to the output of the second feature image; the second feature image is input to the decoder for decoding and perform fusion calculation on the decoded second feature image and the attention gate coefficient matrix to determine the lesion region segmentation result. The embodiment of the present application can perform feature extraction and segment the nasopharyngeal lesion area based on the fused attention network model. The scheme can effectively improve the detection accuracy of the lesion area.
附图说明Description of drawings
图1是本申请一个实施例提供的基于融合注意力的病灶区域确定方法的步骤流程图；Fig. 1 is a flow chart of the steps of a method for determining a lesion area based on fusion attention provided by an embodiment of the present application;
图2是本申请另一个实施例提供的得到第一特征图像的步骤流程图；Fig. 2 is a flow chart of steps for obtaining the first characteristic image provided by another embodiment of the present application;
图3是本申请另一个实施例提供的得到第二特征图像的步骤流程图；FIG. 3 is a flow chart of steps for obtaining a second characteristic image provided by another embodiment of the present application;
图4是本申请另一个实施例提供的将第二特征图像输入至解码器进行解码操作的步骤流程图；Fig. 4 is a flow chart of steps for inputting the second feature image to the decoder for decoding operation provided by another embodiment of the present application;
图5是本申请另一个实施例提供的对鼻咽区域图像进行图像预处理的步骤流程图；Fig. 5 is a flow chart of steps for performing image preprocessing on nasopharyngeal region images provided by another embodiment of the present application;
图6是本申请另一个实施例提供的对鼻咽区域图像进行图像预处理的步骤流程图；Fig. 6 is a flow chart of image preprocessing steps for nasopharyngeal region images provided by another embodiment of the present application;
图7是本申请另一个实施例提供的FA注意力模块的示意图；Fig. 7 is a schematic diagram of the FA attention module provided by another embodiment of the present application;
图8是本申请另一个实施例提供的基于融合注意力的病灶区域确定装置的模块示意图；Fig. 8 is a block diagram of an apparatus for determining a lesion area based on fusion attention provided by another embodiment of the present application;
图9是本申请另一个实施例提供的基于融合注意力的病灶区域确定装置的结构图。Fig. 9 is a structural diagram of an apparatus for determining a lesion area based on fusion attention provided by another embodiment of the present application.
具体实施方式Detailed ways
为了使本申请的目的、技术方案及优点更加清楚明白，以下结合附图及实施例，对本申请进行进一步详细说明。应当理解，此处所描述的具体实施例仅用以解释本申请，并不用于限定本申请。In order to make the purpose, technical solution and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present application, not to limit the present application.
可以理解的是，虽然在装置示意图中进行了功能模块划分，在流程图中示出了逻辑顺序，但是在某些情况下，可以以不同于装置中的模块划分，或流程图中的顺序执行所示出或描述的步骤。说明书、权利要求书或上述附图中的术语“第一”、“第二”等是用于区别类似的对象，而不必用于描述特定的顺序或先后次序。It can be understood that although the functional modules are divided in the schematic diagram of the device and the logical order is shown in the flow chart, in some cases, it can be executed in a different order than the division of modules in the device or the sequence in the flow chart. steps shown or described. The terms "first", "second" and the like in the specification, claims or the above-mentioned drawings are used to distinguish similar objects, and not necessarily used to describe a specific order or sequential order.
现阶段，放射治疗是治疗鼻咽癌的主要方法。通过获取CT或MRI图像等医学图像，并对医学图像进行准确分割鼻咽癌原发灶，能够很好地辅助放射治疗，为放射治疗提供有效的数据基础。但目前，该任务通常由经验丰富的放射科医师通过逐片手工标注的方式来完成，该方式不仅繁琐、耗时耗力，而且还面临着因操作人员自身经验和专业水平的差异，难以保证病灶区域的准确性。At present, radiation therapy is the main method for the treatment of nasopharyngeal carcinoma. By obtaining medical images such as CT or MRI images, and accurately segmenting the primary tumor of nasopharyngeal carcinoma on the medical images, it can well assist radiation therapy and provide an effective data basis for radiation therapy. However, at present, this task is usually completed by experienced radiologists by manually labeling slices one by one. The accuracy of the lesion area.
为解决上述存在的问题，本申请实施例提供了一种基于融合注意力的病灶区域确定方法、装置、存储介质，方法包括：获取鼻咽区域图像；将所述鼻咽区域图像输入至预先训练好的融合注意力网络模型，所述融合注意力网络模型包括编码器和解码器，所述编码器包括依次连接的卷积模块、FA注意力模块和3D交错稀疏自注意力模块；将所述鼻咽区域图像依次输入至所述卷积模块和所述FA注意力模块进行特征提取，得到第一特征图像；对所述第一特征图像进行下采样处理，并将下采样后的第一特征图像输入至所述3D交错稀疏自注意力模块进行图像处理，得到第二特征图像；获取第三特征图像，对所述第二特征图像和所述第三特征图像进行融合计算，得到注意力门系数矩阵，所述第三特征图像为参考网络层的上一层网络层输出的特征图像，所述参考网络层为输出所述第二特征图像对应的网络层；将所述第二特征图像输入至所述解码器进行解码操作，并将解码后的第二特征图像与所述注意力门系数矩阵进行融合计算，确定病灶区域分割结果。本申请实施例能够基于融合注意力网络模型对鼻咽区域图像进行特征提取以及分割鼻咽病灶区域，相较于相关技术中通过医师人工判断划分鼻咽病灶区域的方法，本申请实施例的技术方案能够有效提高病灶区域的检测精度。In order to solve the above existing problems, the embodiment of the present application provides a method, device, and storage medium for determining a lesion area based on fusion attention. The method includes: acquiring an image of the nasopharynx area; A good fusion attention network model, the fusion attention network model includes an encoder and a decoder, and the encoder includes sequentially connected convolution modules, FA attention modules and 3D interleaved sparse self-attention modules; the Nasopharyngeal region images are sequentially input to the convolution module and the FA attention module for feature extraction to obtain a first feature image; the first feature image is down-sampled, and the down-sampled first feature The image is input to the 3D interleaved sparse self-attention module for image processing to obtain a second feature image; a third feature image is obtained, and fusion calculation is performed on the second feature image and the third feature image to obtain an attention gate A coefficient matrix, the third feature image is a feature image output by the upper network layer of the reference network layer, and the reference network layer is the network layer corresponding to the output of the second feature image; the second feature image is input To the decoder to perform a decoding operation, and perform fusion calculation on the decoded second feature image and the attention gate coefficient matrix to determine the lesion region segmentation result. The embodiment of the present application can perform feature extraction and segment the nasopharyngeal lesion area based on the fused attention network model. The scheme can effectively improve the detection accuracy of the lesion area.
下面结合附图，对本申请实施例作进一步阐述。The embodiments of the present application will be further described below in conjunction with the accompanying drawings.
如图1所示，图1是本申请一个实施例提供的基于融合注意力的病灶区域确定方法的步骤流程图，本申请实施例提供了一种基于融合注意力的病灶区域确定方法，包括但不限于有以下步骤：As shown in Figure 1, Figure 1 is a flow chart of the steps of a method for determining a lesion area based on fusion attention provided by an embodiment of the present application. An embodiment of the application provides a method for determining a lesion area based on fusion attention, including but Not limited to the following steps:
步骤S110，获取鼻咽区域图像；Step S110, acquiring an image of the nasopharynx region;
步骤S120，将鼻咽区域图像输入至预先训练好的融合注意力网络模型，融合注意力网络模型包括编码器和解码器，编码器包括依次连接的卷积模块、FA注意力模块和3D交错稀疏自注意力模块；Step S120, input the image of the nasopharynx region into the pre-trained fusion attention network model, the fusion attention network model includes an encoder and a decoder, and the encoder includes sequentially connected convolution modules, FA attention modules and 3D interleaved sparse self-attention module;
步骤S130，将鼻咽区域图像依次输入至卷积模块和FA注意力模块进行特征提取，得到第一特征图像；Step S130, inputting the image of the nasopharynx region to the convolution module and the FA attention module in turn for feature extraction to obtain the first feature image;
步骤S140，对第一特征图像进行下采样处理，并将下采样后的第一特征图像输入至3D交错稀疏自注意力模块进行图像处理，得到第二特征图像；Step S140, performing down-sampling processing on the first feature image, and inputting the down-sampled first feature image to the 3D interleaved sparse self-attention module for image processing to obtain a second feature image;
步骤S150，获取第三特征图像，对第二特征图像和第三特征图像进行融合计算，得到注意力门系数矩阵，第三特征图像为参考网络层的上一层网络层输出的特征图像，参考网络层为输出第二特征图像对应的网络层；Step S150, obtain the third characteristic image, perform fusion calculation on the second characteristic image and the third characteristic image, and obtain the attention gate coefficient matrix, the third characteristic image is the characteristic image output by the upper network layer of the reference network layer, refer to The network layer is a network layer corresponding to outputting the second feature image;
步骤S160，将第二特征图像输入至解码器进行解码操作，并将解码后的第二特征图像与注意力门系数矩阵进行融合计算，确定病灶区域分割结果。Step S160, inputting the second feature image to the decoder for decoding operation, performing fusion calculation on the decoded second feature image and the attention gate coefficient matrix, and determining the lesion region segmentation result.
可以理解的是，本实施例提供的病灶区域确定方法包括：获取鼻咽区域图像；将鼻咽区域图像输入至预先训练好的融合注意力网络模型，融合注意力网络模型包括编码器和解码器，编码器包括依次连接的卷积模块、FA注意力模块和3D交错稀疏自注意力模块；将鼻咽区域图像依次输入至卷积模块和FA注意力模块进行特征提取，得到第一特征图像，通过FA注意力模块能够捕捉小目标的鼻咽病灶区域的位置和空间信息；对第一特征图像进行下采样处理，并将下采样后的第一特征图像输入至3D交错稀疏自注意力模块进行图像处理，得到第二特征图像，能够为进一步优化病灶区域的整体分割效果提供有效的数据基础；获取第三特征图像，对第二特征图像和第三特征图像进行融合计算，得到注意力门系数矩阵，实现融合低层和高层语义信息，对低级语义信息做进一步补偿，进一步改善基于特征图像的边缘分割效果，其中，第三特征图像为参考网络层的上一层网络层输出的特征图像，参考网络层为输出第二特征图像对应的网络层；将第二特征图像输入至解码器进行解码操作，并将解码后的第二特征图像与注意力门系数矩阵进行融合计算，确定病灶区域分割结果。本申请基于融合注意力网络模型对鼻咽区域图像进行特征提取以及分割鼻咽病灶区域，相较于相关技术中通过医师人工判断划分鼻咽病灶区域的方法，本申请实施例的技术方案能够有效提高病灶区域的检测精度。It can be understood that the method for determining the lesion area provided by this embodiment includes: acquiring an image of the nasopharynx region; inputting the image of the nasopharynx region into a pre-trained fusion attention network model, the fusion attention network model includes an encoder and a decoder , the encoder includes a sequentially connected convolution module, FA attention module and 3D interleaved sparse self-attention module; the nasopharynx region image is sequentially input to the convolution module and FA attention module for feature extraction to obtain the first feature image, The position and spatial information of the nasopharyngeal lesion area of the small target can be captured by the FA attention module; the first feature image is down-sampled, and the down-sampled first feature image is input to the 3D interleaved sparse self-attention module for further processing. Image processing to obtain the second feature image, which can provide an effective data basis for further optimizing the overall segmentation effect of the lesion area; obtain the third feature image, perform fusion calculation on the second feature image and the third feature image, and obtain the attention gate coefficient Matrix, realize the fusion of low-level and high-level semantic information, further compensate the low-level semantic information, and further improve the edge segmentation effect based on the feature image, where the third feature image is the feature image output by the previous network layer of the reference network layer, refer to The network layer is the network layer corresponding to the output second feature image; the second feature image is input to the decoder for decoding operation, and the decoded second feature image and the attention gate coefficient matrix are fused and calculated to determine the lesion area segmentation result . The present application extracts features of nasopharyngeal area images and segments nasopharyngeal lesion areas based on the fusion attention network model. Compared with the method of dividing nasopharyngeal lesion areas by manual judgment of physicians in the related art, the technical solution of the embodiment of the application can effectively Improve the detection accuracy of the lesion area.
需要说明的是，本申请实施例并不限制融合注意力网络模型的具体架构，可以是UNet网络模型，本领域技术人员根据实际需求确定即可。It should be noted that the embodiment of the present application does not limit the specific architecture of the fused attention network model, it may be a UNet network model, and those skilled in the art may determine it according to actual needs.
需要说明的是，本申请实施例并不限制具体的鼻咽区域图像的图像格式以及数量，可以包括69例目标对象的鼻咽区域图像数据，每张图像的分辨率可以是1mm×1mm，图像尺寸可以是512×512，每张图像的平面切片之间的距离为3mm，不同的目标对象对应的平面切片的数量的数值范围为[103，152]。It should be noted that the embodiment of the present application does not limit the specific image format and quantity of nasopharyngeal region images, which may include 69 cases of nasopharyngeal region image data of target objects, and the resolution of each image may be 1mm×1mm. The size can be 512×512, the distance between the plane slices of each image is 3mm, and the number of plane slices corresponding to different target objects can range from [103, 152].
需要说明的是，本申请实施例并不限制对第一特征图像进行下采样的具体次数，以及下采样的具体操作，下采样操作可以是：通过1*2*2的最大池化层将第一特征图像的尺寸缩小为原图尺寸的一半，将池化处理后的第一特征图像输入至预设的FA注意力模块进行特征提取，得到第四特征图像，重复3次下采样操作，将第四特征图像依次输入至上述实施例的卷积模块、1*2*2的最大池化层和FA注意力模块进行图像处理，从而得到下采样后的第一特征图像。It should be noted that the embodiment of the present application does not limit the specific number of times of downsampling the first feature image, and the specific operation of downsampling. The downsampling operation may be: the first feature image is pooled by a maximum pooling layer of 1*2*2. The size of a feature image is reduced to half the size of the original image, and the first feature image after pooling is input to the preset FA attention module for feature extraction to obtain the fourth feature image, and the downsampling operation is repeated 3 times, and the The fourth feature image is sequentially input to the convolution module, the 1*2*2 maximum pooling layer and the FA attention module of the above embodiment for image processing, so as to obtain the downsampled first feature image.
另外，参照图2，在一些实施例中，卷积模块包括卷积层、第一BN层和第一激活函数层，图1步骤S130包括但不限于有以下步骤：In addition, referring to FIG. 2, in some embodiments, the convolution module includes a convolution layer, a first BN layer, and a first activation function layer. Step S130 in FIG. 1 includes but is not limited to the following steps:
步骤S210，将鼻咽区域图像输入至第一卷积层进行卷积处理，得到第一中间图像；Step S210, inputting the nasopharyngeal region image to the first convolutional layer for convolution processing to obtain a first intermediate image;
步骤S220，将第一中间图像输入至第一BN层进行图像归一化处理，得到第二中间图像；Step S220, inputting the first intermediate image to the first BN layer for image normalization processing to obtain a second intermediate image;
步骤S230，将第二中间图像输入至第一激活函数层进行非线性变换处理，得到第三中间图像；Step S230, inputting the second intermediate image to the first activation function layer for nonlinear transformation processing to obtain a third intermediate image;
步骤S240，将第三中间图像输入至FA注意力模块进行特征提取，得到第一特征图像。需要说明的是，本申请实施例并不限制卷积模块中的卷积层的参数，可以是1*3*3的卷积层，并且不限制第一激活函数层的具体结构，第一激活函数层可以是Leaky ReLU激活函数层。Step S240, inputting the third intermediate image to the FA attention module for feature extraction to obtain the first feature image. It should be noted that the embodiment of the present application does not limit the parameters of the convolutional layer in the convolution module, which can be a 1*3*3 convolutional layer, and does not limit the specific structure of the first activation function layer. The first activation The function layer may be a Leaky ReLU activation function layer.
需要说明的是，参考图7，本申请实施例的FA注意力模块包括通道注意映射模块和空间注意映射模块，将第三中间图像输入至FA注意力模块进行特征提取，得到第一特征图像，根据以下公式得到：It should be noted that, referring to FIG. 7 , the FA attention module of the embodiment of the present application includes a channel attention mapping module and a space attention mapping module, and the third intermediate image is input to the FA attention module for feature extraction to obtain the first feature image, According to the following formula:
其中，F′为第一特征图像，F∈RC×H×W为第三中间图像，M(F)为FA注意力模块，的表达式为：Among them, F′ is the first feature image, F∈RC×H×W is the third intermediate image, M(F) is the FA attention module, and the expression is:
M(F)＝Sigmoid(MC(F)+MS(F))；M(F)=Sigmoid(MC (F)+MS (F));
其中，Mc(F)为通道注意映射模块，Ms(F)为空间注意映射模块。Among them, Mc (F) is the channel attention mapping module, and Ms (F) is the spatial attention mapping module.
可以理解的是，在通道注意映射模块中，为解决全局池化造成的位置信息丢失，我们将第三中间图像基于通道注意分解为三个平行的一维特征，通过编码过程将空间坐标信息整合到通道注意力的特征向量中。首先利用x、y、z这3个方向的全局平均池化和全局最大池化，分别将3个方向上的输入特征聚合为3个独立的方向感知特征映射，将输入特征图的位置信息嵌入到通道注意力的聚合特征向量，接着，这3个嵌入方向特定信息的特征图被分别编码到3个注意图中，每个注意图捕获输入特征图沿一个空间方向的长期依赖关系，然后通过逐元素相乘将这3种注意图应用于输入特征图，加强感兴趣区域的表示。其中，Mc(F)的表达式如下：It is understandable that in the channel attention mapping module, in order to solve the loss of position information caused by global pooling, we decompose the third intermediate image into three parallel one-dimensional features based on channel attention, and integrate the spatial coordinate information through the encoding process into the feature vector of channel attention. First, using the global average pooling and global maximum pooling in the three directions of x, y, and z, the input features in the three directions are aggregated into three independent direction-aware feature maps, and the position information of the input feature map is embedded. to the aggregated feature vector of channel attention, then, these 3 feature maps embedding direction-specific information are respectively encoded into 3 attention maps, each attention map captures the long-term dependencies of the input feature maps along a spatial direction, and then passed Element-wise multiplication applies these 3 attention maps to the input feature map, strengthening the representation of regions of interest. Among them, the expression of Mc (F) is as follows:
MC(F)＝X1′×Y1′×Z1′；MC (F) = X1 ′×Y1 ′×Z1 ′;
其中，X1′,Y1′,Z1′＝sigmoid(C1Dk(X1,Y1,Z1))，X1,Y1,Z1分别为第三中间图像基于通道注意分解的三个平行的一维特征，C1Dk是卷积核为k的一维卷积，k的取值为3。Among them, X1 ′, Y1 ′, Z1 ′=sigmoid(C1Dk (X1 ,Y1 ,Z1 )), X1 , Y1 , Z1 are the three components of the third intermediate image based on channel attention decomposition. A parallel one-dimensional feature, C1Dk is a one-dimensional convolution with a convolution kernel of k, and the value of k is 3.
需要说明的是，在空间注意映射模块中，本实施例利用特征间的空间信息生成空间注意映射。空间注意是侧重相关信息“在哪里”，与通道注意是相互补充，为了计算空间注意力权值，首先，将特征F∈RC×D×H×W降维投影到RC/r×D×H×W,该过程使用1×1×1卷积整合和压缩跨通道维度的特征映射。然后，为了扩大感受野，更有效的构造空间注意力图，应用两个3×3×3膨胀卷积来有效地利用上下文信息。最后,利用1×1×1卷积将特征再次简化为R1×D×H×W的空间注意力图。相关计算如下：It should be noted that, in the spatial attention mapping module, this embodiment uses the spatial information between features to generate a spatial attention mapping. Spatial attention focuses on the “where” of relevant information, and is complementary to channel attention. In order to calculate the spatial attention weight, first, the feature F∈RC×D×H×W is dimensionally reduced and projected to RC/r×D ×H×W , the process integrates and compresses feature maps across channel dimensions using 1×1×1 convolutions. Then, in order to enlarge the receptive field and construct the spatial attention map more effectively, two 3×3×3 dilated convolutions are applied to effectively utilize the contextual information. Finally, the features are simplified into a spatial attention map of R1×D×H×W again using 1×1×1 convolution. The relevant calculations are as follows:
其中，f为卷积运算，BN为批归一化，f的上标为卷积核大小，两个1×1×1卷积用于通道降维，中间的3×3×3膨胀卷积用于聚合具有更大感受野的上下文信息。Among them, f is a convolution operation, BN is batch normalization, the superscript of f is the size of the convolution kernel, two 1×1×1 convolutions are used for channel dimensionality reduction, and the middle 3×3×3 expansion convolution It is used to aggregate contextual information with a larger receptive field.
另外，执行将下采样后的第一特征图像输入至3D交错稀疏自注意力模块进行图像处理的步骤之前，本申请实施例提供给的方法还包括：将下采样后的第一特征图像输入至FA注意力模块进行图像处理，得到第五特征图像，进一步使得下采样后的第一特征图像专注于学习小目标区域，再将第五特征图像两次输入至新的卷积模块进行特征提取，其中，该新的卷积模块由3*3*3的卷积层、第一BN层和Leaky ReLU激活函数层组成。In addition, before performing the step of inputting the downsampled first feature image to the 3D interleaved sparse self-attention module for image processing, the method provided by the embodiment of the present application further includes: inputting the downsampled first feature image into The FA attention module performs image processing to obtain the fifth feature image, which further makes the downsampled first feature image focus on learning the small target area, and then inputs the fifth feature image twice to the new convolution module for feature extraction. Among them, the new convolution module consists of a 3*3*3 convolution layer, the first BN layer and a Leaky ReLU activation function layer.
可以理解的是，将鼻咽区域图像输入至第一卷积层进行卷积处理，得到第一中间图像，将第一中间图像输入至第一BN层进行图像归一化处理，得到第二中间图像，将第二中间图像输入至第一激活函数层进行非线性变换处理，得到第三中间图像，将第三中间图像输入至FA注意力模块进行特征提取，得到第一特征图像，能够为确定第二特征图像提供有效的数据基础。It can be understood that the nasopharyngeal region image is input to the first convolution layer for convolution processing to obtain the first intermediate image, and the first intermediate image is input to the first BN layer for image normalization processing to obtain the second intermediate image image, the second intermediate image is input to the first activation function layer for nonlinear transformation processing to obtain the third intermediate image, and the third intermediate image is input to the FA attention module for feature extraction to obtain the first feature image, which can be determined The second feature image provides an effective data basis.
另外，参照图3，在一些实施例中，图1步骤S140包括但不限于有以下步骤：3D交错稀疏自注意力模块In addition, referring to FIG. 3 , in some embodiments, step S140 in FIG. 1 includes but is not limited to the following steps: 3D interleaved sparse self-attention module
步骤S310，将下采样后的第一特征图像进行图像划分处理，得到预设数量的第一图像子集，各个第一图像子集的图像尺寸相等；Step S310, performing image division processing on the down-sampled first feature image to obtain a preset number of first image subsets, and the image sizes of each first image subset are equal;
步骤S320，依次从各个图像子集中获取目标像素点，根据全部的目标像素点构造第二图像子集；Step S320, acquiring target pixel points from each image subset in sequence, and constructing a second image subset according to all target pixel points;
步骤S330，根据第二图像子集和预设的自注意力算法计算第一稀疏关联矩阵；Step S330, calculating a first sparse correlation matrix according to the second image subset and a preset self-attention algorithm;
步骤S340，根据第一图像子集和自注意力算法计算第二稀疏关联矩阵；Step S340, calculating a second sparse correlation matrix according to the first image subset and the self-attention algorithm;
步骤S350，根据第一稀疏关联矩阵和第二稀疏关联矩阵得到第二特征图像。Step S350, obtaining a second feature image according to the first sparse correlation matrix and the second sparse correlation matrix.
可以理解的是，为了降低自注意机制的计算代价，将稠密的关联矩阵因式分解为两个稀疏的关联矩阵的乘积。使用两个连续的注意力模块，第一个注意力模块用于估计具有长空间距离的位置子集内的相似度，第二个注意力模块用于估计具有短空间距离的位置子集内的相似度。通过长短距离模块的结合，我们能够将所有的输入位置信息传递到每个输出位置。在处理高分辨率特征图的情况下，与原始的自注意力模块相比，大大降低了计算复杂度。具体地，将下采样后的第一特征图像进行图像划分处理，得到预设数量的第一图像子集(假设为Q个第一图像子集)，各个第一图像子集的图像尺寸相等，每个子集包含P个位置，即N＝P×Q，N为输入图像的大小；对于长距离注意力任务，依次从各个图像子集中获取目标像素点，根据全部的目标像素点构造第二图像子集。每个第二图像子集的位置都具有很长的空间距离，对每个第二图像子集应用自注意力来计算第一稀疏关联矩阵AL；对于短距离注意力任务，直接将自注意力应用到原始Q个第一图像子集上，计算第二稀疏关联矩阵为AS，结合这两种注意机制，根据第一稀疏关联矩阵和第二稀疏关联矩阵得到第二特征图像，可以将信息从每个输入位置传播到所有输出位置。Understandably, in order to reduce the computational cost of the self-attention mechanism, the dense incidence matrix is factorized into the product of two sparse incidence matrices. Using two consecutive attention modules, the first attention module is used to estimate the similarity within the subset of locations with long spatial distance, and the second attention module is used to estimate the similarity within the subset of locations with short spatial distance. similarity. Through the combination of long and short distance modules, we are able to transfer all input location information to each output location. In the case of dealing with high-resolution feature maps, the computational complexity is greatly reduced compared with the original self-attention module. Specifically, the downsampled first feature image is subjected to image division processing to obtain a preset number of first image subsets (assumed to be Q first image subsets), and the image sizes of each first image subset are equal, Each subset contains P positions, that is, N=P×Q, and N is the size of the input image; for long-distance attention tasks, the target pixels are sequentially obtained from each image subset, and the second image is constructed based on all target pixels Subset. The location of each second image subset has a long spatial distance, and self-attention is applied to each second image subset to calculate the first sparse association matrix AL; for short-distance attention tasks, self-attention Applied to the original Q first image subsets, the second sparse association matrix is calculated as AS, combined with these two attention mechanisms, the second feature image is obtained according to the first sparse association matrix and the second sparse association matrix, and the information can be obtained from Each input location is propagated to all output locations.
另外，参照图4，在一些实施例中，解码器包括上采样模块，上采样模块包括反卷积层、第二BN化层和第二激活函数层，图1步骤S150包括但不限于有以下步骤：In addition, referring to FIG. 4, in some embodiments, the decoder includes an upsampling module, and the upsampling module includes a deconvolution layer, a second BN layer, and a second activation function layer. Step S150 in FIG. 1 includes but is not limited to the following step:
步骤S410，将第二特征图像输入至反卷积层进行卷积处理，得到第四中间图像；Step S410, inputting the second feature image to the deconvolution layer for convolution processing to obtain a fourth intermediate image;
步骤S420，将第四中间图像输入至第二BN层进行图像归一化处理，得到第五中间图像；Step S420, inputting the fourth intermediate image to the second BN layer for image normalization processing to obtain a fifth intermediate image;
步骤S430，将第五中间图像输入至第二激活函数层进行非线性变换处理，得到解码后的第二特征图像。In step S430, the fifth intermediate image is input to the second activation function layer for nonlinear transformation processing to obtain a decoded second feature image.
需要说明的是，本申请实施例并不限制上采样模块的具体结构，反卷积层可以是尺寸为1*2*2的的卷积层以及第二激活函数层可以是Leaky ReLU激活函数层。It should be noted that the embodiment of the present application does not limit the specific structure of the upsampling module. The deconvolution layer can be a convolution layer with a size of 1*2*2 and the second activation function layer can be a Leaky ReLU activation function layer .
需要说明的是，本实施例中，将解码后的第二特征图像与注意力门系数矩阵进行融合计算，确定病灶区域分割结果的步骤包括：解码后的第二特征图像输入至FA注意力模块进行图像处理，得到FA注意力处理后的特征图像，再将FA注意力处理后的特征图像两次输入至由尺寸为1*3*3的卷积层、BN层和Leaky ReLU激活函数层组成的卷积模块，进行卷积处理，得到新的特征图像，再将该新的特征图像与注意力门系数矩阵进行融合计算，最后经过1*3*3的卷积层与softmax函数层得到病灶区域分割结果。It should be noted that, in this embodiment, the decoded second feature image and the attention gate coefficient matrix are fused and calculated, and the step of determining the lesion region segmentation result includes: inputting the decoded second feature image to the FA attention module Perform image processing to obtain the feature image after FA attention processing, and then input the feature image after FA attention processing twice to a convolutional layer with a size of 1*3*3, a BN layer and a Leaky ReLU activation function layer The convolution module performs convolution processing to obtain a new feature image, and then fuses the new feature image with the attention gate coefficient matrix for calculation, and finally obtains the lesion through a 1*3*3 convolution layer and a softmax function layer Region segmentation results.
另外，参照图5，在一些实施例中，在执行图1所示实施例中的步骤S120之前，基于融合注意力的病灶区域确定方法还包括但不限于有以下步骤：In addition, referring to FIG. 5 , in some embodiments, before performing step S120 in the embodiment shown in FIG. 1 , the method for determining a lesion area based on fusion attention further includes but is not limited to the following steps:
步骤S510，根据预设规则对鼻咽区域图像进行图像预处理，得到预处理后的目标鼻咽区域图像。Step S510, performing image preprocessing on the image of the nasopharynx region according to preset rules, to obtain a preprocessed image of the target nasopharynx region.
另外，参照图6，在一些实施例中，鼻咽区域图像为CT图像，图5步骤S510包括但不限于有以下步骤：In addition, referring to FIG. 6, in some embodiments, the image of the nasopharynx region is a CT image. Step S510 in FIG. 5 includes but is not limited to the following steps:
步骤S610，确定鼻咽区域图像的Hu值，将Hu值截断至预设范围内，得到中间鼻咽区域图像；Step S610, determining the Hu value of the nasopharyngeal region image, and truncating the Hu value to a preset range to obtain an intermediate nasopharyngeal region image;
步骤S620，对中间鼻咽区域图像进行归一化处理，得到目标鼻咽区域图像。Step S620, performing normalization processing on the middle nasopharyngeal region image to obtain the target nasopharyngeal region image.
可以理解的是，在图像预处理阶段，首先将鼻咽区域图像的强度值截断到[-200,700]的HU值的范围内以增加目标区域的对比度，然后通过归一化处理将图像的Hu值映射在[0，1]的范围，去除因奇异样本数据引起的不好影响，同时加快训练收敛速度。本申请实施例的图像预处理步骤可以根据以下公式得到：It can be understood that in the image preprocessing stage, the intensity value of the nasopharyngeal region image is first truncated to the range of the HU value of [-200,700] to increase the contrast of the target area, and then the Hu value of the image is normalized The mapping is in the range of [0, 1], which removes the bad influence caused by the singular sample data, and at the same time speeds up the training convergence speed. The image preprocessing step of the embodiment of the present application can be obtained according to the following formula:
其中，鼻咽区域图像当前的像素对应的Hu值用x表示，最大值和最小值则分别使用x_max，x_min表示，归一化后的像素对应的Hu值用x_norm表示。Among them, the Hu value corresponding to the current pixel of the nasopharyngeal region image is represented by x, the maximum and minimum values are represented by x_max and x_min respectively, and the Hu value corresponding to the normalized pixel is represented by x_norm.
为了保持目标鼻咽区域图像具有的相同的分辨率，将所有的目标鼻咽区域图像在x、y、z方向上的像素间距均匀插值到1mm*1mm*3mm，目标鼻咽区域图像在x、y的方向上尺寸为512*512，这包含了大量的背景和仪器区域，一定程度上影响模型对于小目标分割的效果。所以，需要根据先验信息减小图像尺寸来除去一定的无效信息.为了更好地学习目标区域相关特征，并兼顾位置信息和生理结构信息。In order to maintain the same resolution of the target nasopharyngeal region images, the pixel spacing of all target nasopharyngeal region images in the x, y, and z directions is evenly interpolated to 1mm*1mm*3mm, and the target nasopharyngeal region images are in the x, y, and z directions. The size in the y direction is 512*512, which contains a large number of background and instrument areas, which affects the effect of the model on small target segmentation to a certain extent. Therefore, it is necessary to reduce the size of the image according to the prior information to remove certain invalid information. In order to better learn the relevant features of the target area, and take into account the location information and physiological structure information.
另外，参照图8，图8是本申请另一个实施例提供的基于融合注意力的病灶区域确定装置，在一实施例中，本申请实施例提供了一种基于融合注意力的病灶区域确定装置800，包括：In addition, referring to Fig. 8, Fig. 8 is an apparatus for determining a lesion area based on fusion attention provided by another embodiment of the present application. In one embodiment, an embodiment of the present application provides an apparatus for determining a lesion area based on
图像获取模块810，图像获取模块810用于获取鼻咽区域图像；An
第一图像处理模块820，第一图像处理模块820用于将鼻咽区域图像输入至预先训练好的融合注意力网络模型，融合注意力网络模型包括编码器和解码器，编码器包括依次连接的卷积模块、FA注意力模块和3D交错稀疏自注意力模块；The first
第二图像处理模块830，第二图像处理模块830用于将鼻咽区域图像依次输入至卷积模块和FA注意力模块进行特征提取，得到第一特征图像；The second image processing module 830, the second image processing module 830 is used to sequentially input the nasopharyngeal region image to the convolution module and the FA attention module for feature extraction, to obtain the first feature image;
第三图像处理模块840，第三图像处理模块840用于对第一特征图像进行下采样处理，并将下采样后的第一特征图像输入至3D交错稀疏自注意力模块进行图像处理，得到第二特征图像；The third image processing module 840, the third image processing module 840 is used to down-sample the first feature image, and input the down-sampled first feature image to the 3D interleaved sparse self-attention module for image processing, to obtain the first feature image Two feature images;
第四图像处理模块850，第四图像处理模块850用于获取第三特征图像，对第二特征图像和第三特征图像进行融合计算，得到注意力门系数矩阵，第三特征图像为参考网络层的上一层网络层输出的特征图像，参考网络层为输出第二特征图像对应的网络层；The fourth image processing module 850, the fourth image processing module 850 is used to obtain the third feature image, perform fusion calculation on the second feature image and the third feature image, and obtain the attention gate coefficient matrix, and the third feature image is the reference network layer The feature image output by the previous network layer, the reference network layer is the network layer corresponding to the output second feature image;
病灶区域分割结果确定模块860，病灶区域分割结果确定模块860用于将第二特征图像输入至解码器进行解码操作，并将解码后的第二特征图像与注意力门系数矩阵进行融合计算，确定病灶区域分割结果。The lesion area segmentation result determination module 860, the lesion area segmentation result determination module 860 is used to input the second feature image to the decoder for decoding operation, and perform fusion calculation on the decoded second feature image and the attention gate coefficient matrix to determine Lesion region segmentation results.
需要说明的是，基于融合注意力的病灶区域确定装置800的具体实施方式与上述基于融合注意力的病灶区域确定方法的具体实施例基本相同，在此不再赘述。It should be noted that the specific implementation of the
另外，参考图9，图9是本申请另一个实施例提供的基于融合注意力的病灶区域确定装置的结构图，本申请的一个实施例还提供了一种基于融合注意力的病灶区域确定装置900，该基于融合注意力的病灶区域确定装置900包括：存储器910、处理器720及存储在存储器910上并可在处理器920上运行的计算机程序。In addition, referring to FIG. 9, FIG. 9 is a structural diagram of a device for determining a lesion area based on fusion attention provided by another embodiment of the present application. An embodiment of the application also provides a device for determining a lesion area based on fusion attention. 900. The device 900 for determining a lesion area based on fusion attention includes: a memory 910, a
处理器920和存储器910可以通过总线或者其他方式连接。The processor 920 and the memory 910 may be connected through a bus or in other ways.
实现上述实施例的基于融合注意力的病灶区域确定方法所需的非暂态软件程序以及指令存储在存储器910中，当被处理器920执行时，执行上述实施例中基于融合注意力的病灶区域确定方法，例如，执行以上描述的图1中的方法步骤S110至步骤S160、图2中的方法步骤S210至步骤S240、图3中的方法步骤S310至步骤S350、图4中的方法步骤S410至步骤S430、图5中的方法步骤S510和图6中的方法步骤S610至步骤S620。The non-transient software programs and instructions required to realize the fusion attention-based lesion area determination method of the above-mentioned embodiment are stored in the memory 910, and when executed by the processor 920, the fusion attention-based lesion area determination method in the above-mentioned embodiment is executed. Determine the method, for example, perform method steps S110 to S160 in FIG. 1 described above, method steps S210 to S240 in FIG. 2 , method steps S310 to S350 in FIG. 3 , method steps S410 to S410 in FIG. 4 Step S430, the method step S510 in FIG. 5, and the method steps S610 to S620 in FIG. 6.
以上所描述的装置实施例仅仅是示意性的，其中作为分离部件说明的单元可以是或者也可以不是物理上分开的，即可以位于一个地方，或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部模块来实现本实施例方案的目的。The device embodiments described above are only illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place, or may be distributed to multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.
此外，本申请的一个实施例还提供了一种计算机可读存储介质，该计算机可读存储介质存储有计算机可执行指令，该计算机可执行指令被一个处理器或控制器执行，例如，被上述基于融合注意力的病灶区域确定装置900实施例中的一个处理器920执行，可使得上述处理器920执行上述实施例中的基于融合注意力的病灶区域确定方法，例如，执行以上描述的图1中的方法步骤S110至步骤S160、图2中的方法步骤S210至步骤S240、图3中的方法步骤S310至步骤S350、图4中的方法步骤S410至步骤S430、图5中的方法步骤S510和图6中的方法步骤S610至步骤S620。本领域普通技术人员可以理解，上文中所公开方法中的全部或某些步骤、系统可以被实施为软件、固件、硬件及其适当的组合。某些物理组件或所有物理组件可以被实施为由处理器，如中央处理器、数字信号处理器或微处理器执行的软件，或者被实施为硬件，或者被实施为集成电路，如专用集成电路。这样的软件可以分布在计算机可读介质上，计算机可读介质可以包括计算机存储介质(或非暂时性介质)和通信介质(或暂时性介质)。如本领域普通技术人员公知的，术语计算机存储介质包括在用于存储信息(诸如计算机可读指令、数据结构、程序模块或其他数据)的任何方法或技术中实施的易失性和非易失性、可移除和不可移除介质。计算机存储介质包括但不限于RAM、ROM、EEPROM、闪存或其他存储器技术、CD-ROM、数字多功能盘(DVD)或其他光盘存储、磁盒、磁带、磁盘存储或其他磁存储装置、或者可以用于存储期望的信息并且可以被计算机访问的任何其他的介质。此外，本领域普通技术人员公知的是，通信介质通常包含计算机可读指令、数据结构、程序模块或者诸如载波或其他传输机制之类的调制数据信号中的其他数据，并且可包括任何信息递送介质。In addition, an embodiment of the present application also provides a computer-readable storage medium, the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are executed by a processor or a controller, for example, by the above-mentioned Executed by a processor 920 in the embodiment of the apparatus 900 for determining a lesion area based on fusion attention, the above-mentioned processor 920 can execute the method for determining a lesion area based on fusion attention in the above embodiment, for example, execute the above-described Figure 1 Method step S110 to step S160 in, method step S210 to step S240 in Fig. 2, method step S310 to step S350 in Fig. 3, method step S410 to step S430 in Fig. 4, method step S510 in Fig. 5 and The method step S610 to step S620 in FIG. 6 . Those skilled in the art can understand that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware and an appropriate combination thereof. Some or all of the physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit . Such software may be distributed on computer readable media, which may include computer storage media (or non-transitory media) and communication media (or transitory media). As known to those of ordinary skill in the art, the term computer storage media includes both volatile and nonvolatile media implemented in any method or technology for storage of information, such as computer readable instructions, data structures, program modules, or other data. permanent, removable and non-removable media. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cartridges, tape, magnetic disk storage or other magnetic storage devices, or can Any other medium used to store desired information and which can be accessed by a computer. In addition, as is well known to those of ordinary skill in the art, communication media typically embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media .
Claims (10)
Translated fromChinese
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310326310.1ACN116402997A (en) | 2023-03-29 | 2023-03-29 | Focus region determination method, device and storage medium based on fusion attention |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310326310.1ACN116402997A (en) | 2023-03-29 | 2023-03-29 | Focus region determination method, device and storage medium based on fusion attention |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116402997Atrue CN116402997A (en) | 2023-07-07 |
Family
ID=87009725
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310326310.1APendingCN116402997A (en) | 2023-03-29 | 2023-03-29 | Focus region determination method, device and storage medium based on fusion attention |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116402997A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117893745A (en)* | 2023-10-10 | 2024-04-16 | 清华大学深圳国际研究生院 | Retinal lesion segmentation method and device, electronic device and storage medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111126258A (en)* | 2019-12-23 | 2020-05-08 | 深圳市华尊科技股份有限公司 | Image recognition method and related device |
| CN111260663A (en)* | 2020-01-15 | 2020-06-09 | 华南理工大学 | Nasopharyngeal carcinoma focus image segmentation device, equipment and computer readable storage medium |
| CN115187613A (en)* | 2022-07-12 | 2022-10-14 | 广东工业大学 | A multimodal nasopharyngeal carcinoma image segmentation method based on attention and graph convolution |
- 2023
- 2023-03-29CNCN202310326310.1Apatent/CN116402997A/enactivePending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111126258A (en)* | 2019-12-23 | 2020-05-08 | 深圳市华尊科技股份有限公司 | Image recognition method and related device |
| CN111260663A (en)* | 2020-01-15 | 2020-06-09 | 华南理工大学 | Nasopharyngeal carcinoma focus image segmentation device, equipment and computer readable storage medium |
| CN115187613A (en)* | 2022-07-12 | 2022-10-14 | 广东工业大学 | A multimodal nasopharyngeal carcinoma image segmentation method based on attention and graph convolution |
Non-Patent Citations (3)
| Title |
|---|
| HUANG, LANG, ET: "Interlaced Sparse Self-Attention for Semantic Segmentation", COMPUTER VISION AND PATTERN RECOGNITION, 30 July 2019 (2019-07-30), pages 1 - 3* |
| PARK, JONGCHA, ET: "BAM: Bottleneck Attention Module", COMPUTER VISION AND PATTERN RECOGNITION, 18 July 2018 (2018-07-18), pages 4* |
| 王天，梁淑芬等: "适用于鼻咽癌原发灶三维自动化分割的MCAS-UNet网络", 五邑大学学报（自然科学版）, vol. 36, no. 4, 30 November 2022 (2022-11-30), pages 51 - 55* |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117893745A (en)* | 2023-10-10 | 2024-04-16 | 清华大学深圳国际研究生院 | Retinal lesion segmentation method and device, electronic device and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11200424B2 (en) | Space-time memory network for locating target object in video content | |
| CN111369581B (en) | Image processing method, device, equipment and storage medium | |
| US8958630B1 (en) | System and method for generating a classifier for semantically segmenting an image | |
| CN114612289B (en) | Stylized image generation method, device and image processing equipment | |
| CN112308866B (en) | Image processing method, device, electronic equipment and storage medium | |
| JP2019526878A (en) | Depth prediction from image data using statistical models | |
| CN110246084B (en) | Super-resolution image reconstruction method, system and device thereof, and storage medium | |
| JP2024507727A (en) | Rendering a new image of a scene using a geometric shape recognition neural network conditioned on latent variables | |
| CN114359361B (en) | Depth estimation method, device, electronic device and computer-readable storage medium | |
| CN109543685A (en) | Image, semantic dividing method, device and computer equipment | |
| CN113807354B (en) | Image semantic segmentation method, device, equipment and storage medium | |
| CN112967293A (en) | Image semantic segmentation method and device and storage medium | |
| CN119130863B (en) | Image recovery method and system based on multiple attention mechanisms | |
| CN114170325B (en) | Method, apparatus, medium, device and program product for determining homography matrix | |
| CN115546270A (en) | Image registration method, model training method and equipment for multi-scale feature fusion | |
| CN112861940A (en) | Binocular disparity estimation method, model training method and related equipment | |
| CN114565953A (en) | Image processing method, image processing device, electronic equipment and computer readable storage medium | |
| CN117115184A (en) | Training method and segmentation method of medical image segmentation model and related products | |
| CN112634331B (en) | Optical flow prediction method and device | |
| CN116363561A (en) | A method, device, equipment and storage medium for timing action positioning | |
| CN116523959A (en) | A method and system for detecting moving objects based on artificial intelligence | |
| CN116402997A (en) | Focus region determination method, device and storage medium based on fusion attention | |
| CN116977674A (en) | Image matching method, related device, storage medium and program product | |
| CN112861858B (en) | Method for generating saliency truth value diagram and method for training saliency detection model | |
| Bao et al. | Distilling resolution-robust identity knowledge for texture-enhanced face hallucination |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |