CN112597797B - Analysis of Nanoprobes for Virus Capture in Dark Field Microscopy Based on Deep Learning - Google Patents

Abstract

Translated fromChinese

本发明公开了基于深度学习分析暗场显微镜中纳米探针捕获病毒的方法，包括如下步骤：(1)基于Opencv的金纳米探针提取和去噪模型设计；(2)ResNet50为纳米探针夹心结构判定的模型设计；(3)使用训练好的基于ResNet50的夹心结构分辨模型和基于Opencv的金纳米颗粒提取去噪模型对暗场显微镜成像原图进行实时分析和判断。该方法基于OpenCV的金纳米探针提取模型有效地去除背景的干扰，基于ResNet50的夹心结构判定模型正确识别金、银双纳米探针与病毒结合形成的“银纳米探针@病毒@金纳米探针”夹心结构，从而保证纳米探针捕获病毒后在暗场中计数的效率和准确性，使以纳米颗粒为信号探针，以暗场显微镜为平台的靶标分子和微生物检测中具备真正的应用价值。

The invention discloses a method based on deep learning to analyze nano-probes to capture viruses in a dark field microscope. Model design for structure determination; (3) Use the trained ResNet50-based sandwich structure resolution model and Opencv-based gold nanoparticle extraction and denoising model to analyze and judge the original image of dark field microscopy in real time. The method is based on the gold nanoprobe extraction model of OpenCV to effectively remove the background interference, and the sandwich structure determination model based on ResNet50 correctly identifies the "silver nanoprobe@virus@gold nanoprobe" formed by the combination of gold and silver double nanoprobes and viruses. Needle "sandwich structure, so as to ensure the efficiency and accuracy of nanoprobes to count the virus in the dark field after capturing the virus, so that the nanoparticle as the signal probe and the dark field microscope as the platform have real applications in the detection of target molecules and microorganisms. value.

Description

Method for capturing virus by nano probe in dark field microscope based on deep learning analysis

Technical Field

The invention relates to the technical field of biological image analysis and computer vision, in particular to a method for capturing viruses by using a nanoprobe in a dark field microscope based on deep learning analysis.

Background

There are many articles for constructing biological detection methods for microorganisms and various molecular targets by using different nanoparticles as dual-signal probes and using a dark-field microscope as a platform. In the final counting link, however, pictures with less impurities and clearer particles in the visual field are mostly selected; or the image analysis software is used for carrying out over-treatment so as to achieve the purposes of deliberately removing impurities and beautifying the result; or a high-end instrument may be used to measure the scatter spectrum. These methods are heavy and time consuming to avoid. There are three reasons why the final imaging result is not as perfect as the ideal state, combined with our previous research analysis. (1) The detection of target molecules by using a dark-field microscope is mostly directly carried out on a common glass slide, and is easily interfered by granular impurities in the surrounding complex environment due to the conventional cleaning (ultrasonic treatment after 100% absolute ethyl alcohol soaking) and sample dripping processes. Therefore, it is difficult to avoid the appearance of particles of foreign matter in the form of starry dots in the final field of view. (2) Nanoparticles that are not bound to the target molecules are also difficult to remove completely during the elution process, resulting in excessive nanoparticles in the final field of view, resulting in a method that is not very convincing for quantification by scattering spectroscopy. (3) The scattering angle to the light source is different due to the anisotropy of the nano-particles, so that the imaging result of different nano-particles under a dark field microscope has slight color difference.

OpenCV is a computer vision processing library, and comprises a plurality of modularized image processing algorithms, so that a user can freely and conveniently process a target picture after importing the target picture through a computer programming language Python. In recent years, due to the rapid development of computer vision technology based on deep learning, the computer vision technology has been successful in the interpretation of extremely complex medical images, is excellent in medical diagnosis and disease analysis, and has remarkable application potential in other fields. Convolutional Neural Networks (CNN) is one of the representative algorithms for deep learning, and is widely used in image processing. The conventional CNN model suffers from loss of information every time information propagates between convolutional Layers due to the presence of nonlinear activation functions with superposition of convolutional Layers (Stacked Layers), resulting in deep-delivered low-dimensional data Collapse (Collapse) and "death" of neurons here. The ResNet model provides a residual learning idea, solves the problem of gradient disappearance to a certain extent, and protects the integrity of information. When processing complex images, the ResNet model stands out when deeper networks are needed to extract image features.

Disclosure of Invention

The purpose of the invention is as follows: the invention aims to provide a method for capturing viruses by using a nanoprobe in a dark field microscope based on deep learning analysis, which can avoid background interference and difficulty in counting caused by scattering color difference due to anisotropy of the nanoprobe, and improves the accuracy of counting and analysis. Meanwhile, the method of the invention can also provide a new idea for dark field microscope imaging analysis, and has strong generalization capability and practical value.

The technical scheme is as follows: the invention provides a method for capturing viruses by a nanoprobe in a dark field microscope based on deep learning analysis, which comprises the following steps:

(1) gold nanoparticle extraction and denoising model design based on Opencv;

(2) model design based on ResNet50 as a nanoparticle sandwich decision;

(3) and (3) carrying out real-time analysis and judgment on the original image of the dark-field microscope imaging by using a trained sandwich structure resolution model based on ResNet50 and an Opencv-based gold nanoparticle extraction denoising model.

Further, the model design method of the step (1) is as follows:

a. obtaining an original image of a dark field microscope;

b. converting the original image from an RGB three channel into an HSV three channel;

c. determining upper and lower limits of gold nanoparticle red characteristic HSV and creating a mask;

d. performing AND operation on the mask created in the step c and the original image;

e. filling gaps among the dispersed color blocks of the single gold nano probe in the image obtained in the step d by using an expansion algorithm to form full color blocks attached to the nano probe;

f. and acquiring two-dimensional space coordinates of each full color block (namely the gold nanoprobe) by utilizing a connected component algorithm.

g. And (4) extracting each gold nanoprobe by a custom function, storing the gold nanoprobe in a designated folder, and creating a data set for deep neural network training.

h. And extracting each gold nanoprobe by the self-defined function and storing the gold nanoprobe in a specified folder.

i. And (3) calculating the number of pixel points of which the Hue value falls in a red range in each extracted graph according to the gold nanoprobes extracted in the step g to obtain the gold and silver nanoprobes, a virus sandwich structure, the gold nanoprobes and an impurity Hue value distribution rule with red characteristics, and thus carrying out secondary screening.

i. And h, supplementing the Hue value distribution rule reacted in the step h as a judgment item in an Opencv-based gold nanoprobe extraction and denoising model.

k. Sending other shot dark field images into a model in batches, leading out a gold-silver nano probe sandwich structure, a single gold nano probe and a very small number of particles with red characteristics, and manually marking the gold-silver nano probe sandwich structure, the single gold nano probe and the very small number of particles as bin and nobind. As a training data set for a deep learning model.

Further, the model design method in the step (2) is as follows:

a. manually labeling gold nanoprobes in a data set, and dividing the gold nanoprobes into a Bind and an Unbind;

b. constructing a ResNet50 deep neural network model, wherein the model is input into a normalized image with classification labels;

c. defining a convolution module and an identity module based on a Keras framework according to a model structure of ResNet50, and then overlapping;

d. setting training parameters, and carrying out model training by using the data set obtained in the step g in the step (1);

e. and applying the trained model to an actual dark-field microscope imaging image for testing.

Further, the model structure of the ResNet50 in step c includes four convolution modules and twelve identity modules.

Further, each convolution module comprises a neural network stacked trunk and side branches of convolutional layers, each trunk comprises three convolutional layers, and each side branch comprises one convolutional layer. Each identity module includes a neural network stacked trunk and side branches without convolutional layers, each trunk including three convolutional layers.

The principle of the method of the invention is as follows:

the scattered light of the gold nanoparticles has stable color gamut information in the image of the dark field microscope, and the gold nanoparticles still keep the prominent red characteristic although the image of each gold nanoparticle has slight color difference due to the anisotropy of the nanoprobe. Based on this, an Opencv computer vision processing library is utilized to transfer the original image from an RGB channel to an HSV channel, a Mask (Mask) is created to extract a red characteristic interval, the primary extraction of gold particle characteristics is completed in this step, and imaging noise of most impurities in the original image is filtered. In order to solve the problem that red features are not uniformly distributed on a single gold probe due to chromatic aberration, a space between single-particle dispersed color blocks is filled by using a smaller kernel (Kernal) in a color extraction image after binarization by using an Opencv expansion algorithm, so that the color gamut information of single gold particles is fuller. And then, carrying out coordinate positioning and single particle image extraction on the previously extracted and expanded binary image by means of a connected component algorithm, wherein the result comprises a gold and silver probe sandwich structure and an independent gold probe structure. And finally, extracting and storing the single particle image by using a self-defined function. However, due to the extraction method based on HSV color characteristics, in practical application, certain impurities also have red characteristics in an imaging visual field, and finally, the impurities and the silver nanoprobes are also in a sandwich structure. In order to avoid the risk of final discrimination failure of the deep learning model, three substances, namely a standard sandwich structure (gold and silver nanometer probe sandwich virus), a single gold nanometer probe and impurity particles with red characteristics, are analyzed on a Hue value, so that a statistical rule is found primarily, and a limited condition is added into the extraction model, so that the impurity particles with the red characteristics are excluded, and the interference on the model is reduced. The ResNet50 model is based on a Keras deep learning framework, an official source code is referred, and an identity module and a convolution module are defined and then overlapped according to the model architecture of ResNet 50. In order to better realize interactive operation, parameters and codes are separated by means of an argparse module of Python, and a model can be directly trained and images can be directly analyzed on the Shell of each platform. The method disclosed by the patent is well represented in the actual scientific research topic and the detection strategy of the gold and silver nano probe sandwich virus. The method solves the problems of inevitable background interference and the counting difficulty caused by scattering color difference due to the anisotropy of the nano-particles. And the method has strong universality by utilizing a computer vision image processing technology and a deep learning model to analyze the dark-field microscope imaging graph, and has great guiding significance for counting and analyzing in the later stage of the detection strategy taking the dark-field microscope as a platform.

Has the advantages that: the invention innovatively provides a method for capturing viruses by using a nanoprobe in a dark-field microscope based on deep learning analysis. The gold nanoprobe extraction model based on OpenCV effectively removes the interference of the background, and the sandwich structure judgment model of ResNet50 based on the deep neural network enhances the efficiency and the scientificity of a virus detection strategy by using gold and silver nanoprobes as a sandwich in a counting link, so that the detection of target molecules and microorganisms by using nanoparticles as signal probes and using a dark field microscope as a platform has real practical value.

Meanwhile, the method of the invention can also provide a new idea for dark field microscope imaging analysis, and has strong generalization capability and practical value.

Drawings

FIG. 1 is a flow chart of the method of the present invention;

FIG. 2 is a flow chart of gold nanoparticle extraction and denoising;

FIG. 3 is a diagram illustrating gold particle labeling of an original drawing;

FIG. 4 shows the distribution rule of Hue value of gold and silver nanoprobes and virus sandwich structures, gold nanoprobes and impurities with red characteristics (y-axis scale 0-700);

FIG. 5 shows the distribution rule of Hue value of gold and silver nanoprobes and virus sandwich structures, gold nanoprobes and impurities with red color characteristics (y-axis scale 0-300);

FIG. 6 is a single gold particle extraction and intelligent discrimination image;

fig. 7 is a diagram of a ResNet50 model structure and a convolution module identity structure.

Detailed Description

In this embodiment, the dark-field microscope imaging analysis based on Opencv and ResNet50 of the present invention has a flow as shown in fig. 1, and specifically includes the following two models:

model one: the gold nanoparticle extraction and denoising model based on Opencv has a processing flow as shown in FIG. 1, and specifically includes the following steps:

a. sucking the silver particles modified with the virus antibody on a glass slide by a liquid transfer machine, and carrying out closed incubation for 15 minutes at normal temperature. Then adding the virus sample and the gold particles modified with the antibody, and carrying out closed incubation for 15 minutes at normal temperature each time.

b. The slides were mounted on the stage of an STL-P1600CMOS3 dark field microscope and viewed with a 50 x objective lens.

c. And using STL-IMCS dark field view acquisition software to acquire dark field imaging pictures in the view field, directly shooting the pictures without processing, and storing the pictures in a designated folder.

d. Setting a threshold value range of a single gold nanometer probe imaging color characteristic (red) in an HSV format specified in Opencv, traversing each pixel of the whole picture through an Inrange function, setting a pigment in the threshold value to be 255 (white), and setting a pigment outside the threshold value to be 0 (black).

lower_red＝numpy.array(156，43，45)，upper_red＝numpy.array(180，255，255)

This step completes the process similar to binarization, but initially extracts the characteristic color block (red-like) in the single gold nanoparticle, has approximate positioning to the gold particles scattered in the whole visual field, and removes the background noise (including unknown impurity particles and excessive silver nanoparticles in the air).

e. And creating a Kernel, and performing expansion operation on the imaging information output in the step d through a scale function of Opencv. numy. ons ((8, 8), np. uint8)

The Kernel size created in this step determines whether the feature color block after the expansion operation can fully represent the gold nanoparticles in the original image, and also depends on the range of the final single gold nanoparticle extraction, and from practice, when the expansion operation is performed on the original image by using the matrix of 8 × 8, the representing and extraction effects are the best.

f. And performing connected component operation on the e-output picture through a connected Components WithStats function of Opencv. And acquiring the position information of each characteristic color block after the expansion operation.

centroids＝cv2.connectedComponentsWithStats(image，connectivity＝8)

The connected component algorithm used in the step is one of the most common algorithms in image analysis, and essentially scans each pixel point of the binary image, combines the pixel points which are the same and are mutually connected into a group, and finally derives the actual position coordinate of each group. From practical operation, when the connected component (connectivity parameter) is 8, the extraction effect is the best.

g. And (4) coordinate information acquired from the f through a Rectangle function, and performing a slicing operation in the original image.

In this step, the gold particles in the original drawing are directly labeled as shown in fig. 3.

h. Extracting single gold nanometer probe image through self-defined function

i. And calculating the number of pixel points of each extracted image with the Hue value falling in the red range to obtain the gold and silver nanoprobes, the virus sandwich structure, the gold nanoprobes and the impurity Hue value distribution rule with red characteristics, thereby carrying out secondary screening. (as shown in FIGS. 4 and 5)

j. And (e) taking the Hue value distribution rule reacted in the step i as a logic judgment item, and supplementing the logic judgment item into an Opencv-based gold nanoprobe extraction and denoising model.

k. And reading the dark field microscope imaging original images of the specified folders in batches, and obtaining the export result of a single gold nanoprobe, a gold and silver nanoprobe sandwich and a few impurities with red characteristics. (very few impurities with red color characteristics were selected by the screening in step i)

The extracted image of a single gold particle in this step is shown in fig. 3.

And o, manually carrying out two-classification treatment on the 1500 images extracted in the step k, judging whether the images are gold and silver probe sandwich structures or single gold nanometer probes, creating two sub-folders in the folder A, wherein the two sub-folders are named as Bind and Unbind, and 700 images are respectively placed in the two folders.

Model two: the nanoparticle sandwich judgment model based on ResNet50 specifically comprises the following steps:

a. ResNet50 was chosen as the nanoparticle sandwich assay model.

ResNet50 is selected as the discriminant model in this step for the following reasons: because we directly process the original image under the dark-field microscope, the impurity condition in the dark-field microscope imaging is complex, the sandwich structure between the double-nanometer probe and the virus is not simple geometric overlapping, and the influence of scattered light on the imaging structure needs to be considered. We have chosen earlier a more "deep" conventional convolutional neural network, but later found that it resolved the effects of sandwich structures and was as good as expected when applied in practice. This is because the deep network degrades as the number of layers increases deeply, so i finally chose ResNet50 as the deep neural network model. It still maintains excellent performance during deep network training.

b. And defining a convolution module and an identity module based on a Keras framework according to the model structure of ResNet50, and then overlapping.

The ResNet50 model structure built in the step is 53 convolutional layers in total, wherein each convolutional module comprises 4 convolutional layers, each identity module comprises 3 convolutional layers, and the structures of the convolutional modules and the identity modules are shown in FIG. 7.

c. And setting training parameters, and performing model training by using the data set obtained in the first model.

d. And applying the trained model to an actual dark-field microscope imaging image for testing. The test results are shown in fig. 6.

Claims (4)

Translated fromChinese

1.一种基于深度学习分析暗场显微镜中纳米探针捕获病毒的方法，其特征在于：包括如下步骤：1. a method based on deep learning analysis of nano-probe to capture virus in dark field microscope, is characterized in that: comprise the steps:（1）基于Opencv的金纳米探针提取和去噪的模型设计；(1) Model design of gold nanoprobe extraction and denoising based on Opencv;（2）基于ResNet50作为纳米探针夹心结构判定的模型设计；(2) Model design based on ResNet50 as the sandwich structure determination of nanoprobes;（3）使用训练好的基于ResNet50的夹心结构分辨模型和基于Opencv的金纳米探针提取去噪模型对暗场显微镜成像原图进行实时分析和判断；(3) Use the trained ResNet50-based sandwich structure resolution model and the Opencv-based gold nanoprobe extraction and denoising model to analyze and judge the original image of the dark field microscope in real time;所述步骤（1）的模型设计方法如下：The model design method of the step (1) is as follows:a、获取暗场显微镜成像原图；a. Obtain the original image of the dark field microscope;b、将原图由RGB通道转换为HSV通道；b. Convert the original image from RGB channel to HSV channel;c、确定金纳米探针红色特征HSV值，以此上下界限创建掩膜；c. Determine the red characteristic HSV value of the gold nanoprobe, and create a mask with this upper and lower limit;d、将步骤c中创建的掩膜与原图进行与操作；d. Perform AND operation between the mask created in step c and the original image;e、利用膨胀算法将步骤d图像中单个金纳米颗粒分散色块中间的空隙填充，使其变为贴合纳米颗粒的饱满色块；e. Use the expansion algorithm to fill the gap in the middle of the single gold nanoparticle dispersed color block in the image in step d, so that it becomes a full color block that fits the nanoparticles;f、利用连通组件算法获取每个饱满色块，即金纳米探针的二维空间坐标，并用矩形框标注；f. Use the connected component algorithm to obtain each full color block, that is, the two-dimensional spatial coordinates of the gold nanoprobe, and mark it with a rectangular frame;g、自定义函数提取每个金纳米探针保存在指定文件夹中；g. The custom function extracts each gold nanoprobe and saves it in the specified folder;h、根据g中所提取的金纳米探针，计算每一张提取图中Hue值落在红色范围内的像素点个数，得到金纳米探针Hue值得分布规律，从而做二次筛选；h. According to the gold nanoprobes extracted in g, calculate the number of pixels whose Hue values fall within the red range in each extraction image, and obtain the distribution law of gold nanoprobe Hue values, so as to do secondary screening;I、将步骤h中反应的Hue值分布规律作为判断项，补充在基于Opencv的金纳米探针提取和去噪模型中；1. The Hue value distribution law of the reaction in step h is used as a judgment item, and is supplemented in the gold nanoprobe extraction and denoising model based on Opencv;J、将其他拍摄的暗场图像批量送入模型，导出的是金银纳米探针夹心结构、单金纳米探针和极少数带有红色特征的颗粒，将金银纳米探针夹心结构手工标注为bind、单金纳米探针和极少数带有红色特征的颗粒手工标注为nobind两大类，作为深度学习模型的训练数据集。J. Send other dark-field images into the model in batches, and export the gold-silver nanoprobe sandwich structure, single gold nanoprobe and a few particles with red features, and manually label the gold-silver nanoprobe sandwich structure The two categories of bind, single gold nanoprobes, and very few particles with red features were manually labeled as nobind, which were used as training datasets for deep learning models.2.根据权利要求1所述的基于深度学习分析暗场显微镜中纳米探针捕获病毒的方法，其特征在于：所述步骤（2）的模型设计方法如下：2 . The method for analyzing the virus captured by nanoprobes in a dark field microscope based on deep learning analysis according to claim 1 , wherein the model design method of the step (2) is as follows:a、手工标注数据集内的金纳米探针，将金银纳米探针夹心结构手工标注为bind、单金纳米探针和极少数带有红色特征的颗粒手工标注为nobind两大类；a. Manually label the gold nanoprobes in the dataset. The sandwich structure of gold-silver nanoprobes is manually labeled as bind, single gold nanoprobes and a few particles with red features are manually labeled as nobind;b、构建ResNet50深度神经网络模型，模型输入为归一化且带分类标签的图像；b. Build a ResNet50 deep neural network model, and the model input is a normalized and classified image;c、根据ResNet50的模型结构,基于Keras框架定义卷积模块和恒等模块后进行叠加；c. According to the model structure of ResNet50, the convolution module and the identity module are defined based on the Keras framework and then superimposed;d、设置训练参数，利用步骤（1）中的步骤J所得到的数据集进行模型训练；d. Set training parameters, and use the data set obtained in step J in step (1) for model training;e、将训练好的模型应用于实际暗场显微镜成像图像进行测试。e. Apply the trained model to the actual dark field microscope imaging image for testing.3. 根据权利要求2所述的基于深度学习分析暗场显微镜中纳米探针捕获病毒的方法，其特征在于：所述步骤c中的 ResNet50的模型结构包括四个卷积模块和十二个恒等模块。3. the method according to claim 2 based on deep learning analysis in the dark field microscope nano probe captures virus, it is characterized in that: the model structure of the ResNet50 in the described step c comprises four convolution modules and twelve constants. and other modules.4.根据权利要求3所述的基于深度学习分析暗场显微镜中纳米探针捕获病毒的方法，其特征在于：每个卷积模块包括神经网络堆叠主干和带卷积层的侧枝，每条主干包括三个卷积层，每条侧枝包括一个卷积层，每个恒等模块包括神经网络堆叠主干和不带卷积层的侧枝，每条主干包括三个卷积层。4. The method according to claim 3, characterized in that: each convolution module comprises a neural network stacking backbone and a side branch with a convolution layer, and each backbone It includes three convolutional layers, each side branch includes one convolutional layer, and each identity module includes a neural network stacking backbone and side branches without convolutional layers, and each backbone includes three convolutional layers.

Images