Disclosure of Invention
The embodiment of the invention provides an infarct change prediction method and system based on brain parenchyma MRI images, which at least can solve part of problems in the prior art.
In a first aspect of an embodiment of the present invention, there is provided a method for predicting infarct change based on MRI images of brain parenchyma, including:
Acquiring brain parenchyma MRI images of a patient, constructing a multi-mode image set, acquiring metadata information for each image in the multi-mode image set, screening according to a preset screening standard to obtain a first image set, carrying out artifact identification on the first image set, calculating the quality score of each image according to an artifact identification result, screening according to a preset scoring threshold value, generating a second image set, preprocessing the second image set to obtain standard image data, and carrying out feature extraction on the standard image data through multi-scale deep learning to obtain a brain region division mask;
Adding the standard image data and the brain region segmentation mask into a pre-constructed three-dimensional depth residual error convolutional neural network, extracting features through densely connected residual blocks, obtaining multi-mode feature data by combining a two-way multi-scale feature fusion algorithm and a progressive downsampling strategy, constructing topological relations among different brain regions based on the multi-mode feature data, extracting high-order semantic features, obtaining brain region feature data, and selecting the brain region feature data through a feature selection network to obtain high-quality feature data;
And constructing a time sequence feature matrix based on the high-quality feature data, carrying out time sequence modeling by combining an infarct prediction model, distributing weights for each time step by a memory unit in the infarct prediction model, carrying out element-level addition by combining residual connection to generate residual output, splicing the residual output to obtain comprehensive time sequence features, predicting the comprehensive time sequence features by a fully-connected neural network to generate an infarct variation prediction result, carrying out prediction again by combining a prognosis prediction model based on a random forest algorithm, generating a prediction prognosis result by combining clinical data of a patient and dangerous factors, and generating a comprehensive prediction report by combining the infarct variation prediction result and the prediction prognosis result.
In an alternative embodiment of the present invention,
Acquiring brain parenchyma MRI images of a patient and constructing a multi-mode image set, acquiring metadata information for each image in the multi-mode image set, screening according to a preset screening standard to obtain a first image set, carrying out artifact identification on the first image set, calculating a quality score of each image according to an artifact identification result, screening by combining a preset scoring threshold value to generate a second image set, preprocessing the second image set to obtain standard image data, carrying out feature extraction on the standard image data through multi-scale deep learning, and obtaining a brain region division mask, wherein the method comprises the following steps:
acquiring an MRI image of the brain parenchyma of a patient, wherein the MRI image comprises a T1 weighted image, a T2 weighted image, a liquid attenuation inversion recovery sequence image and a diffusion weighted image, performing de-identification processing on the MRI image of the brain parenchyma and constructing the multi-mode image set;
for each image in the multi-mode image set, acquiring metadata information corresponding to the current image, wherein the metadata information comprises image size, resolution and acquisition parameters, screening the images in the multi-mode image set according to preset screening criteria, and removing the images which do not meet the screening criteria to obtain the first image set;
For the images in the first image set, calculating a correlation coefficient between successive slices, estimating motion field removal motion artifacts of an image sequence by combining an optical flow method, determining tissue signal intensity corresponding to each pixel in the images by carrying out intensity statistical analysis on image data, identifying metal artifacts and removing by combining a signal intensity threshold, carrying out edge detection on the images, determining a signal mutation region and a discontinuous region, carrying out truncation artifact identification on the signal mutation region and the discontinuous region, removing the identified truncation artifacts, calculating quality scores of each image by combining the types, the areas and the positions of the artifacts, comparing the quality scores corresponding to each image with a preset quality score threshold, and removing the images with the quality scores lower than the quality score threshold to obtain the second image set;
preprocessing the second image set, removing a non-brain tissue region through a head region segmentation algorithm based on morphological operation, performing image registration through a non-rigid registration algorithm based on mutual information, registering images of different modes to a standard space, combining histogram equalization and a non-local mean value filtering algorithm to improve contrast and signal to noise ratio, obtaining standard image data, performing feature extraction and forward reasoning on the standard image data through a multi-scale convolutional neural network, obtaining a brain region segmentation probability map, and obtaining a brain region segmentation mask through thresholding operation.
In an alternative embodiment of the present invention,
Based on the identified artifacts, the quality score for each image is calculated in combination with the artifact type, area and location as shown in the following equation:
Wherein Qw represents the weighted quality score of the image, lambdaj represents the weight of the jth region of interest, nj represents the number of artifacts identified in the jth region of interest, wti represents the weight coefficient of the ith artifact type, ti represents the type coefficient of the ith artifact, wa represents the weight coefficient of the artifact area,A nonlinear transformation function representing the i-th artifact area, ai representing the area of the artifact region, a representing the area of the current image, wd representing the weight coefficient of the artifact location,The nonlinear transformation function representing the i-th artifact location, Di represents the center-to-image center distance of the artifact region, and D represents half the image diagonal length.
In an alternative embodiment of the present invention,
Adding the standard image data and the brain region segmentation mask into a pre-constructed three-dimensional depth residual convolutional neural network, extracting features through densely connected residual blocks, obtaining multi-mode feature data by combining a two-way multi-scale feature fusion algorithm and a progressive downsampling strategy, constructing topological relations among different brain regions based on the multi-mode feature data, extracting high-order semantic features, obtaining brain region feature data, and selecting the brain region feature data through a feature selection network, wherein obtaining high-quality feature data comprises the following steps:
Adding the standard image data and the brain region division mask into a pre-constructed three-dimensional depth residual error convolution neural network, wherein the three-dimensional depth residual error convolution neural network comprises a plurality of residual error blocks and a transition layer, and each residual error block is connected through a dense connection mechanism;
For input standard image data, extracting features through two independent three-dimensional depth residual error convolutional neural networks, extracting features of each residual error block through two convolutional layers, reusing and directly transmitting the features in gradient through identical mapping and jump connection, and splicing the outputs of all residual error blocks by combining the dense connection mechanism to serve as the outputs of the current residual error blocks to obtain trans-scale features;
Based on the trans-scale features, reducing a pooling window and a step length according to a progressive downsampling strategy, setting a downsampling rate and modifying a reduction speed through superparameters, fusing the trans-scale features among different modalities through self-adaptive summation, and combining the last sampling and splicing operation to align to the same resolution to obtain multi-modality feature data;
Based on the multi-mode feature data, extracting high-order semantic features through a graph convolution neural network, constructing a brain region topological graph by taking anatomical connection and functional association between brain regions as edges, learning feature representation of brain region nodes through graph convolution operation, and determining interaction information and dependency relationship between brain regions;
and adding the brain region characteristic data into a preset characteristic selection network, determining gating weight corresponding to each brain region characteristic data through a gating attention mechanism based on the interaction information and the dependency relationship, sequencing the brain region characteristic data based on the gating weight, selecting the first 20% brain region characteristic data as the high-quality characteristic data, and adding the high-quality characteristic data into a pre-constructed high-quality characteristic set.
In an alternative embodiment of the present invention,
Based on the multi-modal feature data, extracting high-order semantic features through a graph convolution neural network is shown in the following formula:
Wherein X(l+1) represents the node feature matrix of the layer i+1, σ represents the activation function, K represents the order of the current neighbor, K represents the highest order of the aggregated neighbor, C represents the node degree matrix, bk represents the K power of the adjacency matrix B, X(l) represents the node feature matrix of the layer i, and W(l,k) represents the weight matrix of the layer i corresponding to the K-th neighbor.
In an alternative embodiment of the present invention,
Constructing a time sequence feature matrix based on the high-quality feature data, combining an infarct prediction model for time sequence modeling, distributing weights for each time step through a memory unit in the infarct prediction model, performing element-level addition by combining residual connection to generate residual output, splicing the residual output to obtain comprehensive time sequence features, predicting the comprehensive time sequence features through a fully-connected neural network to generate an infarct change prediction result, predicting again through a prognosis prediction model based on a random forest algorithm, combining clinical data of a patient and dangerous factors to generate a prediction prognosis result, and synthesizing the infarct change prediction result and the prediction prognosis result to generate a comprehensive prediction report, wherein the method comprises the following steps of:
Based on the high-quality feature data, constructing a corresponding time sequence feature matrix for feature dimensions in each time step, wherein rows in the time sequence feature matrix represent feature vectors of the current time step, and columns represent values of the current high-quality feature data in different time steps;
Adding the time sequence feature matrix into the infarction prediction model, determining forgetting information in each time step through a memory unit, generating corresponding time sequence output based on the forgetting information, distributing corresponding forgetting weight for the current time step according to the duty ratio of the forgetting information in original high-quality feature data, generating initial output of the current forgetting unit based on the forgetting weight, adding the output of the current memory unit to the next layer in combination with a residual mechanism, generating the residual output through element-by-element addition for the initial output of each memory unit, splicing the residual output into a hidden state corresponding to the current time sequence feature matrix, and obtaining the comprehensive time sequence feature;
Inputting the comprehensive time sequence characteristics into a fully-connected neural network, carrying out matrix multiplication on a fully-connected weight matrix in the fully-connected neural network and the comprehensive time sequence characteristics, determining the output of a current fully-connected layer by combining a preset bias vector and an activation function, repeatedly calculating and carrying out output prediction by the activation layer to obtain an initial infarction change prediction result, determining a potential characteristic space by combining a preset generation countermeasure network based on the initial infarction change prediction result, generating a high-quality synthetic sample by combining a generator network and a discriminator network, adding the high-quality synthetic sample into the fully-connected neural network, generating a corresponding synthetic prediction result, comparing the corresponding synthetic prediction result with a real label, dynamically adjusting super parameters in the fully-connected neural network according to the comparison result, and predicting the comprehensive time sequence characteristics by using the fully-connected neural network to obtain the infarction change prediction result;
Generating a training set based on the infarct change prediction result by combining self-help sampling and random feature selection, generating a plurality of decision trees by a prognosis prediction model based on a random forest algorithm, adding elements in the training set into the decision trees for training to obtain the prognosis prediction model, adding the infarct change prediction result into the prognosis prediction model, predicting by combining imaging features fused with clinical data and dangerous factors of a current patient, obtaining a prediction prognosis result, and generating a comprehensive prediction report according to the prediction prognosis result and the infarct change prediction result.
In an alternative embodiment of the present invention,
The corresponding loss functions of the generator network and the arbiter network are shown as follows:
LG＝-E[log(H(G(z)))]+λ·||G(z)-x||1;
wherein LG represents a generator loss value, E represents a mathematical expectation, H represents a discriminator network, G represents a generator network, λ represents regularized term weights, x represents a random noise vector, z represents a random noise vector, and G (z) represents an output result of the generator;
Wherein LH represents the arbiter loss value, μ represents the gradient penalty term weight,The output result of the H (x) discriminator is represented by the gradient of the random noise vector x.
In a second aspect of embodiments of the present invention, there is provided an infarct change prediction system based on MRI images of brain parenchyma, comprising:
The device comprises a first unit, a second unit and a third unit, wherein the first unit is used for acquiring brain parenchyma MRI images of a patient and constructing a multi-mode image set, acquiring metadata information for each image in the multi-mode image set, screening according to a preset screening standard to obtain a first image set, performing artifact identification on the first image set, calculating the quality score of each image according to an artifact identification result, screening according to a preset scoring threshold value to generate a second image set, preprocessing the second image set to obtain standard image data, and performing feature extraction on the standard image data through multi-scale deep learning to obtain a brain region segmentation mask;
The second unit is used for adding the standard image data and the brain region division mask into a pre-constructed three-dimensional depth residual error convolutional neural network, extracting features through densely connected residual blocks, obtaining multi-mode feature data by combining a two-way multi-scale feature fusion algorithm and a progressive downsampling strategy, constructing topological relations among different brain regions based on the multi-mode feature data, extracting high-order semantic features, obtaining brain region feature data, and selecting the brain region feature data through a feature selection network to obtain high-quality feature data;
And the third unit is used for constructing a time sequence feature matrix based on the high-quality feature data, combining an infarct prediction model for time sequence modeling, distributing weights for each time step through a memory unit in the infarct prediction model, combining residual connection for element-level addition to generate residual output, splicing the residual output to obtain comprehensive time sequence features, predicting the comprehensive time sequence features through a fully-connected neural network to generate an infarct variation prediction result, predicting again through a prognosis prediction model based on a random forest algorithm, combining clinical data of a patient and dangerous factors to generate a prediction prognosis result, and combining the infarct variation prediction result and the prediction prognosis result to generate a comprehensive prediction report.
In a third aspect of an embodiment of the present invention,
There is provided an electronic device including:
A processor;
A memory for storing processor-executable instructions;
wherein the processor is configured to invoke the instructions stored in the memory to perform the method described previously.
In a fourth aspect of an embodiment of the present invention,
There is provided a computer readable storage medium having stored thereon computer program instructions which, when executed by a processor, implement the method as described above.
According to the invention, high-quality standard image data are obtained through the steps of multi-mode image fusion, quality control, preprocessing and the like, multi-scale deep learning is adopted to extract brain region characteristics, multi-mode characteristics are mined by combining a three-dimensional residual network, a brain region topological relation is constructed, high-quality characteristics are selected, an end-to-end characteristic engineering flow is formed, the image phenotype of cerebral infarction can be comprehensively and accurately characterized, a solid foundation is laid for subsequent prediction, the time dependency relation of a memory unit and residual connection learning cerebral infarction is adopted, the evolution characteristics of a focus at different stages are mined, comprehensive time sequence characteristics are generated, dynamic change tracks of cerebral infarction from an acute stage to a chronic stage can be described, prediction continuity and consistency are improved, comprehensive modeling is carried out through a machine learning algorithm such as random forest and the like, disease risks are analyzed from multiple dimensions, image-clinical association modes are found, accuracy of prognosis prediction is improved, an accurate prognosis evaluation driven by image group is realized, an accurate prognosis doctor is intuitively presented through generating a comprehensive prediction report, an image change trend, a prediction result, key image characteristic and the like of a patient is provided with quantitative index form, an auxiliary diagnosis opinion is fast, an intelligent diagnosis is realized, and an intelligent diagnosis is provided for the diagnosis is realized, and an intelligent diagnosis is applied to the medical diagnosis is provided in the fields, and has high medical diagnosis and clinical diagnosis is provided in the fields, and has high medical diagnosis and medical diagnosis has high clinical application value and intelligent diagnosis and medical diagnosis has high-aid.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are only some embodiments of the present invention, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
The technical scheme of the invention is described in detail below by specific examples. The following embodiments may be combined with each other, and some embodiments may not be repeated for the same or similar concepts or processes.
Fig. 1 is a flowchart of an infarct change prediction method based on brain parenchyma MRI images according to an embodiment of the invention, as shown in fig. 1, the method includes:
S1, acquiring brain parenchyma MRI images of a patient and constructing a multi-mode image set, acquiring metadata information for each image in the multi-mode image set, screening according to a preset screening standard to obtain a first image set, carrying out artifact identification on the first image set, calculating a quality score of each image according to an artifact identification result, screening according to a preset scoring threshold value, generating a second image set, preprocessing the second image set to obtain standard image data, and carrying out feature extraction on the standard image data through multi-scale deep learning to obtain brain region division masks;
The brain parenchyma MRI image refers to image data of brain parenchyma obtained by using a Magnetic Resonance Imaging (MRI) technology, and is commonly used for checking and analyzing brain structures and diagnosing lesions (such as tumors, infarcts and the like), the metadata information is data describing the data, relevant information related to the MRI image such as acquisition time, equipment parameters, imaging conditions, patient information and the like is provided, storage, retrieval and processing of the data are convenient, an artifact identification result refers to detection and identification of artifacts caused by imaging equipment or motion in the MRI image, the artifacts possibly interfere with image quality and diagnosis result, and a brain region division mask refers to a binary image of each brain region (such as brain cortex, white matter, gray matter and the like) extracted by a division algorithm in the MRI image, and is used for subsequent analysis and processing.
In an alternative embodiment of the present invention,
Acquiring brain parenchyma MRI images of a patient and constructing a multi-mode image set, acquiring metadata information for each image in the multi-mode image set, screening according to a preset screening standard to obtain a first image set, carrying out artifact identification on the first image set, calculating a quality score of each image according to an artifact identification result, screening by combining a preset scoring threshold value to generate a second image set, preprocessing the second image set to obtain standard image data, carrying out feature extraction on the standard image data through multi-scale deep learning, and obtaining a brain region division mask, wherein the method comprises the following steps:
acquiring an MRI image of the brain parenchyma of a patient, wherein the MRI image comprises a T1 weighted image, a T2 weighted image, a liquid attenuation inversion recovery sequence image and a diffusion weighted image, performing de-identification processing on the MRI image of the brain parenchyma and constructing the multi-mode image set;
for each image in the multi-mode image set, acquiring metadata information corresponding to the current image, wherein the metadata information comprises image size, resolution and acquisition parameters, screening the images in the multi-mode image set according to preset screening criteria, and removing the images which do not meet the screening criteria to obtain the first image set;
For the images in the first image set, calculating a correlation coefficient between successive slices, estimating motion field removal motion artifacts of an image sequence by combining an optical flow method, determining tissue signal intensity corresponding to each pixel in the images by carrying out intensity statistical analysis on image data, identifying metal artifacts and removing by combining a signal intensity threshold, carrying out edge detection on the images, determining a signal mutation region and a discontinuous region, carrying out truncation artifact identification on the signal mutation region and the discontinuous region, removing the identified truncation artifacts, calculating quality scores of each image by combining the types, the areas and the positions of the artifacts, comparing the quality scores corresponding to each image with a preset quality score threshold, and removing the images with the quality scores lower than the quality score threshold to obtain the second image set;
preprocessing the second image set, removing a non-brain tissue region through a head region segmentation algorithm based on morphological operation, performing image registration through a non-rigid registration algorithm based on mutual information, registering images of different modes to a standard space, combining histogram equalization and a non-local mean value filtering algorithm to improve contrast and signal to noise ratio, obtaining standard image data, performing feature extraction and forward reasoning on the standard image data through a multi-scale convolutional neural network, obtaining a brain region segmentation probability map, and obtaining a brain region segmentation mask through thresholding operation.
The optical flow method is an algorithm for tracking pixel movement in an image sequence, the motion of an object is analyzed by calculating pixel intensity changes of the image, the motion detection or tracking in a dynamic image is used, the intensity statistical analysis is used for carrying out statistical processing on the signal intensity of each pixel or voxel in the MRI image, indexes such as mean value and variance are calculated, the indexes are commonly used for identifying abnormal areas or feature extraction, the signal mutation area is an area with significant changes in the signal intensity in the MRI image and is usually related to lesions or abnormal tissues, the damage area is possibly predicted, the cut-off artifact is an artifact caused by signal cut-off, incomplete image acquisition or hardware faults, the accuracy of MRI image quality and diagnosis is influenced, the head area segmentation algorithm based on morphological operation is an image processing method, the areas of the head are segmented by using mathematical morphological operation (such as expansion, corrosion, division and closure operation), the boundaries of the head area are removed, the non-rigid registration algorithm based on mutual information is used for aligning the images under different conditions or under different modes, the non-rigid transformation is used for accurately calculating the probability of image segmentation of the brain area by using a mean value, the image is compared with the specific image with the brain area by using a specific noise reduction algorithm, the probability of the image is calculated by comparing the local area with the brain area with the specific noise reduction probability.
Acquiring brain parenchyma MRI images of a patient, performing de-identification processing on a plurality of modes including a T1 weighted image, a T2 weighted image, a liquid attenuation inversion recovery sequence image (FLAIR), a Diffusion Weighted Image (DWI) and the like, deleting privacy information such as patient names, IDs and the like contained in the images, constructing a multi-mode image dataset, traversing each image in the multi-mode image dataset, extracting metadata information of the images, including the size, resolution, acquisition parameters and the like of the images, screening the images according to preset screening criteria such as the size, resolution, acquisition equipment model and the like of the images, and removing the images which do not meet the standard to obtain a first image dataset;
Judging whether artifacts caused by patient movement exist in each image sequence in the first image set by calculating a correlation coefficient between successive slices, if so, estimating a motion field of the image sequence by using an optical flow method, performing motion correction on the images, removing the motion artifacts, performing intensity statistical analysis on corrected image data, calculating the signal intensity of each pixel, combining a preset signal intensity threshold value, identifying an area with abnormally high signal intensity, judging as a metal artifact area, removing the metal artifact area, performing edge detection on the images, positioning a signal intensity mutation area and a discontinuous area, analyzing morphological characteristics of the areas, identifying artifacts possibly generated due to cutting off and the like, and removing the artifacts;
Calculating the quality score of each image according to the identified artifact and combining the type, area, position and other information, comparing the quality score with a preset threshold value, removing images with the quality score lower than the threshold value to obtain a second image set, dividing the head region of the image by utilizing a morphological operation-based algorithm, removing non-brain tissue regions such as scalp and skull, registering images of different modes into a standard space through a non-rigid registration algorithm based on mutual information to realize standardization of the images, enhancing the contrast of the registered images by utilizing a histogram equalization algorithm, denoising by utilizing a non-local mean value filtering algorithm to improve the signal-to-noise ratio, obtaining standardized image data, inputting the standardized image data into a pre-trained multi-scale convolutional neural network, obtaining the probability that each pixel belongs to different brain regions through forward reasoning, generating a brain region division probability map, thresholding the probability map, and marking the pixels with the probability larger than the threshold value as corresponding brain regions to obtain a final brain region division mask;
For example, a patient Zhang Mou suspected of alzheimer's disease is received by a neurology department of a hospital, for explicit diagnosis, MRI is performed on the patient Zhang Mou, multi-mode brain images including a T1 weighted image, a T2 weighted image, a FLAIR image and a DWI image are obtained, the images are de-identified, information such as patient name and ID contained in the images is deleted, a multi-mode image dataset is constructed, each image in the dataset is traversed, metadata information such as size, resolution and acquisition parameters of the images is extracted, and 2T 2 weighted images with resolution not meeting requirements are removed according to preset screening criteria, so as to obtain a first image set;
Calculating the correlation coefficient between the continuous slices of each sequence in the first image set, finding out that 3 areas with abnormally reduced correlation coefficient exist in the FLAIR sequence, judging as motion artifact, estimating the motion field of the FLAIR sequence by using an optical flow method, performing motion correction on the motion field, performing intensity statistics on the corrected image, finding out that 1T 1 weighted image has abnormally increased signal intensity in the left temporal lobe area, judging as metal artifact and removing the metal artifact;
Performing edge detection on all images, co-locating the 8 signal mutation areas and the 5 discontinuous areas, performing morphological analysis to judge that the 2 signal mutation areas and the 1 discontinuity areas are artifacts caused by scanning truncation, removing the artifacts, calculating the quality score of each image according to an artifact identification result, removing the artifacts with the quality score of 1 DWI image being 0.6 and lower than a preset threshold value of 0.8, finally obtaining a second image set containing 5T 1 weighted images, 4T 2 weighted images, 3 FLAIR images and 2 DWI images, preprocessing the second image set, removing a non-brain tissue area by using morphological operation, registering all the images to an MNI152 standard space by using a registration algorithm based on mutual information, performing histogram equalization on the registered images, enhancing the contrast of gray matter, denoising the images by using non-local mean filtering, and finally obtaining standardized image data;
The standardized images are input into a pretrained multi-scale convolutional neural network, a probability map of each pixel belonging to gray matter, white matter and cerebrospinal fluid is obtained for reasoning, thresholding is carried out on the probability map, a brain region division mask is generated, quantitative calculation finds that the sea horse volume of a patient Zhang Mou is obviously smaller than that of normal people in the same age period, and a doctor finally diagnoses the patient as Alzheimer disease in combination with clinical symptoms.
In the embodiment, through a series of preprocessing operations such as de-labeling, quality screening and artifact removal on the multi-mode MRI image, the standardization degree of data is effectively improved, the problem of unstable segmentation performance caused by data quality difference is reduced, automatic head extraction is realized by using morphological operation, automatic cross-modal registration is realized by using a registration algorithm based on mutual information, the processing efficiency is greatly improved, complementary biophysical information is provided for brain soft tissue structures by images of different modes, brain tissues can be described from different angles by multi-mode fusion, the performance of the segmentation algorithm is improved, in particular the segmentation effect of tissues with complex structures such as gray matter and lack of contrast is facilitated, in conclusion, the embodiment realizes robust and efficient brain segmentation, can quantitatively describe morphological characteristics of brain regions, provides a new method and a new method for imaging auxiliary diagnosis of brain diseases, is hopeful to improve the accuracy and efficiency of disease diagnosis, and has wide application prospect.
In an alternative embodiment of the present invention,
Based on the identified artifacts, the quality score for each image is calculated in combination with the artifact type, area and location as shown in the following equation:
Wherein Qw represents the weighted quality score of the image, lambdaj represents the weight of the jth region of interest, nj represents the number of artifacts identified in the jth region of interest, wti represents the weight coefficient of the ith artifact type, ti represents the type coefficient of the ith artifact, wa represents the weight coefficient of the artifact area,A nonlinear transformation function representing the i-th artifact area, ai representing the area of the artifact region, a representing the area of the current image, wd representing the weight coefficient of the artifact location,The nonlinear transformation function representing the i-th artifact location, Di represents the center-to-image center distance of the artifact region, and D represents half the image diagonal length.
In this embodiment, the quality evaluation formula integrates multiple local factors such as artifact types, areas, positions, and the like, so that local detail differences of image quality can be more comprehensively and carefully described, the quality score is a continuous numerical value, quality of different images can be conveniently compared and sequenced, subsequent quality screening and filtering are facilitated, the weighted strategy enables quality evaluation to pay more attention to areas with important significance for disease diagnosis, artifacts of areas with infrequent occurrence of lesions are relatively insensitive, pertinence of quality evaluation is improved, different weight coefficients are introduced to different types of artifacts, sensitivity of quality scores to different artifact types can be automatically adjusted, so that evaluation results are more in line with subjective feelings of actual image quality.
S2, adding the standard image data and the brain region division mask into a pre-constructed three-dimensional depth residual error convolutional neural network, extracting features through densely connected residual blocks, obtaining multi-mode feature data by combining a two-way multi-scale feature fusion algorithm and a progressive downsampling strategy, constructing topological relations among different brain regions based on the multi-mode feature data, extracting high-order semantic features, obtaining brain region feature data, and selecting the brain region feature data through a feature selection network to obtain high-quality feature data;
The three-dimensional depth residual convolutional neural network (3D ResNet) is a deep learning model, combines a convolutional neural network and a residual network, is used for processing three-dimensional image data (such as MRI images) and extracting high-level features, the two-way multi-scale feature fusion algorithm uses a two-way structure and fuses image features on multiple scales, the recognition capability of the model on brain areas or focuses of different sizes and shapes is enhanced, the three-dimensional depth residual convolutional neural network is suitable for medical image analysis, the progressive downsampling strategy is to gradually reduce the resolution ratio when processing the images so as to reduce the calculation burden, simultaneously preserve key details in the high-resolution images and is commonly used for segmentation and classification tasks, the brain area feature data refer to feature information, such as volume, density, signal strength and the like, of each brain area extracted from the MRI images, and is commonly used for disease detection or research, and the feature selection network is a deep learning model and is used for automatically selecting the most relevant features, reducing the input dimension and improving the prediction performance of the model, especially under a large-scale feature space.
In an alternative embodiment of the present invention,
Adding the standard image data and the brain region segmentation mask into a pre-constructed three-dimensional depth residual convolutional neural network, extracting features through densely connected residual blocks, obtaining multi-mode feature data by combining a two-way multi-scale feature fusion algorithm and a progressive downsampling strategy, constructing topological relations among different brain regions based on the multi-mode feature data, extracting high-order semantic features, obtaining brain region feature data, and selecting the brain region feature data through a feature selection network, wherein obtaining high-quality feature data comprises the following steps:
Adding the standard image data and the brain region division mask into a pre-constructed three-dimensional depth residual error convolution neural network, wherein the three-dimensional depth residual error convolution neural network comprises a plurality of residual error blocks and a transition layer, and each residual error block is connected through a dense connection mechanism;
For input standard image data, extracting features through two independent three-dimensional depth residual error convolutional neural networks, extracting features of each residual error block through two convolutional layers, reusing and directly transmitting the features in gradient through identical mapping and jump connection, and splicing the outputs of all residual error blocks by combining the dense connection mechanism to serve as the outputs of the current residual error blocks to obtain trans-scale features;
Based on the trans-scale features, reducing a pooling window and a step length according to a progressive downsampling strategy, setting a downsampling rate and modifying a reduction speed through superparameters, fusing the trans-scale features among different modalities through self-adaptive summation, and combining the last sampling and splicing operation to align to the same resolution to obtain multi-modality feature data;
Based on the multi-mode feature data, extracting high-order semantic features through a graph convolution neural network, constructing a brain region topological graph by taking anatomical connection and functional association between brain regions as edges, learning feature representation of brain region nodes through graph convolution operation, and determining interaction information and dependency relationship between brain regions;
and adding the brain region characteristic data into a preset characteristic selection network, determining gating weight corresponding to each brain region characteristic data through a gating attention mechanism based on the interaction information and the dependency relationship, sequencing the brain region characteristic data based on the gating weight, selecting the first 20% brain region characteristic data as the high-quality characteristic data, and adding the high-quality characteristic data into a pre-constructed high-quality characteristic set.
The identity mapping refers to a mapping function with input equal to output, and is commonly used in a residual error network structure, and unnecessary layers can be skipped by the network through the identity mapping, so that the problem of gradient disappearance is alleviated
Adding normalized multi-mode MRI image data and brain region division masks obtained through a convolutional neural network into a pre-constructed three-dimensional depth residual error convolutional neural network for feature extraction, wherein the network consists of a plurality of residual error blocks and transition layers, the residual error blocks are connected in a dense connection mode to realize feature reuse and gradient direct transmission, and the input standard image data are respectively sent into two independent three-dimensional residual error networks for feature extraction, each residual error block comprises two convolutional layers, and feature reuse is realized through identical mapping and jump connection;
The output of all residual blocks is spliced together through dense connection to be used as the output of a current residual block, so that cross-scale image characteristics are obtained, on the basis of the cross-scale characteristics, a feature map is downsampled through a progressive downsampling strategy, the downsampling rate and the shrinking speed are controlled through setting super parameters, the features extracted in different modes are fused through self-adaptive summation operation, and the feature map is aligned to the same resolution through upsampling and splicing operation, so that multi-mode fused feature data are obtained;
Inputting multi-modal feature data into a graph convolution neural network, extracting high-order semantic features, taking brain regions as nodes of a graph, taking anatomical connection and functional association between the brain regions as edges of the graph, constructing a brain region topological graph, learning feature representation of each brain region node through graph convolution operation, mining interaction information and dependency relationship between the brain regions, sending the learned brain region feature data into a preset feature selection network for high-quality feature selection, calculating importance weights of each brain region feature according to the interaction information and the dependency relationship between the brain regions by using a gating attention mechanism, sequencing the brain region features according to weight magnitudes, selecting the first 20% of features as high-quality features, and adding the high-quality features into a preset high-quality feature set.
The method comprises the steps of collecting structural MRI and functional MRI data of 100 healthy volunteers by a research team, mining key brain areas related to memory capacity and connection modes thereof through multi-mode image fusion and brain network analysis, providing imaging marks for early warning of memory impairment diseases, carrying out standardized preprocessing on all tested MRI data to obtain standardized T1, DTI and fMRI images, carrying out brain area division on the T1 images by using a pre-trained multi-scale convolutional neural network to obtain segmentation masks of 90 brain areas, inputting the standardized images and the segmentation masks into a three-dimensional residual convolutional neural network for feature extraction, wherein each residual block comprises 2 convolutional layers, and the residual blocks are connected in a dense connection mode to respectively extract features of images of three modes of T1, DTI and fMRI to obtain a cross-scale image feature map;
the method comprises the steps of performing downsampling on a feature map by using a progressive downsampling strategy, gradually increasing the downsampling rate from 1 to 4, fusing the feature maps of three modes by using a self-adaptive summation algorithm, aligning the features to the same resolution by upsampling to obtain fused multi-mode feature data, constructing a topological map of 90 brain areas according to an anatomical template, inputting the multi-mode feature data into a graph convolution network for graph embedding, extracting high-order features of each brain area node by graph convolution operation, learning interaction modes among brain areas, inputting the learned brain area features into a gated attention network for high-quality feature selection, automatically calculating importance weights of the features of each brain area by the network according to interaction intensity and dependency among the brain areas, sorting all the brain area features according to the weights from large to small, selecting the first 20% of features as high-quality features, forming a key brain area feature set related to memory capacity, finally determining 10 key brain areas related to memory capacity including hippocampus, endothelial layers, postbuckle strap back and the like, mining the key brain areas related to the memory capacity, and establishing a function of the key brain areas related to memory capacity, developing the key brain area feature sets, and establishing a clinical interaction mode, and further verifying the clinical interaction mode.
In the embodiment, the multi-modal fusion can comprehensively utilize brain structure and function information provided by different physical imaging mechanisms, the characteristics of brain regions are characterized by multiple angles, the accuracy and the robustness of subsequent analysis are improved, the low-level residual block extracts local detail features, the high-level residual block extracts global abstract features, the features with different scales are spliced in channel dimensions, the richness and the distinguishability of feature representation can be improved by simultaneously utilizing the local and global feature information, the adaptive fusion can automatically adjust the fusion strategy according to the local mode and the global statistics of feature graphs, so that the multi-modal feature representation with more consistent semantics is obtained, the brain networking method considers anatomical connection and functional dependence among brain regions, the brain network features which are higher in level and more in line with the cognitive mechanism are extracted by modeling brain region interaction modes, the distributed processing characteristics of the brain cognitive function are facilitated to be characterized by the attention mechanism, the high-quality feature subset with more discriminance and stability can be selected according to task target self-adaptive adjustment feature weights, the pertinence and the effectiveness of subsequent analysis are improved, the embodiment fully utilizes the advantages of deep learning features in extraction and the global statistics, the deep learning feature extraction can be directly combined with the brain images in the aspects of brain network, the aspects of the deep learning feature extraction and the deep learning, the deep learning feature is more important in the aspects, the brain images can be more studied, the brain images have the important and the important characteristics are more important in the aspects, and the important images are directly researched, and have the important and has the important and high clinical and clinical aspects, and has the important clinical aspects.
In an alternative embodiment of the present invention,
Based on the multi-modal feature data, extracting high-order semantic features through a graph convolution neural network is shown in the following formula:
Wherein X(l+1) represents the node feature matrix of the layer i+1, σ represents the activation function, K represents the order of the current neighbor, K represents the highest order of the aggregated neighbor, C represents the node degree matrix, bk represents the K power of the adjacency matrix B, X(l) represents the node feature matrix of the layer i, and W(l,k) represents the weight matrix of the layer i corresponding to the K-th neighbor.
In the embodiment, by aggregating the characteristic information of neighbors with different orders, the multi-hop connection relation and long-range dependence among the nodes can be captured, the high-order semantic features contained in the brain network are extracted, the functional interaction modes among brain regions are more comprehensively described, the optimal characteristic aggregation strategy can be automatically found through the joint optimization of the node features and the neighbor weights in an end-to-end training mode, the distinguishing property and the robustness of the characteristic representation are improved, the node features can be integrated with more structural information and semantic information layer by layer, and are abstracted layer by layer, finally, the high-level brain network characteristic representation is finally obtained.
S3, constructing a time sequence feature matrix based on the high-quality feature data, carrying out time sequence modeling by combining an infarct prediction model, distributing weights for each time step by a memory unit in the infarct prediction model, carrying out element-level addition by combining residual connection to generate residual output, splicing the residual output to obtain comprehensive time sequence features, predicting the comprehensive time sequence features by a fully-connected neural network to generate an infarct variation prediction result, carrying out prediction again by a prognosis prediction model based on a random forest algorithm, generating a prediction prognosis result by combining clinical data of a patient and dangerous factors, and synthesizing the infarct variation prediction result and the prediction prognosis result to generate a comprehensive prediction report.
The time sequence feature matrix refers to features of time sequence data expressed in a matrix form, rows of the matrix represent time points, columns represent different features and are used for time sequence prediction or analysis, the infarction prediction model is a model for predicting cerebral infarction risk, based on historical data and image features of a patient, the possibility of occurrence of infarction is predicted by combining statistical analysis or a machine learning algorithm, the memory unit refers to an information unit used for storing and processing long-term dependency relationship in a circulating neural network (such as LSTM or GRU) and is used for processing time sequence data, the prognosis prediction model based on a random forest algorithm is used for predicting prognosis (such as rehabilitation condition, complication occurrence and the like) of diseases according to the features and the medical history data of the patient, and the risk factors refer to features or conditions for increasing the probability of occurrence of the diseases and possibly include age, life habit, gene and the like.
In an alternative embodiment of the present invention,
Constructing a time sequence feature matrix based on the high-quality feature data, combining an infarct prediction model for time sequence modeling, distributing weights for each time step through a memory unit in the infarct prediction model, performing element-level addition by combining residual connection to generate residual output, splicing the residual output to obtain comprehensive time sequence features, predicting the comprehensive time sequence features through a fully-connected neural network to generate an infarct change prediction result, predicting again through a prognosis prediction model based on a random forest algorithm, combining clinical data of a patient and dangerous factors to generate a prediction prognosis result, and synthesizing the infarct change prediction result and the prediction prognosis result to generate a comprehensive prediction report, wherein the method comprises the following steps of:
Based on the high-quality feature data, constructing a corresponding time sequence feature matrix for feature dimensions in each time step, wherein rows in the time sequence feature matrix represent feature vectors of the current time step, and columns represent values of the current high-quality feature data in different time steps;
Adding the time sequence feature matrix into the infarction prediction model, determining forgetting information in each time step through a memory unit, generating corresponding time sequence output based on the forgetting information, distributing corresponding forgetting weight for the current time step according to the duty ratio of the forgetting information in original high-quality feature data, generating initial output of the current forgetting unit based on the forgetting weight, adding the output of the current memory unit to the next layer in combination with a residual mechanism, generating the residual output through element-by-element addition for the initial output of each memory unit, splicing the residual output into a hidden state corresponding to the current time sequence feature matrix, and obtaining the comprehensive time sequence feature;
Inputting the comprehensive time sequence characteristics into a fully-connected neural network, carrying out matrix multiplication on a fully-connected weight matrix in the fully-connected neural network and the comprehensive time sequence characteristics, determining the output of a current fully-connected layer by combining a preset bias vector and an activation function, repeatedly calculating and carrying out output prediction by the activation layer to obtain an initial infarction change prediction result, determining a potential characteristic space by combining a preset generation countermeasure network based on the initial infarction change prediction result, generating a high-quality synthetic sample by combining a generator network and a discriminator network, adding the high-quality synthetic sample into the fully-connected neural network, generating a corresponding synthetic prediction result, comparing the corresponding synthetic prediction result with a real label, dynamically adjusting super parameters in the fully-connected neural network according to the comparison result, and predicting the comprehensive time sequence characteristics by using the fully-connected neural network to obtain the infarction change prediction result;
Generating a training set based on the infarct change prediction result by combining self-help sampling and random feature selection, generating a plurality of decision trees by a prognosis prediction model based on a random forest algorithm, adding elements in the training set into the decision trees for training to obtain the prognosis prediction model, adding the infarct change prediction result into the prognosis prediction model, predicting by combining imaging features fused with clinical data and dangerous factors of a current patient, obtaining a prediction prognosis result, and generating a comprehensive prediction report according to the prediction prognosis result and the infarct change prediction result.
The forgetting information refers to the phenomenon that a model cannot remember long-term dependence due to overlong time span or information loss in the process of weight updating in a cyclic neural network (such as LSTM), the decision tree is a classification or regression algorithm, and a tree structure is constructed recursively by dividing data into a plurality of subsets for prediction or decision, so that the decision tree is commonly used in the construction of the model with strong interpretation.
For each time step, taking a characteristic vector of the current time step as a row of a matrix, taking high-quality characteristic values of different time steps as columns of the matrix to form a time sequence characteristic matrix, inputting the time sequence characteristic matrix into a memory unit of an infarction prediction model, determining forgetting information of each time step through the memory unit, generating corresponding time sequence output based on the forgetting information, distributing forgetting weight for the current time step according to the duty ratio of the forgetting information in the original high-quality characteristic, generating initial output of the current forgetting unit based on the forgetting weight, adopting a residual mechanism, adding the output of the current memory unit into a next layer of memory unit, generating residual output by adding element by element for the initial output of each memory unit, taking the residual output of all the time steps as a hidden state corresponding to the current time sequence characteristic matrix, and splicing the residual outputs of all the time steps to obtain comprehensive time sequence characteristics;
Multiplying the full-connection weight matrix with the comprehensive time sequence characteristics through matrix multiplication, determining the output of the current full-connection layer by combining the bias vector and the activation function, repeating full-connection calculation for a plurality of times, passing through the activation layer to obtain an initial infarction change prediction result, determining a potential characteristic space by generating an countermeasure network based on the initial prediction result, generating a high-quality synthetic sample by utilizing a generator network and a discriminator network, adding the synthetic sample into the full-connection network for prediction, comparing the synthetic prediction result with a real label, and dynamically adjusting the super-parameters of the full-connection network according to the comparison result to finally obtain the infarction change prediction result;
Based on the infarct change prediction result, self-help sampling and random feature selection are adopted to generate a training set, a random forest algorithm is used to construct a prognosis prediction model, a plurality of decision trees are trained according to the training set to form a random forest prediction model, the infarct change prediction result is input into the prognosis prediction model, and is predicted by combining with imaging features of clinical data, dangerous factors and the like of a patient, so that a prognosis prediction result is obtained, and a comprehensive prediction report is generated according to the infarct change prediction result and the prognosis prediction result, so that reference is provided for clinical diagnosis and treatment;
For example, a neurological department of a trimethyl hospital receives and treats a 65-year-old male ischemic cerebral apoplexy patient, symptoms are weakness of a right limb when the patient is admitted, the patient is admitted and subjected to a post-admission MRI examination, high-quality brain region characteristic data are extracted according to clinical data records and sequence MRI images, 7 time points (1 st, 3 rd, 7 th, 14 th, 30 th, 90 th and 180 th days) of MRI characteristics are constructed into a time sequence characteristic matrix, the matrix dimension is 7×128, the time sequence characteristic matrix is input into an LSTM network containing 3 layers and 128 units of each layer, the LSTM memory unit controls memory updating through a forgetting door, forgetting weight is generated according to forgetting proportions of the high-quality characteristics in different time steps, the hidden state and the memory state at the current moment are updated according to the input at the current moment, the hidden state and the memory state at the last moment, residual connection is introduced between the LSTM layers, the output of the previous layer and the output of the current layer are added element by element, residual output of the current time steps is generated, the dimension is 128, and the residual output of 7 time steps is spliced in the time dimension, and the comprehensive time sequence characteristic of 1024 dimensions is obtained;
The comprehensive time sequence features pass through a full-connection network containing 3 hidden layers (1024, 512 and 256 neurons), a ReLU activation function is adopted, a network output layer adopts a Sigmoid function to generate an initial infarction change probability prediction value, a countermeasure network is built and generated, the countermeasure network comprises 1 generator (3 layers of full-connection) and 1 discriminator (4 layers of full-connection), a potential feature space is extracted by training and generating the countermeasure network based on a real sample and the initial prediction value, 500 synthetic samples are generated, the synthetic samples are input into the full-connection network and cross entropy loss is calculated with the real tag, parameters of the full-connection network are optimized through back propagation, and finally the prediction accuracy of the full-connection network on a test set reaches 92%, so that an infarction change probability prediction result is obtained;
Sampling 500 samples from original data by self-help sampling is adopted as a new training set, a sub-feature set is constructed by randomly selecting 30% of feature dimensions, a random forest model comprising 100 decision trees is trained based on the new training set, a prognosis prediction model is obtained, infarct change prediction results of a patient are input into the random forest model, 11 clinical dangerous factors such as age, sex, smoking history and hypertension of the patient are combined, and 90-day functional prognosis of the patient is predicted by using 5 imaging features such as NIHSS score and Barthel index at the time of admission, mRS score is obtained and is relatively good, comprehensive prediction reports of the patient are generated, the content comprises basic information of the patient, clinical data, dangerous factors and the like, MRI imaging features at the time of admission, infarct change trend prediction curves of 7 time points show that infarct volume reaches a peak value at day 3 and then gradually decreases, the functional prognosis result is mRS2 score at the time of 90 days, and standardized treatment and rehabilitation training are recommended to be continued.
In the embodiment, the sequential characteristic representation can delineate the continuous change track of a focus, the implicit time dependency relationship is extracted, richer characteristic information is provided for prediction analysis, residual learning directly transmits the front layer characteristic to the rear layer, reuse and optimization of the characteristic are realized, convergence can be accelerated, accuracy of characteristic representation can be improved, hidden states of different time steps are added element by element through introducing residual connection between LSTM layers, residual outputs of all time steps are spliced to form comprehensive sequential characteristics, local and global sequential information can be aggregated on different time scales, more accurate and comprehensive cerebral infarction evolution characteristic representation is constructed, a basis for subsequent prediction analysis is provided, a random forest model can automatically discover key imaging indexes and clinical dangerous factors which influence prognosis, prediction results of each decision tree are synthesized through a voting mechanism, deviation of individual decision trees is reduced, reliability of prediction is improved, organic combination of an image phenotype and a clinical phenotype is realized through introducing residual connection between the infarct change prediction results and clinical data of patients, dangerous factors and the like in the prognosis prediction process, characteristics of different dimensions can be more accurately extracted, a clinical diagnosis and a comprehensive diagnosis and clinical diagnosis support mode can be provided for the development of the patients, the clinical prediction mode is more accurate and the clinical prediction has a comprehensive correlation, the clinical diagnosis has a comprehensive and important and clinical prediction has a comprehensive correlation, a comprehensive prediction and important aspect is provided, and a comprehensive prediction and clinical prediction has a comprehensive and clinical prediction.
In an alternative embodiment of the present invention,
The corresponding loss functions of the generator network and the arbiter network are shown as follows:
LG＝-E[log(H(G(z)))]+λ·||G(z)-x||1;
wherein LG represents a generator loss value, E represents a mathematical expectation, H represents a discriminator network, G represents a generator network, λ represents regularized term weights, x represents a random noise vector, z represents a random noise vector, and G (z) represents an output result of the generator;
Wherein LH represents the arbiter loss value, μ represents the gradient penalty term weight,The output result of the H (x) discriminator is represented by the gradient of the random noise vector x.
In the embodiment, the soft constraint mode can effectively enhance the stability and generalization of the discriminator, improve the authenticity of a generated sample, ensure the richness of the generated sample while improving the generation quality through regularization weights, avoid pattern collapse, map noise to a real sample space through a training generator, discover the internal structure and attribute of data distribution, mine implicit characteristic representation, have important values for tasks such as data enhancement, anomaly detection and the like, can alleviate the overfitting problem through generating a vivid synthesized sample, improve the prediction performance, and on the whole, can effectively improve the problems of instability, gradient disappearance and the like in the generation of countermeasures in network training, balance the generation quality and diversity through the punishment gradient norm soft constraint discriminator, and form an end-to-end optimization framework with a downstream prediction model, thereby improving the performance and generalization capability of the whole cerebral infarction intelligent prediction system, and playing an important role in the aspects of data enhancement, feature extraction, model optimization and the like.
Fig. 2 is a schematic structural diagram of an infarct variation prediction system based on MRI images of brain parenchyma according to an embodiment of the invention, as shown in fig. 2, the system includes:
The device comprises a first unit, a second unit and a third unit, wherein the first unit is used for acquiring brain parenchyma MRI images of a patient and constructing a multi-mode image set, acquiring metadata information for each image in the multi-mode image set, screening according to a preset screening standard to obtain a first image set, performing artifact identification on the first image set, calculating the quality score of each image according to an artifact identification result, screening according to a preset scoring threshold value to generate a second image set, preprocessing the second image set to obtain standard image data, and performing feature extraction on the standard image data through multi-scale deep learning to obtain a brain region segmentation mask;
The second unit is used for adding the standard image data and the brain region division mask into a pre-constructed three-dimensional depth residual error convolutional neural network, extracting features through densely connected residual blocks, obtaining multi-mode feature data by combining a two-way multi-scale feature fusion algorithm and a progressive downsampling strategy, constructing topological relations among different brain regions based on the multi-mode feature data, extracting high-order semantic features, obtaining brain region feature data, and selecting the brain region feature data through a feature selection network to obtain high-quality feature data;
And the third unit is used for constructing a time sequence feature matrix based on the high-quality feature data, combining an infarct prediction model for time sequence modeling, distributing weights for each time step through a memory unit in the infarct prediction model, combining residual connection for element-level addition to generate residual output, splicing the residual output to obtain comprehensive time sequence features, predicting the comprehensive time sequence features through a fully-connected neural network to generate an infarct variation prediction result, predicting again through a prognosis prediction model based on a random forest algorithm, combining clinical data of a patient and dangerous factors to generate a prediction prognosis result, and combining the infarct variation prediction result and the prediction prognosis result to generate a comprehensive prediction report.
In a third aspect of an embodiment of the present invention,
There is provided an electronic device including:
A processor;
A memory for storing processor-executable instructions;
wherein the processor is configured to invoke the instructions stored in the memory to perform the method described previously.
In a fourth aspect of an embodiment of the present invention,
There is provided a computer readable storage medium having stored thereon computer program instructions which, when executed by a processor, implement the method as described above.
The present invention may be a method, apparatus, system, and/or computer program product. The computer program product may include a computer readable storage medium having computer readable program instructions embodied thereon for performing various aspects of the present invention.
It should be noted that the above embodiments are merely for illustrating the technical solution of the present invention and not for limiting the same, and although the present invention has been described in detail with reference to the above embodiments, it should be understood by those skilled in the art that the technical solution described in the above embodiments may be modified or some or all of the technical features may be equivalently replaced, and these modifications or substitutions do not make the essence of the corresponding technical solution deviate from the scope of the technical solution of the embodiments of the present invention.