Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. The components of the embodiments of the present invention generally described and illustrated in the figures herein may be arranged and designed in a wide variety of different configurations. Thus, the following detailed description of the embodiments of the invention, as presented in the figures, is not intended to limit the scope of the invention, as claimed, but is merely representative of selected embodiments of the invention. All other embodiments, which can be made by a person skilled in the art without making any inventive effort, are intended to be within the scope of the present invention.
It should be noted that: like reference numerals and letters denote like items in the following figures, and thus once an item is defined in one figure, no further definition or explanation thereof is necessary in the following figures. Meanwhile, in the description of the present invention, the terms "first", "second", and the like are used only to distinguish the description, and are not to be construed as indicating or implying relative importance.
The current classification diagnosis methods for diseases (such as epilepsy) mainly include a genetic marker auxiliary test method, a diagnosis method for extracting epileptic related signals based on brain wave signals (EEG), an epileptic diagnosis method based on a dynamic brain network and a long-short time memory network, and a multimode fusion discrimination method based on videos and EEG. The method is generally suitable for disease auxiliary diagnosis, and the main flow is to extract signals related to epilepsy as priori features, and transmit the priori features to a pre-trained classifier for prediction to obtain a prediction result, so that an auxiliary means is provided for disease diagnosis.
Most of the above methods are disease auxiliary diagnosis methods of non-specific people, but the auxiliary diagnosis of certain specific people or specific diseases is limited in application, for example, for the diagnosis of epilepsy of children, the characteristics distribution of individual samples is different from that of adults due to the fact that the children are in a rapid physiological development period, and the existing methods cannot be used for predicting the epilepsy of children in an ineffective way when the existing methods are used for the diagnosis of the epilepsy of children. Meanwhile, biological signal characteristics (such as brain electrical signals) on which the existing method depends are all descriptions of limited visual angles of diseases, and as the brain electrical signals are low latitude time signals, biological characteristic information is single and the causative relation with children epilepsy is not clear, other characteristics are also low latitude data, and the change of a diseased region cannot be comprehensively described from the perspective of time space. In addition, the existing disease auxiliary diagnosis method mostly extracts the characteristics manually according to the acquired priori knowledge when extracting the characteristics, the description of the disease characteristics is limited in the existing understanding and cognition range of people, and the wider expansion is difficult to realize.
Aiming at the problems of the existing auxiliary diagnosis method for diseases, the embodiment of the invention enables a machine to self-organize and learn the characteristics related to the diseases by comprehensively reflecting fMRI data of space-time transformation of the diseases, carries out disease prediction through a mixed neural network with stronger data expression capability, and carries out visual treatment on the predicted diseased region at the same time, so that the finally obtained visible view of the diseased region can provide powerful auxiliary diagnosis reference for doctor diagnosis.
Referring to fig. 1, fig. 1 is a block diagram of an electronic device 100 according to an embodiment of the invention. The electronic device 100 may be in communication connection with the magnetic resonance scanner, and the electronic device 100 may perform disease auxiliary diagnosis according to the fMRI image acquired by the magnetic resonance scanner, so as to obtain a corresponding visual view of the affected area. The electronic device 100 may be, but is not limited to, a notebook computer, desktop computer, server, portable computer, and the like. The electronic device 100 comprises a processor 101, a memory 102, a bus 103 and a communication interface 104, the processor 101, the memory 102 and the communication interface 104 being connected by the bus 103.
The memory 102 may include high-speed random access memory (RAM: random Access Memory) and may also include non-volatile memory (non-volatile memory), such as at least one disk memory. The electronic device 100 enables a communication connection between the electronic device 100 and the magnetic resonance apparatus via at least one communication interface 104 (which may be wired or wireless).
The memory 102 is used to store a program, such as the disease auxiliary diagnostic device 200 shown in fig. 4. The disease-assisted diagnosis apparatus 200 includes at least one software functional module that can be stored in the memory 102 in the form of software or firmware (firmware) or cured in the operating system of the electronic device 100. The processor 101 may execute the program stored in the memory 102 after receiving the execution instruction to implement the disease auxiliary diagnosis method disclosed in the following embodiments.
The processor 101 may be an integrated circuit chip having signal processing capabilities for executing executable modules stored in the memory 102, such as computer programs, during which the steps of the disease-assisted diagnostic method may be performed by hardware integrated logic circuits or instructions in the form of software in the processor 101. The processor 101 may be a general-purpose processor, including a central processing unit (Central Processing Unit, CPU for short), a network processor (Network Processor, NP for short), and the like; but may also be a Digital Signal Processor (DSP), a graphics processor (Graphics Processing Unit, GPU for short), a Field Programmable Gate Array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware components.
Bus 103 may be an ISA bus, a PCI bus, an EISA bus, or the like. Only one double arrow is shown in fig. 1, but not only one bus or one type of bus.
Embodiments of the present invention also provide a computer-readable storage medium having stored thereon a computer program which, when executed by the processor 101, implements the disease-assisted diagnosis method disclosed in the following embodiments.
Referring to fig. 2, fig. 2 is a flowchart illustrating a disease auxiliary diagnosis method according to an embodiment of the invention. The disease auxiliary diagnosis method comprises the following steps:
step S101, acquiring an fMRI image to be diagnosed, which is acquired by a magnetic resonance scanner, and converting the fMRI image to be diagnosed into a data matrix format to obtain fMRI data to be diagnosed.
In one embodiment, the fMRI image to be diagnosed acquired by the magnetic resonance scanner is four-dimensional data, including X, Y, Z spatial dimension and T time dimension, and when performing auxiliary diagnosis of a disease, the fMRI image to be diagnosed needs to be converted into a data matrix format which can be input into a hybrid neural network to obtain fMRI data to be diagnosed. Taking the diagnosis of benign epilepsy of children as an example, the brain of the children needs to be put into a magnetic resonance scanner, and the head of the children is scanned to obtain an fMRI image to be diagnosed.
It will be understood by those skilled in the art that the description of acquiring the fMRI image to be diagnosed and converting the fMRI image to be diagnosed into fMRI data in step S101 in the present embodiment is exemplary and not limiting, and in the disease-assisted diagnosis process, the fMRI image to be diagnosed may be acquired on site and converted into fMRI data to be diagnosed for subsequent disease prediction, or the fMRI image to be diagnosed may be acquired in advance and converted into fMRI data to be diagnosed and stored in the electronic device 100 for subsequent disease prediction, and step S101 may be skipped when fMRI data to be diagnosed is prestored in the electronic device 100.
Step S102, obtaining fMRI data to be diagnosed.
In one embodiment, fMRI data to be diagnosed may be obtained by acquiring fMRI images to be diagnosed using a magnetic resonance scanner and performing data format conversion; the fMRI image to be diagnosed may be acquired in advance and converted into fMRI data to be diagnosed and stored in the electronic device 100.
And step S103, inputting the fMRI data to be diagnosed into a pre-trained hybrid neural network to predict diseases, and obtaining auxiliary diagnosis results corresponding to the fMRI data to be diagnosed.
In one embodiment, the hybrid neural network is formed by splicing a deep convolutional neural network and a long-short-time memory network, and is a data processing model constructed from different dimensions of a time space. The deep convolution neural network is used for learning spatial features in the fMRI data from a three-dimensional space and achieving data dimension reduction of the high-dimensional fMRI data, and the long-short-term memory network is used for learning features of a time dimension.
The hybrid neural network is composed of multiple layers of nonlinear functions to form a model with high abstract capability, which is abstracted by a formula:
wherein x isi For fMRI data to be diagnosed, it can be expressed as xi ＝[xi (1),…,xi (t),...,xi (T)]；Spatial dimension features extracted for deep convolutional neural network, < >>For parameters of layer I in deep convolutional neural network, < >>The method comprises the steps that an output characteristic diagram of a first layer in a deep convolutional neural network is provided, and each layer of the deep convolutional neural network is provided with a corresponding output characteristic diagram; f (f)lstm Representing a long-short-term memory network, and y is fMRI data x to be diagnosed by the hybrid neural networki Is an auxiliary diagnostic result.
In one embodiment, the method for inputting fMRI data to be diagnosed into a pre-trained hybrid neural network for disease prediction includes: firstly, fMRI data to be diagnosed is input into a hybrid neural network, and the deep convolution neural network is utilized to extract the data to be diagnosedSpatial dimension features in broken fMRI data, i.e., in formula (1)Then, inputting the obtained space dimension feature into a long-short time memory network, and performing iterative calculation of time dimension on the space dimension feature by using the long-short time memory network to obtain an auxiliary diagnosis result, namely, using the +_in formula (1)>And performing iterative computation for a plurality of times until t+1 is equal to the time dimension characteristic T in the fMRI data to be diagnosed, and finally obtaining a result which is an auxiliary diagnosis result.
And step S104, when the auxiliary diagnosis result is diseased, determining a visual view of a diseased region corresponding to fMRI data to be diagnosed based on the hybrid neural network.
In one embodiment, after fMRI data to be diagnosed is input into a pre-trained hybrid neural network to perform disease prediction to obtain an auxiliary diagnosis result, the auxiliary diagnosis result is analyzed to be healthy or diseased according to the classification to which the auxiliary diagnosis result belongs. When the auxiliary diagnosis result is diseased, the visual view of the diseased region corresponding to the fMRI data to be diagnosed needs to be determined, specifically, firstly, the output characteristic diagram of each layer of the hybrid neural network in the disease prediction process is obtained, namely,then, carrying out maximum activation treatment on the output characteristic diagram of each layer to obtain a corresponding activation diagram; and calculating the average value of all the activation graphs in the time dimension, marking a diseased region in the original fMRI image based on the obtained calculation result, and obtaining a visual view of the diseased region, wherein the visual view can be represented by the following formula:
wherein y is3D For the mean of all activation graphs in the time dimension,a feature map is output for each layer.
Referring to fig. 3, fig. 3 shows a flowchart of a training method of the hybrid neural network, and the trained hybrid neural network can be obtained by the method described in steps S201 to S204, which is described in detail below:
step S201, obtaining a plurality of fMRI training samples and a sample label of each fMRI training sample.
In one embodiment, taking benign epileptic auxiliary diagnosis of a child as an example, when a plurality of fMRI training samples are acquired, the brain of the child is first put into a magnetic resonance scanner, and scanned to obtain fMRI images, in this embodiment, fMRI images obtained by single scanning of each child are used as one sample data. Preprocessing each fMRI image to obtain a plurality of fMRI training samples; meanwhile, for each collected fMRI training sample, a professional childhood neurologist marks whether the childhood benign epileptic disease exists according to the actual outpatient diagnosis result, namely, the label of each fMRI training sample is determined.
Specifically, the process of obtaining a plurality of fMRI training samples and a sample tag for each fMRI training sample may include: firstly, acquiring a plurality of original fMRI images acquired by a magnetic resonance scanner, and converting each original fMRI image into a data matrix format to obtain a plurality of fMRI data; then, unifying the spatial resolutions of the plurality of fMRI data, so that the time dimension of each fMRI data is the same, for example, the acquisition time length of the first fMRI data is 20min, the acquisition time length of other fMRI data is 15min, and the acquisition time length of the first fMRI data is also adjusted to be 15min; next, the color space of each fMRI data is normalized to obtain a plurality of fMRI training samples, i.e., the color space of each fMRI data is normalized to [0-1 ]]And zero-filling normalization of fMRI data of non-uniform spatial resolution to the sameIs a time dimension of (a). Meanwhile, performing one-hot coding on the label of each sample to obtain a sample label of each fMRI training sample, namely, if the label of the ith sample is healthy, setting the target output of the ith sample as di ＝[1,0]T The method comprises the steps of carrying out a first treatment on the surface of the If the label of the ith sample is diseased, its target output is set to di ＝[0,1]T 。
Step S202, training the constructed hybrid neural network based on the fMRI training samples and the sample labels of the fMRI training samples.
In one embodiment, after acquiring a data set composed of a plurality of fMRI training samples and sample labels of each fMRI training sample, the data set may be divided into 80% training set and 20% test set, the training of the hybrid neural network is performed by using the 80% training set, and the performance of the trained hybrid neural network is verified by using the 20% test set; the data sets can also be all used as data sets to train the hybrid neural network, and then the fMRI test sample and the sample label of the fMRI test sample are obtained to test the trained hybrid neural network.
The hybrid neural network can be represented by formula (1), where x in formula (1) is the time the hybrid neural network is trainedi For the fMRI training samples,parameters of a first layer in the hybrid neural network; y is the hybrid neural network versus fMRI training sample xi Is a predicted result of (a). Because the hybrid neural network needs to learn a large number of parameters, the large-scale parameter learning needs to carry out a large number of sample training, and the self-coding network can learn the data distribution in an unsupervised mode, so that the parameter distribution of the hybrid neural network is initialized, the dependence of the hybrid neural network on the large-scale training data is reduced, and the auxiliary diagnosis of the hybrid neural network on the benign epilepsy of children under the condition of small samples is realized.
In one embodiment, the step of training the constructed hybrid neural network based on fMRI training samples and sample tags of fMRI training samples comprises: firstly, inputting an fMRI training sample into a hybrid neural network, and extracting spatial dimension characteristics in the fMRI training sample by using a deep convolution neural network of the hybrid neural network; then, inputting the obtained space dimension characteristics into a long-short time memory network of the hybrid neural network, and performing iterative computation of time dimension on the space dimension characteristics by using the long-short time memory network to obtain a prediction result of the fMRI training sample; finally, according to the prediction result and the sample label of the fMRI training sample, carrying out parameter updating on the hybrid neural network, wherein the parameter updating can be realized based on a predefined performance function of the hybrid neural network, and the performance function can be expressed by the following formula:
where n is the number of fMRI training samples, di Training sample x for fMRIi Sample tag flstm (w,xi ) Predicting results of fMRI training samples; hybrid neural networks compute gradients for each parameter by reversing the direction of computationRealizing the supervised learning of the data characteristics, optimizing all parameters w in the hybrid neural network, namely, according to the formula +.>And updating parameters.
And step S203, evaluating the trained hybrid neural network to obtain an evaluation result.
In one embodiment, after training the hybrid neural network, the trained hybrid neural network needs to be evaluated, and a specific evaluation process may include: firstly, obtaining an fMRI test sample and a sample label of the fMRI test sample, wherein the obtaining process of the fMRI test sample and the sample label of the fMRI test sample is similar to the obtaining process of the fMRI test sample and the sample label of the fMRI test sample in step S201, and is not repeated here; then, inputting the fMRI test sample and the sample label of the fMRI test sample into the trained hybrid neural network to obtain the prediction result of the fMRI test sample; next, substituting the predicted result and the sample label of the fMRI test sample into a predefined evaluation formula to obtain an evaluation result, wherein the evaluation formula can be represented by the following formula:
wherein Acc is the evaluation result, yi D is the predicted result of fMRI test samplei For the sample label of fMRI test samples, N is the number of fMRI test samples.
Step S204, judging whether the mixed neural network reaches a preset condition according to the evaluation result, and obtaining the trained mixed neural network when the mixed neural network reaches the preset condition.
In one embodiment, the preset condition may be that the prediction accuracy reaches a preset threshold (e.g., 90%) or reaches a prescribed number of iterations (e.g., 200), that is, if the evaluation result satisfies the model condition, a trained hybrid neural network is obtained; if the evaluation result does not meet the model condition, training the hybrid neural network is continued until the model converges or a prescribed number of iterations (e.g., 200 times) is reached.
Compared with the prior art, the embodiment of the invention has the following beneficial effects:
firstly, aiming at characteristic groups or specific diseases, a mixed neural network is designed to realize disease prediction, so that disease auxiliary diagnosis is more targeted;
secondly, when the hybrid neural network training is carried out, features related to diseases are autonomously learned from an fMRI training sample in an unsupervised mode, feature extraction is optimized in a supervised learning mode, all features are derived from original data, and the inherent limitation of artificial priori features is broken through; meanwhile, the self-coding network is utilized to initialize the mixed neural network to obtain initial parameters, so that the dependence of the mixed neural network on large-scale data can be reduced, and the auxiliary diagnosis of the mixed neural network on benign epilepsy of children under the condition of small samples can be realized;
thirdly, single fMRI data is used for carrying out disease auxiliary diagnosis, dependency of the existing auxiliary diagnosis method on EEG data is broken through, and because the fMRI data consists of three-dimensional space scanning and three-dimensional image changes in time, compared with EEG data collected based on multiple probes or various artificial features extracted based on priori knowledge, the fMRI data has more comprehensive reflection on diseases, and more accurate disease prediction results can be obtained;
fourth, three-dimensional visualization of the affected area is realized, and three-dimensional visualization of the predicted affected area can be realized while disease diagnosis can be rapidly realized within 10 seconds, so that powerful auxiliary diagnosis reference is provided for doctor diagnosis.
Referring to fig. 4, fig. 4 is a block diagram illustrating a disease auxiliary diagnostic device 200 according to an embodiment of the invention. The disease-assisted diagnosis apparatus 200 includes an image acquisition module 201, a data acquisition module 202, a disease prediction module 203, a patient visual assistance module 204, and a model training module 205.
The image acquisition module 201 is configured to acquire an fMRI image to be diagnosed acquired by the magnetic resonance scanner, and convert the fMRI image to be diagnosed into a data matrix format, so as to obtain fMRI data to be diagnosed.
A data acquisition module 202 is configured to acquire fMRI data to be diagnosed.
The disease prediction module 203 is configured to input fMRI data to be diagnosed into a pre-trained hybrid neural network to perform disease prediction, so as to obtain an auxiliary diagnosis result corresponding to the fMRI data to be diagnosed.
In one embodiment, the disease prediction module 203 is specifically configured to input fMRI data to be diagnosed into a hybrid neural network, and extract spatial dimension features in the fMRI data to be diagnosed by using a deep convolutional neural network; and inputting the obtained space dimension characteristics into a long-short-time memory network, and performing iterative computation of time dimension on the space dimension characteristics by using the long-short-time memory network to obtain an auxiliary diagnosis result.
And the patient visual auxiliary diagnosis module 204 is used for determining a visual view of a diseased region corresponding to the fMRI data to be diagnosed based on the hybrid neural network when the auxiliary diagnosis result is diseased.
In one embodiment, the patient visual auxiliary diagnosis module 204 is specifically configured to obtain an output feature map of each layer of the hybrid neural network in the disease prediction process when the auxiliary diagnosis result is a disease; carrying out maximum activation treatment on the output characteristic diagram of each layer to obtain a corresponding activation diagram; and calculating the average value of all the activation images in the time dimension, and marking the affected area in the original fMRI image based on the obtained calculation result to obtain the visual view of the affected area.
The model training module 205 is configured to obtain a trained hybrid neural network in the following manner: acquiring a plurality of fMRI training samples and a sample label of each fMRI training sample; training the constructed hybrid neural network based on the fMRI training samples and sample labels of the fMRI training samples; evaluating the trained hybrid neural network to obtain an evaluation result; judging whether the mixed neural network reaches a preset condition according to the evaluation result, and obtaining the trained mixed neural network when the mixed neural network reaches the preset condition.
In one embodiment, the hybrid neural network isWherein x isi For fMRI training samples, < >>Parameters for the first layer in the hybrid neural network; y is the hybrid neural network versus fMRI training sample xi Is a predicted result of (a).
In one embodiment, model training module 205 performs a manner of acquiring a plurality of fMRI training samples, including: acquiring a plurality of original fMRI images acquired by a magnetic resonance scanner, and converting each original fMRI image into a data matrix format to obtain a plurality of fMRI data; unifying the spatial resolutions of the plurality of fMRI data so that the time dimension of each fMRI data is the same; and normalizing the color space of each fMRI data to obtain a plurality of fMRI training samples.
In one embodiment, model training module 205 performs a training of the constructed hybrid neural network based on fMRI training samples and sample tags of fMRI training samples, including: inputting the fMRI training sample into a hybrid neural network, and extracting spatial dimension characteristics in the fMRI training sample by using a deep convolution neural network of the hybrid neural network; inputting the obtained space dimension characteristics into a long-short time memory network of a hybrid neural network, and performing iterative computation of time dimension on the space dimension characteristics by using the long-short time memory network to obtain a prediction result of the fMRI training sample; and updating parameters of the hybrid neural network according to the prediction result and the sample label of the fMRI training sample.
In one embodiment, the model training module 205 performs an evaluation on the trained hybrid neural network to obtain an evaluation result, including: obtaining a sample label of an fMRI test sample and an fMRI test sample; inputting the fMRI test sample and the sample label of the fMRI test sample into the trained hybrid neural network to obtain the prediction result of the fMRI test sample; substituting the predicted result of the fMRI test sample and the sample label into a predefined evaluation formulaObtaining an evaluation result, wherein Acc is the evaluation result, yi D is the predicted result of fMRI test samplei For the sample label of fMRI test samples, N is the number of fMRI test samples.
It will be clearly understood by those skilled in the art that, for convenience and brevity of description, the specific working process of the disease auxiliary diagnostic apparatus 200 described above may refer to the corresponding process in the foregoing method embodiment, and will not be repeated here.
In summary, the method and device for assisting diagnosis of diseases provided by the embodiment of the invention include: acquiring fMRI data to be diagnosed; inputting the fMRI data to be diagnosed into a pre-trained hybrid neural network to predict diseases, and obtaining auxiliary diagnosis results corresponding to the fMRI data to be diagnosed; and when the auxiliary diagnosis result is diseased, determining a visual view of a diseased region corresponding to fMRI data to be diagnosed based on the hybrid neural network. Compared with the prior art, the embodiment of the invention can realize the visualization of the predicted diseased region, thereby providing a powerful auxiliary diagnosis reference for doctor diagnosis.
In the several embodiments provided in this application, it should be understood that the disclosed apparatus and method may be implemented in other manners as well. The apparatus embodiments described above are merely illustrative, for example, of the flowcharts and block diagrams in the figures that illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
In addition, functional modules in the embodiments of the present invention may be integrated together to form a single part, or each module may exist alone, or two or more modules may be integrated to form a single part.
The functions, if implemented in the form of software functional modules and sold or used as a stand-alone product, may be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art or in a part of the technical solution, in the form of a software product stored in a storage medium, comprising several instructions for causing a computer device (which may be a personal computer, a server, a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes. It is noted that relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
The above description is only of the preferred embodiments of the present invention and is not intended to limit the present invention, but various modifications and variations can be made to the present invention by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention. It should be noted that: like reference numerals and letters denote like items in the following figures, and thus once an item is defined in one figure, no further definition or explanation thereof is necessary in the following figures.