Detailed Description
The present invention will be described in detail with reference to the accompanying drawings, and fig. 1 is a schematic flow chart of an artificial intelligence-based ultrasound image data classification method according to an embodiment of the present invention, and the detailed description of the artificial intelligence-based ultrasound image data classification method will be described below.
Step S110, a real-time ultrasonic scanning signal sequence of a target object is obtained, wherein the real-time ultrasonic scanning signal sequence comprises a plurality of frames of dynamic ultrasonic images, and time sequence change characteristics of an organization elastic parameter and a blood flow dynamic parameter are embedded in each frame of dynamic ultrasonic image.
In this embodiment, taking a target object as an example of a patient suffering from liver disease, a doctor uses an ultrasonic diagnostic apparatus to scan a liver portion of the patient, and ultrasonic waves emitted by the ultrasonic apparatus propagate in liver tissue and reflect back, are received by the ultrasonic apparatus, and are converted into electrical signals, thereby forming a real-time ultrasonic scanning signal sequence. Wherein the ultrasound scanning signal sequence comprises a plurality of frames of dynamic ultrasound images. For example, in a first frame of images, the tissue elasticity parameters may reflect that the texture of a region of the liver is relatively stiff, possibly suggesting that there is fibrosis or stiffening of that region. The time sequence variation characteristic of the hemodynamic parameters is that the blood flow velocity and the blood flow in the blood vessels in the liver show regular variation along with the beating period of the heart. For example, in systole, the blood flow rate into the liver increases, and the blood flow increases, which is manifested as an increase in intravascular signal intensity in the image, and vice versa in diastole. Changes of the tissue elasticity parameters and the hemodynamic parameters are continuously recorded in each frame of images along with the time, and in the subsequent frames of images, the gradual decrease of the tissue elasticity around a certain suspected lesion area of the liver can be found, and the local abnormal fluctuation of the blood flow velocity appears, so that the changes have reference value for the subsequent judgment of the nature and the development trend of the lesion.
And step S120, carrying out noise suppression and motion artifact compensation processing on the real-time ultrasonic scanning signal sequence, generating a denoised standardized ultrasonic image sequence, and marking anatomical boundary coordinates of the region of interest in the standardized ultrasonic image sequence.
Taking a liver patient as an example, the original real-time ultrasonic scanning signal sequence has noise due to electronic noise of ultrasonic equipment, interference of surrounding environment and the like. First, each frame of ultrasonic scanning signal in the real-time ultrasonic scanning signal sequence is subjected to wavelet transformation decomposition. For example, for an ultrasonic scan signal containing the whole and part of the surrounding tissue of the liver, after wavelet transformation and decomposition, the high-frequency noise component may show some messy, weak and irregular signal fluctuation, and the low-frequency tissue signal component shows the rough outline of the main structure of the liver tissue, blood vessels and the like. And then, carrying out layer-by-layer filtering on the high-frequency noise component by adopting an adaptive threshold algorithm, wherein the set threshold value can be dynamically adjusted according to the characteristics of the signal in the process. For example, for regions where liver edges are relatively blurred, the threshold may be relaxed appropriately to preserve weak signals that may belong to liver tissue, while filtering is strict for regions of significant noise. After filtering, intermediate frequency signal components related to the movement of the vessel wall, such as the blood flow signal of the portal vein in the liver, are retained.
And then, carrying out inverse wavelet transformation reconstruction on the filtered ultrasonic scanning signals to obtain an ultrasonic image subjected to preliminary denoising. However, the images are subject to motion artifacts due to the possible slight breathing or body movement of the patient during the scan. The tissue displacement vector field between the ultrasonic scanning signals of the adjacent frames is calculated based on an optical flow method, and for the liver image, the moving direction and distance of the liver tissue between the adjacent frames are calculated. According to the tissue displacement vector field, motion track compensation is carried out on the preliminarily denoised ultrasonic image, for example, a certain characteristic point on the liver generates certain displacement between adjacent frames due to respiration of a patient, and the position of the characteristic point in the image is more accurate through compensation. And then, carrying out gray scale normalization operation on the ultrasonic image after the motion compensation, and mapping the pixel intensity to a preset standard dynamic range. Assuming that the preset standard dynamic range is 0-255, the gray level of the normal tissue in the liver may be adjusted to a specific range, such as 100-150, and the lesion area may have different gray level ranges.
And then carrying out edge-preserving smoothing treatment on the normalized ultrasonic image by adopting an anisotropic diffusion algorithm, wherein in the process, the algorithm can reduce the influence of noise on the premise of keeping sharp edges in the edge region of the liver image. And then the detail enhancement is carried out on the smooth ultrasonic image through a bilateral filter, for example, the boundary of the blood vessel in the liver is clearer, and the level of the blood flow signal in the blood vessel is clearer. Finally, region matching is carried out on the ultrasonic image with enhanced details according to a preset anatomical structure template, a standardized ultrasonic image sequence containing the boundary of the whole liver organ is cut out, and anatomical boundary coordinates of the liver, such as coordinate points of the upper edge, the lower edge, the boundary of the left and right leaves and the like of the liver are marked.
Step S130, performing multi-scale anatomical structure decomposition on each frame of standardized ultrasonic image based on the anatomical boundary coordinates, and generating local feature map sets corresponding to different tissue levels, wherein each local feature map in the local feature map sets comprises a three-dimensional matrix of edge sharpness indexes, texture consistency indexes and blood flow signal intensity distribution.
When processing a standardized ultrasound image of a liver patient, a rectangular region of interest containing the liver is truncated in each frame of the standardized ultrasound image according to previously marked anatomical boundary coordinates of the liver. For example, the rectangular region just covers a substantial portion of the liver and the major vessel branches. And then carrying out Gaussian pyramid decomposition on the rectangular region of interest to generate a multi-scale image group containing original resolution, half resolution and quarter resolution. For the original resolution image, the method can display the finest structure of liver tissue, such as the rough outline of tiny blood vessel branches and liver lobules, the half resolution image summarizes the structural characteristics of a larger area of the liver to a certain extent, the grasping of the whole structure is more beneficial, and the quarter resolution image can highlight macro-characteristics such as the whole shape of the liver and main blood vessel trend.
A multi-directional Gabor filter bank is applied to each scale image in the multi-scale image set. Taking the blood vessel in the liver as an example, the Gabor filter bank can extract a frequency domain response characteristic map matched with the branching direction of the blood vessel. In the original resolution image, the frequency domain features of the fine vessel branches may be accurately extracted, reflecting the texture direction of the vessel wall and the local variation of the blood flow signal. And inputting the frequency domain response feature graphs with different scales into a three-dimensional convolutional neural network, and aggregating edge gradient information under different resolutions through a trans-scale feature fusion layer. For example, at the liver edge, the edge gradient information in the original resolution image may be very detailed, and the edge gradient information in the quarter resolution image is more focused on the whole contour, so that the liver edge characteristics can be obtained more comprehensively and accurately after fusion.
And performing non-maximum suppression processing on the aggregated feature map, eliminating redundant edge response in the liver image and reserving maximum intensity edge points. For example, in a complex vascular network in the liver, some weak edges due to noise or artifacts are removed, leaving only the strongest edge points that truly represent important structures such as vessel boundaries. And constructing a space delaunay triangular grid based on the reserved edge points, and calculating curvature distribution and area occupation ratio indexes of each triangular patch of the triangular grid. For curved portions of the surface or internal structure of the liver, such as the fornix of the liver, the curvature distribution of the triangular patches may reflect the change in shape thereof, and the area ratio index may represent the relative importance of the different regions in the overall structure.
And finally, carrying out region division on the triangular meshes according to curvature distribution and area occupation ratio indexes to generate sub-mesh sets corresponding to different tissue types. In the liver, different sub-grids of liver parenchyma regions, vascular regions, etc. may be divided. And carrying out statistics on the space-time variation variance of the blood flow signal intensity in each sub-grid of the sub-grid set, and generating a local feature map set containing edge geometric features and hemodynamic features. For example, in a subgrid of a liver parenchymal region, the variance of the temporal-spatial variation of the blood flow signal intensity may be relatively small, whereas in a vascular subgrid near a lesion region, the variance of the temporal-spatial variation of the blood flow signal intensity may increase due to the influence of the lesion on the blood flow, these features being recorded in a three-dimensional matrix of edge sharpness index, texture uniformity index, and blood flow signal intensity distribution of a local feature map set.
Step S140, inputting the local feature map set into a cascade depth classification network, and performing cross-frame feature fusion and dynamic weight adjustment on the local feature map set through a space-time feature alignment module and a multi-granularity attention allocation module nested in the cascade depth classification network, so as to output classification probability distribution and a spatial topological relation map of an abnormal region in each frame of standardized ultrasonic image.
The local feature map set for the liver patient is input into a cascaded depth classification network. Firstly, a long-short time memory network of a space-time characteristic alignment module is entered to carry out time sequence state transfer. For example, in successive multi-frame liver ultrasound images, the lesion region may change in morphology, size, or blood flow characteristics over time. The long-short-term memory network can capture the rules of the changes and generate a time dimension feature vector containing the evolution rules of the focus morphology between continuous frames. For example, the lesion region appears circular in the first few frames of images and the boundary is relatively clear, becomes irregular in shape and blurred over time, and these changing features are accurately recorded in the time-dimensional feature vector.
And then inputting the time dimension feature vector into a deformable convolution layer to perform space deformation modeling. In ultrasound images of the liver, certain non-linear deformations of the liver tissue occur due to the effects of respiratory movements of the liver itself and heart beats. The deformable convolution layer can output a spatial deformation characteristic matrix matched with the nonlinear deformation of the liver tissue structure, and accurately reflects the actual morphological change of the liver in space.
And splicing the time dimension feature vector and the space deformation feature matrix to generate a space-time joint feature matrix, and inputting the space-time joint feature matrix into a multi-head self-attention mechanism of the multi-granularity attention distribution module. For different characteristic channels in the liver image, such as channels representing liver tissue texture, blood flow signals, edge characteristics and the like, the multi-head self-attention mechanism calculates global correlation weights among the different characteristic channels and outputs a global attention weight matrix. For example, when determining whether a liver lesion is malignant, the blood flow signal feature channel may have a higher correlation with the malignancy of the lesion, and then the weight corresponding to the blood flow signal feature channel in the global attention weight matrix may be higher.
And inputting the global attention weight matrix into a gating linear unit for feature channel screening, reserving a feature channel related to the current classification task (judging liver abnormal region), and generating a gating screening feature vector. For example, some characteristic channels associated with the interference of tissues around the liver may be screened, while channels associated with the internal structure and pathological features of the liver are emphasized. And then inputting the gating screening feature vector into a local window attention calculation layer, enhancing the spatial correlation of adjacent feature nodes, and generating a local window attention weight matrix. In liver images, there is a certain correlation in space between feature nodes between adjacent liver lobules or vessel branches, and the multi-granularity attention allocation module can enhance the correlation.
And weighting and fusing the global attention weight matrix and the local window attention weight matrix to generate a multi-granularity attention distribution map, and overlapping the multi-granularity attention distribution map to the space-time joint feature matrix through residual connection to generate an optimized classification feature vector. And finally, inputting the optimized classification feature vector into a fully-connected classification layer, and outputting classification probability distribution and a spatial topological relation map of abnormal regions (such as possible tumors, fibrosis regions and the like in the liver) in each frame of standardized ultrasonic image. The classification probability distribution represents the probability that each region of the liver is an abnormal region, and the spatial topological relation map accurately depicts the spatial relation between the abnormal region and surrounding normal tissues, blood vessels and other structures, such as whether the abnormal region is close to a large blood vessel, invades surrounding liver tissues and the like.
And step S150, generating a multi-mode diagnosis report according to the classification probability distribution and the spatial topological relation map, and comparing the similarity of the multi-mode diagnosis report with the characteristics of the historical cases in the medical image database to generate an interactive three-dimensional visual interface comprising a risk grade label and a treatment suggestion.
In detail, a multi-modal diagnostic report may be generated based on the classification probability distribution and the spatial topological relation map of the liver patient, mapping the classification probability distribution to spatial coordinates of the normalized ultrasound image sequence, generating an abnormal region overlay with translucent color coding. For example, in a liver image, a region with a high probability of abnormality is covered with a red translucent color, a region with a low probability is covered with a yellow translucent color, and a normal region maintains a primary color or represents a normal range with a green translucent color. And inputting the spatial topological relation map into a three-dimensional volume rendering engine, and constructing a three-dimensional volume rendering model for marking the vascular branches and the focus infiltration range. In a three-dimensional model of the liver, the trend of blood vessels in the liver and the infiltration of abnormal areas (such as tumors) into surrounding tissues can be clearly seen.
And carrying out spatial registration on the abnormal region covering layer and the three-dimensional rendering model to generate a visual basic frame fusing the two-dimensional probability distribution and the three-dimensional anatomical structure. And then extracting hemodynamic characteristic parameters in the classification probability distribution, searching historical case characteristics with similar hemodynamic characteristic parameters in a medical image database, and generating a case similarity comparison result. For liver patients, it is possible to find some historical liver disease cases that also have abnormal blood flow velocity and similar blood flow changes.
And (3) associating the case similarity comparison result with the visual basic framework to generate a side comparison view and a key difference parameter labeling layer. For example, in the side contrast view, information such as liver images, hemodynamic parameters and the like of the current patient and the historical case can be displayed at the same time, and the key difference parameter labeling layer can highlight differences between the current patient and the historical case in terms of liver size, lesion position, blood flow speed and the like. Based on the key difference parameter labeling layer matching with a preset treatment scheme flow chart, a logic association table of a risk level label and treatment suggestions is generated, and then the logic association table is embedded into a visual basic framework to generate an interactive three-dimensional visual interface comprising an interactive time axis control, a real-time ultrasonic playback function and a voice broadcasting engine. The doctor can check the liver ultrasonic images and analysis results at different time points through the time axis control, and confirm the details in the images again through the real-time ultrasonic playback function, and the voice broadcasting engine can broadcast the information such as the diagnosis results, the risk level, the treatment advice and the like, so that the doctor can conveniently operate in the diagnosis process and comprehensively judge the illness state of the patient.
Based on the steps, the embodiment of the application improves the accuracy and efficiency of ultrasonic diagnosis, effectively suppresses noise and compensates motion artifacts by acquiring and processing an ultrasonic scanning signal sequence in real time, ensures image quality, combines a multi-scale anatomical structure decomposition technology with comprehensive analysis of edge sharpness, texture consistency and blood flow signal intensity, deeply digs fine granularity characteristics in an ultrasonic image, realizes intelligent fusion and dynamic weight adjustment of cross-frame characteristics, accurately captures space-time variation of an abnormal region, and finally generates a multi-mode diagnosis report, thereby not only providing comprehensive classification probability and space topology information, but also outputting visual risk level and treatment suggestion by similarity comparison with historical cases, constructing a highly interactive three-dimensional visual interface, and greatly enhancing doctor understanding and decision support of the illness state.
In one possible implementation, step S120 includes:
Step S121, performing wavelet transform decomposition on each frame of ultrasonic scanning signal in the real-time ultrasonic scanning signal sequence, and separating high-frequency noise components and low-frequency tissue signal components.
Taking the previous ultrasound scan of a liver patient as an example, in this process, it is assumed that the wavelet transform uses a Daubechies wavelet basis function (e.g., db4 wavelet) that is effective in decomposing the signal into subbands of different frequencies. For a frame of liver ultrasonic scanning signal, after wavelet transformation and decomposition, high-frequency noise components are usually represented by signal parts with higher frequency, relatively lower energy and no obvious regularity, and the high-frequency noise may be derived from electronic elements of ultrasonic equipment, tiny interference of surrounding environment and the like. The low frequency tissue signal component contains the main structural information of liver tissue, such as the overall outline of liver, the general trend of larger blood vessels, etc., and has lower frequency and relatively concentrated energy.
Step S122, filtering the high-frequency noise component layer by adopting an adaptive threshold algorithm to obtain a filtered ultrasonic scanning signal, and reserving an intermediate frequency signal component related to the movement of the blood vessel wall.
For example, for the first layer of the high-frequency noise component of a liver ultrasound scanning signal of a certain frame, the algorithm analyzes the statistical parameters such as standard deviation, mean value and the like of the signal of the layer, if the standard deviation is 5 (assumed unit), the threshold value of the layer is 15 according to a certain algorithm rule (for example, the standard deviation is multiplied by a coefficient, and the coefficient is assumed to be 3). The signal portions above the threshold are considered to be potentially useful signals to be preserved, and those below the threshold are determined to be noise and filtered out. This adaptive thresholding can effectively remove noise while preserving intermediate frequency signal components associated with vessel wall motion throughout the layer-by-layer filtering process. For vessels in the liver, the frequency of the signals generated by the blood flow is in the intermediate frequency range, for example, the frequency is between 1 and 5MHz (assumption), and the intermediate frequency signals have important significance for the subsequent analysis of the motion state of the vessel wall, the blood flow speed and the like.
And step S123, performing inverse wavelet transformation reconstruction on the filtered ultrasonic scanning signals to generate an ultrasonic image subjected to preliminary denoising.
After the high-frequency noise component is filtered, the filtered ultrasonic scanning signals are subjected to inverse wavelet transformation reconstruction, and the operation is that the sub-band signals subjected to filtering treatment are recombined according to the inverse process of wavelet transformation, so that an ultrasonic image subjected to preliminary denoising is generated. In this frame of preliminary denoised liver ultrasound image, some blurred parts of the liver tissue become relatively sharp due to the removal of high frequency noise, the contours of the blood vessels are also more pronounced, but motion artifacts may be present in the image due to factors such as respiration of the patient during scanning, slight body movements, etc.
Step S124, calculating a tissue displacement vector field between adjacent frame ultrasonic scanning signals based on an optical flow method, performing motion track compensation on the primarily denoised ultrasonic image according to the tissue displacement vector field, performing gray scale normalization operation on the motion compensated ultrasonic image, and mapping pixel intensity to a preset standard dynamic range to obtain a normalized ultrasonic image.
To eliminate motion artifacts, a tissue displacement vector field between adjacent frames of ultrasound scan signals is calculated based on an optical flow method. The optical flow method determines the direction and speed of motion of an object by analyzing the brightness variation of pixels in adjacent frame images. For ultrasound images of a liver patient, assuming a time interval between two adjacent frames of images of 0.1 seconds (which is a typical time interval determined by the scan frame rate of the ultrasound device), the optical flow method calculates the displacement vector of the liver tissue within this 0.1 seconds. For example, if a specific point on the liver (assuming that the coordinates are (x 1, y 1)) is shifted by (x 2, y 2) from the position in the first frame image to the position in the second frame image, the calculated displacement vector is ((x 2-x 1), (y 2-y 1)). And performing motion track compensation on the primarily denoised ultrasonic image according to the tissue displacement vector field. If a suspected lesion area in the liver has obvious displacement in the image due to respiratory motion of the patient before the compensation, the position of the suspected lesion area in the image can more accurately reflect the real position of the suspected lesion area in the liver after the compensation through the motion trail.
And then performing gray scale normalization operation on the motion compensated ultrasonic image. Assuming that the preset standard dynamic range is 0-255 (which is a common image pixel intensity range), in the liver ultrasound image after motion compensation, the gray value range of all pixels in the image is counted first. For example, the gray scale value range of liver tissue may be 50-180 and the gray scale value range of blood vessels may be 30-100 (here, only assumed to be a general range). These raw gray values are then mapped to a standard dynamic range of 0-255 by means of a linear mapping. The aim of this is to make the images of different frames and the images of different patients comparable in gray scale, facilitating subsequent analysis and processing. And after the gray scale normalization operation, obtaining a normalized ultrasonic image.
And step S125, performing edge-preserving smoothing treatment on the normalized ultrasonic image by adopting an anisotropic diffusion algorithm, performing detail enhancement on the smoothed ultrasonic image by using a bilateral filter, performing region matching on the detail-enhanced ultrasonic image according to a preset anatomical structure template, and cutting out a standardized ultrasonic image sequence containing a complete organ boundary.
In this embodiment, when the anisotropic diffusion algorithm processes the liver ultrasound image, the direction and intensity of diffusion are adjusted according to the gradient information of the ultrasound image. At the liver edges, the algorithm limits diffusion perpendicular to the edge direction due to the large gray gradient of the edge, thus preserving the sharpness of the edge. For example, for a pixel point on the edge of the liver, the gray level difference value of the surrounding pixels is larger, and the algorithm can smooth along the edge direction preferentially in the diffusion process of the pixel point, so as to avoid blurring the edge. After being processed by an anisotropic diffusion algorithm, the detail enhancement is carried out on the ultrasonic image after the smoothing processing by a bilateral filter. The bilateral filter considers the space distance and gray scale difference of pixels, and can strengthen the detail characteristics of structures such as blood vessels, liver lobules and the like in the liver. For example, after the blood flow signal in the blood vessel is processed by the bilateral filter, the layering sense of the signal is stronger, and the boundary of the blood vessel wall is clearer.
The preset liver anatomical structure template contains information such as standard shape, size, and position of main vessel branches of the liver. And for the liver ultrasonic image with enhanced details, finding a liver region corresponding to the template in the image through a template matching algorithm. For example, the coordinate range of the upper edge of the liver in the image is defined as (y 1-y 2) in the template, and the region which accords with the coordinate range and has the liver tissue characteristics is found in the image through an algorithm. And finally cutting out a standardized ultrasonic image sequence containing the boundary of the complete liver organ, wherein each frame of image in the standardized ultrasonic image sequence is subjected to a series of processing, so that the standardized ultrasonic image sequence has lower noise, a clear structure and accurate liver region positioning, and high-quality image data is provided for subsequent analysis.
In one possible implementation, step S130 includes:
Step S131, in each frame of standardized ultrasonic image, a rectangular region of interest containing a target organ is intercepted according to the anatomical boundary coordinates.
In this embodiment, the anatomical boundary coordinates precisely define the location of the liver in the image, e.g., the upper border coordinate range of the liver is 100-150 rows of pixels from the top of the image down (assuming a total number of rows of the image of 500 rows), and the left and right border coordinate ranges are 200-300 columns of pixels and 400-500 columns of pixels from the left to the right of the image (assuming a total number of columns of the image of 600 columns), respectively. The rectangular region of interest defined by these coordinates encompasses a substantial portion of the liver and major vessel branches, so that subsequent analysis can be focused on the liver region, avoiding interference with surrounding tissue.
Step S132, performing gaussian pyramid decomposition on the rectangular region of interest, and generating a multi-scale image set including an original resolution, a half resolution, and a quarter resolution.
For the original resolution image, it maintains the finest structural information of the liver region, such as tiny branches of blood vessels in the liver, which may be as small as 1-2 pixels in diameter (assuming that one pixel in the image represents the actual length of 0.1 mm), and the general outline of the liver lobules is also clearly visible. The half resolution image is obtained by downsampling the original resolution image, and at this scale, the structural features of a larger region of the liver are more obvious, for example, features of a larger structure such as the interlobe of the liver are easier to grasp integrally, some tiny vessel branches can be combined into a thicker vessel representation, but the overall vessel trend is still clear. The quarter resolution image further summarizes the macroscopic features of the liver, the main vessel branches are more prominent, and the overall shape and general tissue structure distribution of the liver can be clearly seen.
And step S133, a multidirectional Gabor filter set is applied to each scale image in the multi-scale image set, and a frequency domain response characteristic diagram matched with the blood vessel branch direction is extracted.
For vessels in the liver, the Gabor filter bank captures certain directional and texture features because of the vessel. For example, assume that the directions of the Gabor filter group are set to four directions of 0 °, 45 °, 90 °, and 135 ° (the number and angle of directions can be adjusted according to actual demands). In the liver image of the original resolution, when the Gabor filter is filtered along an angle close to the blood vessel branching direction, a strong response can be obtained in the frequency domain. For smaller diameter vessel branches, such as intrahepatic arterioles (assuming a diameter of 2-3 pixels), the frequency domain response profile will exhibit a specific frequency and amplitude distribution in the Gabor filter direction coincident with the vessel direction, reflecting the vessel wall texture direction and local changes in the blood flow signal. In half resolution and quarter resolution images, the same principle is adopted, but due to the reduced resolution, the representation of the vessel branches is more generalized, and the frequency domain response characteristic diagram correspondingly reflects the comprehensive characteristics of a wider range of vessels.
Step S134, inputting frequency domain response feature graphs with different scales into a three-dimensional convolutional neural network, aggregating edge gradient information under different resolutions through a trans-scale feature fusion layer, performing non-maximum suppression processing on the aggregated feature graphs, eliminating redundant edge responses and reserving maximum intensity edge points, constructing a space Delaunay triangular grid based on the reserved edge points, and calculating curvature distribution and area occupation ratio indexes of each triangular patch of the triangular grid.
For the edges of the liver, the edge gradient information in the original resolution image is very detailed, e.g. at the boundary of the liver edge with the surrounding tissue, the gray value change of the pixels may have a large gradient change over several pixels, e.g. a sudden change from 100 (liver tissue gray value) to 50 (surrounding tissue gray value). The gradient information of the liver edge in the quarter resolution image is focused on the whole outline, and the change trend of the liver edge on a larger scale can be reflected although the detail is inferior to that of the original resolution image. And fusing the edge gradient information under different resolutions through a trans-scale feature fusion layer to obtain more comprehensive and accurate liver edge feature representation.
Further, in the liver image, there may be some redundant edge response due to the presence of edge information of various structures. For example, in the vicinity of complex vascular networks and hepatic lobular structures within the liver, some edge responses due to noise or weak structures do not truly represent edges of important structures. The non-maxima suppression process eliminates edge responses that are not local maxima by comparing the edge intensities of each pixel to its neighborhood. For the edge of the liver vessel, only the pixels truly representing the vessel boundary (with maximum edge intensity) will be preserved, and the other weaker edge responses will be suppressed. In this process, a comparison neighborhood, for example, a 3 x 3 pixel neighborhood (which may be adjusted according to the actual situation) may be set, in which the edge intensities are compared.
Further, for the surface or internal structure of the liver, these edge points constitute the vertices of a triangular mesh. The triangular mesh can be constructed to accurately reflect the shape change of the curved portion such as the hepatic fornix. Then, the curvature distribution and the area ratio index of each triangular patch of the triangular mesh are calculated. Assuming that a certain triangular patch on the liver is composed of three vertex coordinates (x 1, y1, z 1), (x 2, y2, z 2) and (x 3, y3, z 3) (here, three-dimensional image coordinates are assumed, and the z axis represents the depth direction), the curvature distribution is determined by calculating geometrical parameters such as the side length and angle of the triangle. For example, if the area of the triangular patch is relatively flat, the curvature value may be small, such as 0.1 (assuming a unit of curvature), and if the curvature value is large, such as 0.5, in the curved portion of the liver. The area ratio index is obtained by calculating the ratio of the area of each triangular patch to the total area of the whole triangular mesh, for example, the area of a certain triangular patch is 10 square pixels (assumed), and the total area of the whole triangular mesh is 100 square pixels, and the area ratio is 0.1.
And S135, carrying out region division on the triangular meshes according to the curvature distribution and the area ratio index, generating a sub-mesh set corresponding to different tissue types, and carrying out statistics on the space-time variation variance of the blood flow signal intensity in each sub-mesh of the sub-mesh set to generate a local feature map set containing edge geometric features and hemodynamic features.
In the liver, different sub-grids such as a hepatic parenchymal region and a vascular region can be divided according to these indices. For example, for areas with a smaller curvature and a larger area occupation may be divided into hepatic parenchyma subgrid, while for areas with a larger curvature and related to vessel trend, the area is divided into vessel subgrids. The space-time variance of the blood flow signal intensities is counted in each sub-grid of the sub-grid set. For a relatively slow and steady blood flow within the hepatic parenchymal subgrid, assuming that the mean value of the blood flow signal intensity is 80 (assuming intensity units) over a period of time (e.g., 5 seconds, assuming continuous scan time of the ultrasound device), the fluctuation range is small, and the space-time variance may be 5 (assuming variance units). In the vascular subgrid near the lesion area, the fluctuation of the blood flow signal intensity is larger due to the influence of the lesion on the blood flow, and the average value of the blood flow signal intensity is 100 in the same 5 seconds, but the fluctuation range is larger, and the space-time variation variance may be 20. From this statistics, a set of local feature maps is generated that includes edge geometry features and hemodynamic features.
In one possible implementation, step S140 includes:
step S141, inputting the local feature map set into a long-short time memory network of a space-time feature alignment module for time sequence state transfer, generating a time dimension feature vector containing a continuous inter-frame focus form evolution rule, inputting the time dimension feature vector into a deformable convolution layer for space deformation modeling, and outputting a space deformation feature matrix matched with the tissue structure nonlinear deformation.
In this embodiment, first, a local feature map set of a liver patient is input into a long short time memory network (LSTM) of a spatio-temporal feature alignment module for time-series state transfer. In the ultrasound image sequence of the liver, successive multi-frame images contain information about the change over time of the liver lesion, if present. It is assumed that the ultrasound image sequence comprises 10 frames of images (the number of frames is determined according to the duration of the actual scan and the frame rate of the device), each frame of images comprising previously generated local feature maps comprising information such as the edge geometry and hemodynamic features of the liver. The LSTM network gradually learns the form evolution rule of the focus among the continuous frames by carrying out time sequence state transfer on the local feature images corresponding to the 10 frames of images. For example, in the first frame of image, the suspected lesion area in the liver appears to be approximately circular, with a diameter of about 5 pixels (assuming that one pixel in the image represents an actual length of 0.1 mm), the shape of the lesion area gradually becomes elliptical as time goes by to the 5 th frame of image, the major axis direction goes along a certain blood vessel in the liver, the major axis length becomes 8 pixels, and the minor axis length becomes 4 pixels. The variation information of the focus in shape, size and relative position is encoded into a time dimension feature vector. The time dimension feature vector contains information of a plurality of dimensions, each dimension corresponds to different feature parameters, such as a shape change parameter, a position movement parameter, a relation change parameter with surrounding tissues and the like, and the specific numerical values of the parameters are obtained through calculation of an LSTM network according to the actual change condition of a lesion area in a liver image.
Then, the liver tissue structure may be non-linearly deformed due to physiological movements of the liver itself (e.g. respiratory movements, liver displacements and deformations caused by heart beats) and possibly the influence of lesions on surrounding tissues. The deformable convolution layer is capable of modeling such nonlinear deformation of liver tissue structures based on information in the time-dimensional feature vector. For example, in a region of the liver, which is normally a relatively flat portion of the liver, the liver parenchyma is depressed by compression in the region due to proximity to a growing lesion. The deformable convolution layer outputs a spatial deformation feature matrix matched with the nonlinear deformation of the tissue structure by analyzing relevant parameters in the time dimension feature vector. The element values in the spatial deformation feature matrix reflect the degree and direction of deformation of the liver tissue at different positions, for example, in the above-mentioned concave region, the corresponding matrix element may be expressed as a displacement of-2 pixels in the x-direction (assuming one direction of the image coordinate system) (representing 2 pixels shifted towards the negative x-axis direction), a displacement of-1 pixel in the y-direction, and a displacement of-0.5 pixel in the depth z-direction (if a three-dimensional image) (the displacement value is calculated by the deformable convolution layer according to the actual deformation condition).
Step S142, splicing the time dimension feature vector and the space deformation feature matrix to generate a space-time joint feature matrix, inputting the space-time joint feature matrix into a multi-head self-attention mechanism of a multi-granularity attention distribution module, calculating global correlation weights among different feature channels, and outputting a global attention weight matrix.
In this embodiment, the spatio-temporal joint feature matrix integrates the feature information of liver lesions in time and space. In the feature representation of the liver ultrasound image, there are a plurality of feature channels, such as channels representing texture features of liver tissue, blood flow signal feature channels, edge feature channels, etc. The multi-headed self-attention mechanism computes global correlation weights between different characteristic channels. The importance of different characteristic channels is different for judging abnormal regions of the liver. For example, in determining whether a liver lesion is malignant, the blood flow signal characteristic channel may have a high correlation with the malignancy of the lesion, as malignant lesions are often accompanied by abnormal blood flow supply. It is assumed that the correlation weight of the blood flow signal characteristic channel with the lesion judgment is calculated to be 0.6 (the weight value is calculated according to the data characteristic through a multi-head self-attention mechanism), and the correlation weight of the tissue texture characteristic channel is calculated to be 0.3, and the correlation weight of the edge characteristic channel is calculated to be 0.1. Through such a calculation, a global attention weight matrix is output, each element of which represents a global correlation weight of the corresponding feature channel.
Step S143, inputting the global attention weight matrix into a gating linear unit for feature channel screening, reserving a feature channel related to a current classification task, generating a gating screening feature vector, inputting the gating screening feature vector into a local window attention calculation layer, enhancing the spatial relevance of adjacent feature nodes, and generating a local window attention weight matrix.
In this embodiment, the gating linear unit retains a characteristic channel related to the current classification task (determining the abnormal liver region) according to the weight value in the global attention weight matrix. For example, for feature channels with weights below 0.2 (e.g., the edge feature channels described above), the gating linear unit may screen out the feature channels, while the higher-weight blood flow signal feature channels and tissue texture feature channels are emphasized. And generating a gating screening feature vector after screening. The gating screening feature vector mainly comprises feature information which is most valuable for judging abnormal liver areas.
Next, in the liver image, there is a certain correlation in space between feature nodes between adjacent liver lobules or vessel branches. The local window attention calculation layer enhances the spatial relevance of adjacent feature nodes by analyzing feature information in the gating screening feature vector. For example, for two adjacent liver leaflet regions within the liver, there is some spatial continuity and correlation of their tissue texture characteristics and blood flow signal characteristics. The local window attention calculating layer generates a local window attention weight matrix by calculating the association weight between the feature nodes corresponding to the two areas. The element values in the local window attention weighting matrix reflect the strength of spatial correlation between adjacent feature nodes, e.g. for two highly correlated liver lobular regions, the correlation weight between them may be 0.8 (where the weight values are calculated by the local window attention calculation layer based on the actual feature correlation).
Step S144, the global attention weight matrix and the local window attention weight matrix are combined in a weighted mode, a multi-granularity attention distribution map is generated, and the multi-granularity attention distribution map is overlapped to the space-time joint feature matrix through residual connection, so that an optimized classification feature vector is generated.
In this embodiment, the multi-granularity attention distribution map integrates the attention information of different granularities (global and local). For example, for a characteristic channel with a weight of 0.5 in the global attention weight matrix and a weight of 0.4 in the local window attention weight matrix, the final weight of the characteristic channel may be calculated according to a certain weighting rule, assuming 0.45 (the weighting rule here may be a simple linear weighting or other rule defined according to the network structure) in the multi-granularity attention distribution map. And superposing the multi-granularity attention distribution atlas to the space-time joint feature matrix through residual connection to generate an optimized classification feature vector. Residual connection is beneficial to retaining partial information in the original space-time joint feature matrix, avoiding information loss in the multi-layer network processing process, and the optimized classification feature vector integrates multiple aspects of attention information and original space-time features.
Step S145, inputting the optimized classification feature vector into a full-connection classification layer, and outputting classification probability distribution and a spatial topological relation map of an abnormal region in each frame of standardized ultrasonic image.
In the context of liver ultrasound images, a classification probability distribution and a spatial topological relation map of an abnormal region (such as a possible tumor, a fibrosis region and the like in the liver) in each frame of standardized ultrasound image are output. The classification probability distribution indicates the size of the probability that each region of the liver is an abnormal region, for example, the probability that a region in the liver is calculated as an abnormal region is 0.8, which means that the region is 80% likely to be abnormal based on a previous series of feature analysis and model calculation. The spatial topological relation map accurately depicts the spatial relation between the abnormal region and the surrounding normal tissue, blood vessels and other structures, for example, whether the abnormal region is close to a large blood vessel, whether the boundary between the abnormal region and the surrounding liver tissue is clear or fuzzy, whether the abnormal region invades surrounding liver lobules and the like. Such information is of great importance for accurate diagnosis of liver disease, assessment of lesion severity, and subsequent treatment planning.
In one possible implementation, step S150 includes:
And step S151, mapping the classification probability distribution to the space coordinates of the standardized ultrasonic image sequence, and generating an abnormal region covering layer with semitransparent color codes.
In this embodiment, for the diagnosis process of a liver patient, first, the classification probability distribution of the liver patient is mapped to the spatial coordinates of the standardized ultrasound image sequence, and an abnormal region cover layer with semitransparent color coding is generated. In the standardized ultrasound image sequence of the liver, each pixel has its corresponding spatial coordinates. The classification probability distribution indicates the size of the probability that each region of the liver is an abnormal region. For example, for a pixel point in a liver image, whose coordinates are (x, y) (assuming an image coordinate system), if the probability that the point is classified as an abnormal region is 0.8 (the probability value is calculated from the previous classification), the pixel point is coded and marked with a specific translucent color. For areas with a high probability of abnormality, such as areas with a probability of greater than 0.7, highly suspected lesions are represented by covering with a red translucent color, for areas with a probability of between 0.4 and 0.7, areas with a probability of possibly having a risk of lesions are represented by covering with a yellow translucent color, and for areas with a probability of less than 0.4, normal or low risk areas are represented by covering with a green translucent color. Thus, the whole liver image is covered with an abnormal region covering layer with color codes, and doctors can intuitively see the pathological change risk distribution conditions of different regions of the liver.
Step S152, inputting the space topological relation map into a three-dimensional rendering engine, and constructing a three-dimensional rendering model for marking the vascular branches and the focus infiltration range.
For example, a spatial topological relationship map of the liver contains spatial relationship information between various structures within the liver (e.g., blood vessels, liver lobules, diseased regions, etc.). The three-dimensional volume rendering engine builds a three-dimensional model of the liver from this information. In the three-dimensional model, blood vessel branches in the liver are marked clearly, and information such as trend, diameter and the like of portal veins and branches thereof can be accurately displayed. For lesion infiltration scope, if there is a tumor lesion in the liver, the model will accurately show the location, size, and infiltration relationship of the tumor to surrounding tissue within the liver. Assuming that the tumor in the liver is located in the right lobe of the liver, its center coordinates are (x 1, y1, z 1) (where the z axis represents the depth direction), and the diameter is 10 pixels (assuming that one pixel represents the actual length of 0.1 mm), the boundary of the tumor periphery and the surrounding liver tissue is represented in different colors or textures in the model to show the degree of infiltration thereof.
And step 153, performing spatial registration on the abnormal region covering layer and the three-dimensional rendering model to generate a visual basic frame fusing the two-dimensional probability distribution and the three-dimensional anatomical structure.
In this process, it is ensured that each pixel point in the abnormal region coverage layer is spatially exactly matched with a corresponding position in the three-dimensional volume rendering model. For example, the high risk areas of the red translucent markers in the abnormal region overlay are also accurately located at corresponding locations within the liver in the three-dimensional volume rendering model. Through the spatial registration, the generated visual basic frame fuses two-dimensional probability distribution information (abnormal region coverage layer) with three-dimensional anatomical structure information (three-dimensional rendering model), and a doctor can view two-dimensional risk distribution and three-dimensional structure relation of the liver in one view at the same time, so that the health condition of the liver can be more comprehensively known.
Step S154, extracting hemodynamic characteristic parameters in the classification probability distribution, searching historical case characteristics with similar hemodynamic characteristic parameters in a medical image database, and generating a case similarity comparison result.
In the classified probability distribution of the liver patient, the hemodynamic characteristic parameter includes information such as blood flow velocity, blood flow rate, blood flow direction, and the like of blood vessels in the liver. For example, the blood flow rate of a major blood vessel in the liver is 50 cm/s (this value is obtained from previous ultrasound image analysis) and the blood flow is 100 ml/min. In the medical image database, image data of a large number of historical cases of liver disease and their associated features are stored. By retrieving the hemodynamic characteristic parameters of the cases in the database, historical cases with similar hemodynamic characteristics to the current patient are found. Assuming that 3 historical cases are found, their hemodynamic characteristics differ from the current patient by within a certain allowable range (e.g., blood flow velocity differences are within + -10% and blood flow differences are within + -15%). And then comparing the historical cases with the current patient in detail, including comparison of the size of the liver, the position of the lesion, the type of the lesion and the like, and generating a case similarity comparison result.
Step S155, associating the case similarity comparison result with the visual basic frame to generate a side comparison view and a key difference parameter labeling layer, matching a preset treatment scheme flow chart based on the key difference parameter labeling layer to generate a logic association table of risk level labels and treatment suggestions, and embedding the logic association table into the visual basic frame to generate an interactive three-dimensional visual interface comprising an interactive time axis control, a real-time ultrasonic playback function and a voice broadcasting engine.
In the side contrast view, relevant information of the current patient and the history case is displayed at the same time. For example, a three-dimensional model of the liver of the current patient is displayed with an abnormal region overlay on one side, and information such as its hemodynamic characteristics, liver size, etc. is displayed, and the same type of information of the historical case is displayed on the other side. The key discrepancy parameter labeling layer highlights the important discrepancy between the current patient and the historical case. For example, if the current patient has a liver tumor size of 20 cubic millimeters and the tumor size in the historical case is 15 cubic millimeters, this size difference is explicitly noted in the key difference parameter labeling layer. And matching a preset treatment scheme flow chart based on the key difference parameter labeling layer, and generating a logic association table of the risk level label and the treatment suggestion. The preset treatment scheme flow chart comprises different treatment schemes and corresponding risk levels formulated according to different liver disease characteristics (such as lesion size, position, hemodynamic characteristics and the like). For example, if a liver tumor is less than 10 cubic millimeters in size and has normal hemodynamic characteristics, the risk level may be low, the corresponding treatment recommendation may be a periodic observation, and if the tumor is between 10-20 cubic millimeters in size, the risk level is medium risk, the treatment recommendation may be a medical treatment or a local intervention. And generating a logic association table of the risk level label and the treatment advice according to the difference between the current patient and the historical case and the preset rules.
Finally, the interactable timeline control allows a physician to view liver ultrasound images and related analysis results at different points in time. For example, the physician may go back to a previous ultrasound examination via the time axis to see the state of the liver at that time and the analysis results. The real-time ultrasound review function allows the physician to review the real-time course of the liver ultrasound scan again in order to more carefully view certain suspicious regions. The voice broadcasting engine can broadcast information such as diagnosis results, risk grades, treatment suggestions and the like, and a doctor can conveniently acquire the information in the operation process. For example, the voice broadcast can clearly say that "the right lobe of the liver of the current patient has suspected tumor lesions, the risk level is medium risk, and the treatment advice is drug treatment or local intervention treatment" and the like. The interactive three-dimensional visual interface provides a comprehensive and convenient diagnostic tool for doctors, and is beneficial to improving the diagnostic accuracy of liver diseases and the rationality of treatment decisions.
In one possible embodiment, the method further comprises:
step S210, inputting a training sample library with a multi-dimensional label into a data enhancement module, and executing random elastic deformation, local pixel shielding and multi-angle affine transformation to generate an expanded countermeasure training sample set.
In detail, the training sample library comprises a plurality of liver ultrasonic image samples, and each sample is provided with a multi-dimensional label which comprises information of health condition (normal or lesion type), hemodynamic characteristic parameters, tissue elasticity parameters and the like of the liver. For random elastic deformation operation, taking a liver region in a liver ultrasonic image as an example, the shape of the liver is changed according to a certain elastic deformation rule in the coordinate space of the image. Assuming a liver image of 500 x 600 pixels in size (where the pixel size is determined by the resolution of the ultrasound device imaging), during random elastic deformation, certain pixels at the liver edge may be displaced in the x-direction (horizontal) and y-direction (vertical). For example, a pixel point on the upper edge of the liver, originally having coordinates (200, 100), may become (203,98) after elastic deformation, and this deformation simulates the image deformation caused by respiration of the patient, body movement or physiological movement of the liver itself (such as micro displacement of the liver caused by heart beat) during actual ultrasonic scanning.
Local pixel occlusion is the random selection of pixels in small areas of the image for occlusion. For liver images, small areas inside the liver may be selected, for example, a region of 10 x 10 pixels in the right lobe of the liver (where the region size is determined according to a preset occlusion rule) for occlusion. This simulates the partial occlusion of tissue by other structures or local disturbances in the image acquisition process that may occur in an actual scan. The multi-angle affine transformation comprises operations such as rotation, scaling and translation on the image. Taking rotation as an example, the liver ultrasound image is rotated about the center of the image by an angle, such as 15 ° (this angle is a randomly determined affine transformation parameter). And expanding samples in the original training sample library through the random elastic deformation, the local pixel shielding and the multi-angle affine transformation operation to generate an expanded countermeasure training sample set. The number of samples in the challenge training sample set is significantly increased compared with the original training sample library, and the sample set contains more diversified sample forms, thereby being beneficial to improving the generalization capability of the subsequent network.
Step S220, inputting the expanded countermeasure training sample set into a focus loss function of a first training stage, and optimizing an inter-class balance weight parameter of a cascade deep classification network.
In this cascade of deep classification networks, there are different categories, such as liver normal, liver fibrosis, liver tumor etc. The focus loss function aims to solve the problem of unbalanced sample numbers of different types during training. Suppose there are 500 normal liver samples, 300 liver fibrosis samples, and 200 liver tumor samples in the augmented challenge training sample set. Due to the difference in the number of samples, the network may tend to better classify a larger number of normal samples, while ignoring a smaller number of fibrotic and tumor samples, under conventional loss functions. The focus loss function makes the network more concerned with the minority class samples which are difficult to classify by weighting and adjusting the losses of the different class samples. During training, a loss value of each sample is calculated according to the focus loss function, and the inter-class balance weight parameters of the cascade depth classification network are updated according to the loss values. For example, for liver tumor samples, the focus loss function increases the weight of the loss of the liver tumor samples because of the small number and difficult classification, so that the network pays more attention to the study of the characteristics of the liver tumor samples in the study process, and the balance weight parameters among the classes are adjusted to improve the classification accuracy of the samples of different classes.
Step S230, inputting the optimized balance weight parameters between classes into an countermeasure training strategy in a second training stage, and generating a countermeasure disturbance sample set through gradient symbol attack.
In the countermeasure training strategy, a countermeasure disturbance sample set is generated by a gradient sign attack method based on the optimized balance weight parameters among classes. For a cascade depth classification network, it is assumed that an input liver ultrasound image sample propagates forward through the network to obtain a prediction result. During a gradient sign attack, the gradient of the loss function with respect to the input samples is calculated from the prediction results and the real labels. Taking a liver ultrasound image sample as an example, the image data is a matrix of 500×600×3 (width×height×number of channels, where the number of channels is 3 representing a color image). The calculated gradient represents the rate of change of the loss function with the input image pixel value. The immunity is then generated from the sign (positive or negative) of the gradient, for example, at a certain pixel of the image, the value of that pixel is increased if the gradient is positive, and the value of that pixel is decreased if the gradient is negative. In this way, samples in the training sample set are perturbed, generating an anti-perturbation sample set. These challenge samples can increase the robustness of the network to challenge samples, i.e. increase the correct classification ability of the network in the face of input samples that may be maliciously modified or have disturbances.
Step S240, inputting the disturbance countermeasure sample set into a knowledge distillation framework of a third training stage, and compressing model complexity parameters of the student network by using a soft label transfer mechanism of the teacher network.
In the knowledge distillation framework, there is a teacher network and a student network. The teacher network is a pre-trained network with high accuracy and complexity, predicts the disturbance-resistant sample set, and the obtained prediction result exists in the form of soft labels. The soft tag contains not only probability information that the sample belongs to each category, but also more uncertainty information. For example, for an anti-disturbance sample of a liver ultrasound image, the teacher network predicts that it is a normal liver probability of 0.6, a liver fibrosis probability of 0.3, and a liver tumor probability of 0.1. A student network is a relatively simple, low model complexity network that attempts to learn useful information from the soft labels of the teacher network. Through the soft label transmission mechanism, the student network does not need to learn complex characteristic representation again, but obtains information such as relations among categories from soft labels of the teacher network, so that model complexity parameters of the student network are compressed on the premise of keeping certain accuracy. For example, a student network may have more convolution layers and neuron connections, and after knowledge distillation, some unnecessary convolution layers or neuron connections are removed, thereby reducing the complexity of the model and improving the training efficiency and generalization ability of the network.
And S250, inputting the compressed model complexity parameters into a verification set monitoring module, triggering an early-stopping mechanism according to the classification precision and the generalization performance, and rolling back to an optimal weight snapshot.
The validation set contains a portion of the liver ultrasound image samples and their labels independent of the training set. And applying the network corresponding to the compressed model complexity parameter to the verification set for testing. In this process, the classification accuracy of the network on the validation set is calculated, e.g., the ratio of the number of samples predicted to be correct to the total number of validation samples is calculated. Meanwhile, the generalization performance of the network is evaluated, the generalization performance can be measured by comparing the performance on the verification set with the performance on the training set, and if the performance on the verification set is not greatly different from the performance on the training set, the network is better in generalization capability. Assuming that the verification set has 200 samples, when the classification accuracy reaches 90% (the value is a performance index set according to actual requirements) and the generalization performance is good, the verification set monitoring module can consider that the current network has reached a better state. If the classification accuracy begins to drop or the generalization performance becomes worse in the subsequent training process, the early shutdown system will be triggered and the network will roll back to the weight snapshot when the optimal classification accuracy and generalization performance were reached before. The weight snapshot saves the weight parameters of the network in the optimal state, and by rolling back to the state, the problems of over fitting and the like of the network in the over training process can be avoided, and the performance of the network is ensured.
Step S260, inputting the optimal weight snapshot into a dynamic quantization encoder to generate a network weight parameter for deployment of 8-bit integer coding.
Step S270, inputting the network weight parameters for deployment into an online increment learning module, and updating the parameters of the full-connection layer through clinical feedback data to adapt to the imaging characteristic difference of the ultrasonic probe.
The dynamic quantization encoder carries out quantization processing on the weight parameters in the optimal weight snapshot. In the original network weight parameters, it may be represented by a high precision data type (e.g., 32-bit floating point number). The dynamic quantization encoder converts these weight parameters into an 8-bit integer code. For example, for a weight parameter representing a classification characteristic of liver lesions, the original value is 0.12345 (32-bit floating point number representation), and after being processed by a dynamic quantization encoder, the original value is converted into an 8-bit integer code value (the specific conversion process is performed according to a quantization algorithm). The network weight parameters for the deployment of the 8-bit integer codes generated in the way greatly reduce the storage requirement and the calculation amount of the network on the premise of keeping certain accuracy, so that the network is more suitable for being deployed in actual medical equipment (such as ultrasonic diagnostic equipment). The network weight parameter for deployment is input into an online increment learning module, and the full-connection layer parameter is updated through clinical feedback data to adapt to the imaging characteristic difference of the ultrasonic probe. In practical clinical applications, different ultrasound probes may have different imaging characteristics, such as differences in imaging resolution, imaging sharpness for different tissues, etc. The clinical feedback data comprises liver ultrasonic images obtained by imaging according to different ultrasonic probes in the actual use process and corresponding diagnosis results. The parameters of the full-connection layer are updated through the data, for example, if the display of the liver edge is not clear enough in a liver image obtained by imaging a certain ultrasonic probe, so that some deviation occurs in the classification process, the online increment learning module can adjust the weight parameters in the full-connection layer according to the clinical feedback data, so that the network can better adapt to the imaging characteristic differences of different ultrasonic probes, and the diagnosis accuracy is improved.
In one possible embodiment, the method further comprises:
Step S310, comparing the output confidence entropy value of the cascade deep classification network with a preset threshold value, triggering an uncertainty early warning event, inputting the uncertainty early warning event into an anti-network reconstruction module, calculating the reconstruction error distribution of the current ultrasonic image, performing conflict detection on the reconstruction error distribution and the classification probability distribution, positioning an artifact interference area and triggering a standby classification model switching instruction.
The confidence entropy reflects the degree of uncertainty of the cascaded deep classification network on the classification result. For example, in the classification of the ultrasound image of the liver, if the probability that a certain area of the liver is normal is 0.8, the probability that the area is lesion is 0.2, and the confidence entropy value can be calculated according to the calculation formula of entropy (- Σp (i) ×log (p (i)), where p (i) is the probability of each class. Assuming that the preset threshold is 0.5, when the calculated confidence entropy value is greater than the preset threshold, an uncertainty early warning event is triggered. This means that the classification result of the current liver ultrasound image by the network is not well defined, and there may be some interference factors or limitations of the network itself.
For liver ultrasound images, the image size is assumed to be 500 x 600 pixels, each pixel having its corresponding feature value. The reconstruction error distribution is obtained by generating an countermeasure network to attempt to reconstruct the original image, and calculating the difference between the reconstructed image and the original image. For example, in a 10×10 pixel sub-region of a liver region, the pixel value of the liver tissue in the original image averages 100 (assuming that the value represents a certain characteristic intensity of the liver tissue), and the pixel value of the corresponding region in the reconstructed image averages 110, so that there is a certain reconstruction error in the region. And carrying out conflict detection on the reconstruction error distribution and the classification probability distribution. If in areas where the classification probability is high (e.g. 0.8 is normal), the reconstruction error is large, which may imply that there is an artifact interference. By such collision detection, an artifact interference region can be located. For example, a region located to one 20×20 pixel in the liver image is an artifact interference region whose coordinate range is ((x 1, y 1) - (x 2, y 2)), and then a standby classification model switching instruction is triggered.
Step S320, inputting the artifact interference region coordinates and the model switching instruction into a self-healing log construction module, recording input data characteristics and output distribution deviation degree, generating a self-healing log, inputting the self-healing log into a graph neural network trainer, constructing a fault knowledge graph of an abnormal event association relationship, inputting the fault knowledge graph into a network architecture optimizer, and dynamically adjusting the number of attention heads and the convolution kernel size of a cascade type depth classification network.
For the input data features, features of the liver ultrasound image near the artifact interference region, such as hemodynamic feature parameters, tissue elasticity parameters, etc., are recorded. Assuming that the blood flow velocity is measured at 40 cm/s (which is extracted from the raw ultrasound image data) near the artifact interference region, the tissue elasticity parameter is a particular value. The output distribution deviation degree refers to the degree of difference between the classification result and the classification result under the condition of no interference in the presence of artifact interference. For example, the probability of a region of the liver being classified as normal is 0.9, the probability of a region of the liver being classified as a possible lesion in the presence of an artifact disturbance is 0.6, and the change in the probability is the degree of deviation of the output distribution. By recording this information, a self-healing log is generated.
The graph neural network trainer can analyze the data in the self-healing log to find out the association relation between different abnormal events (such as artifact interference at different positions, output distribution deviation degree at different degrees and the like). For example, it was found that when the artifact interference region is located at the edge of the liver and the blood flow velocity measurement is between 30-50 cm/s, this correlation is built into the fault knowledge graph, which often results in a shift of the classification result towards the lesion. The fault knowledge graph graphically represents the association relationships, the nodes possibly represent different abnormal events or data features, and the edges represent the association strength between the abnormal events or data features.
For cascaded deep classification networks, the number of attention headers and the convolution kernel size have a significant impact on the performance of the network. If the fault knowledge graph is displayed under certain specific ultrasonic image characteristics of the liver (such as specific hemodynamic characteristics and artifact interference modes), the classification effect of the network is poor, and the network architecture optimizer adjusts the attention head number according to the information. Assuming that the original attention header number is 8, it was found that reducing to 6 attention headers may improve the performance of the network in this case, based on analysis of the fault knowledge graph. For convolution kernel sizes, such as the original convolution kernel size of 3 x3 pixels, it is possible to adjust to 5 x 5 pixels to better capture features in the ultrasound image of the liver, particularly features near the artifact interference region.
And step S330, inputting the optimized network architecture parameters into a full-scale self-checking module, repairing numerical overflow nodes in the weight matrix, resetting internal state parameters of the optimizer, and synchronizing the repaired network architecture parameters and the self-healing rule base to the distributed edge nodes to maintain consistency of exception handling strategies among multiple devices.
Finally, in the weight matrix, there may be value overflow nodes due to previous adjustment or value overflow in the calculation process. For example, a certain weight value should be originally between-1 and 1, but becomes a very large value (e.g., 1000) due to calculation errors or data type limitations, which is a value overflow node. The full self-test module restores the value overflow nodes to a reasonable value range. Meanwhile, the internal state parameters of the optimizer are reset, so that the optimizer can be ensured to carry out subsequent training or adjustment in a correct state. And then synchronizing the repaired network architecture parameters and the self-healing rule base to the distributed edge nodes. In a multi-facility diagnostic system, distributed edge nodes may be distributed among different medical sites (e.g., different hospital departments or different regional medical centers). By synchronizing the repaired network architecture parameters with the self-healing rule base, each distributed edge node can keep a consistent exception handling strategy, so that the consistency of diagnosis results and handling modes among different devices is ensured when liver ultrasonic images are handled, and the reliability and accuracy of the whole diagnosis system are improved.
In a possible implementation manner, after the step S150, the method further includes:
Step S410, receiving an interaction instruction triggered by a user through a time axis control or a real-time ultrasonic playback function in the interactive three-dimensional visual interface, and analyzing a parameter adjustment request and focus region re-labeling coordinates in the interaction instruction.
For example, when a doctor looks at the visualization interface of a liver patient, the doctor returns to an ultrasonic image at a certain previous time point by using a time axis control, or looks back at a certain segment in the scanning process through a real-time ultrasonic playback function, which all generate interactive instructions. Therefore, the parameter adjustment request and the focus region re-labeling coordinates in the interaction instruction can be analyzed. Assuming that there may be some inaccuracy in the diagnosis before the doctor finds in the viewing process, the parameter adjustment request in the interactive instruction contains information of the confidence threshold offset in order to adjust the decision boundary of the classification probability distribution. For example, if the confidence threshold value, which is originally determined that a region of the liver is an abnormal region, is 0.7, the doctor adjusts the threshold value to 0.6 by the interactive instruction, that is, the confidence threshold value offset is-0.1. Meanwhile, if the doctor finds that the coordinates of the previously marked focus area are inaccurate, the focus area is marked again, and new coordinate information is also contained in the interactive instruction.
And step S420, correcting the judgment boundary of the classification probability distribution according to the confidence threshold offset in the parameter adjustment request, generating a corrected abnormal region coverage layer, carrying out topology consistency check on the focus region remark coordinate and the space topology relation map, and updating the blood vessel branch connection path in the three-dimensional rendering model.
In a liver ultrasound image, the classification probability distribution originally determines the probability that each region is an abnormal region. Taking an area of the right lobe of the liver as an example, the probability of being an abnormal area is calculated to be 0.65, and the abnormal area is determined to be a normal area when the confidence threshold is 0.7. But when the confidence threshold is adjusted to 0.6, the region is included in the outlier region. In this way, the whole liver image is re-determined, and a corrected abnormal region cover layer is generated. In this cover layer, the color coding may change according to a new determination result, for example, an originally green (indicating normal) region may change to yellow (indicating suspected abnormality). And meanwhile, carrying out topological consistency check on the focus region remark coordinates and the space topological relation map. The spatial topological relation map comprises the spatial relation among blood vessels, tissues and focuses in the liver. For the remarked focal region coordinates, it is checked whether it maintains a reasonable topological relation with other structures in the map. For example, if the relative position of the lesion area and the surrounding blood vessels conforms to the physiological structure, if the remarked lesion area is close to a certain blood vessel, but the area is far away from the certain blood vessel in the previous spatial topological relation map, further examination and adjustment are needed. And updating the blood vessel branch connection path in the three-dimensional volume rendering model according to the verification result. If the position of the focus area changes, the connection relation with the blood vessel branches may be affected, for example, the focus is originally considered to be not associated with a certain tiny blood vessel branch, and after re-labeling and verification, the connection path of the blood vessel branches needs to be updated in the three-dimensional rendering model to accurately reflect the structural relation inside the liver.
And S430, re-registering the corrected abnormal region coverage layer and the updated three-dimensional rendering model to generate a secondary verification visualization frame, and extracting a change gradient of the hemodynamic characteristic parameters in the secondary verification visualization frame.
In this process, it is ensured that each pixel point in the modified abnormal region coverage layer exactly matches a corresponding position in the updated three-dimensional volume rendering model. For example, a region marked as yellow (suspected abnormality) in the corrected abnormal region cover layer is also accurately located at a corresponding position in the liver in the three-dimensional volume rendering model. Through this re-registration, the generated secondary verification visualization framework integrates the modified two-dimensional probability distribution information and the updated three-dimensional anatomical structure information. And then extracting the change gradient of the hemodynamic characteristic parameters in the secondary verification visualization framework. In liver images, hemodynamic characteristic parameters include blood flow velocity, blood flow, and the like. Assuming that in the previous visualization frame the blood flow velocity of a certain major vessel of the liver was 50 cm/s, in the second verification visualization frame the blood flow velocity becomes 45 cm/s due to the re-labeling of the lesion area or other factors, the gradient of the change in blood flow velocity is-5 cm/s. Similar calculation is performed on other hemodynamic characteristic parameters such as blood flow, and the change gradient of the hemodynamic characteristic parameters in the whole secondary verification visualization framework is obtained.
Step S440, performing incremental search in the medical image database based on the hemodynamic characteristic parameter variation gradient, screening out a historical case subset matched with the corrected classification probability distribution, and recalculating risk level weight coefficients in case similarity comparison results.
The medical image database stores a great number of historical case image data of liver diseases and relevant characteristics thereof. Taking the blood flow velocity change gradient of-5 cm/s as an example, historical cases with similar hemodynamic characteristic parameter change conditions are searched in a database. Assuming 1000 historical cases exist in the database, and analyzing the hemodynamic characteristic parameters of each case to screen out the cases which are similar to the change gradient of the hemodynamic characteristic parameters in the current secondary verification visualization framework. For example, historical cases with blood flow velocity changes in the range of-4 to-6 cm/s are screened out to form a subset of historical cases that match the modified classification probability distribution. And then recalculating risk level weight coefficients in the case similarity comparison result. In the previous case similarity comparison, the risk level weight coefficient may be determined according to the liver size, lesion position and other factors. The risk level weight coefficient is re-evaluated after adding a new factor of the change gradient of the hemodynamic characteristic parameter. For example, for a historical case, the risk level weight coefficient originally calculated according to the liver size and lesion location is 0.6, and after considering the gradient of the change of the hemodynamic characteristic parameter, the risk level weight coefficient may be adjusted to be 0.7 or 0.5, and the specific value depends on the influence of the new factor on the overall risk assessment.
Step S450, adjusting treatment proposal priority order in the logic association table according to the recalculated risk level weight coefficient, generating a dynamically optimized treatment proposal flow chart, and superposing the dynamically optimized treatment proposal flow chart and the secondary verification visualization framework to generate a final diagnosis report confirmed by a doctor.
The logical association table originally determines different treatment recommendations and priorities thereof according to factors such as risk levels. For example, for liver lesions with moderate risk levels, the previous treatment recommendation may be to first take a medication and observe for a period of time, if the condition does not improve and take into account the intervention, this is a priority determined based on the previous risk level weighting coefficients. When the risk level weight coefficient changes, for example, the risk level becomes higher, the priority of the treatment advice may be adjusted to directly perform the interventional treatment, and then the drug treatment is assisted. In this way, the treatment advice prioritization in the logical association table is adjusted according to the recalculated risk level weight coefficients, resulting in a dynamically optimized treatment advice flow chart. The dynamically optimized treatment recommendation flow chart is overlaid with a secondary verification visualization framework to generate a final diagnostic report after physician confirmation. In the final diagnosis report, doctors can simultaneously see the optimized treatment advice and information such as liver images, structural relations, related characteristic parameters and the like in the secondary verification visualization framework, and the doctor can conveniently confirm the final diagnosis.
Step S460, inputting the correction parameters and the remark coordinates in the final diagnosis report confirmed by the doctor into an online increment learning module, updating the full-connection layer weight of the cascade deep classification network, and synchronizing the updated full-connection layer weight to a standby classification model of the distributed edge node to finish consistency calibration of the cross-equipment diagnosis strategy.
For example, correction parameters such as confidence threshold offset, hemodynamic characteristic parameter variation gradient, and remarked lesion area coordinates in the final diagnostic report are input to the online delta learning module. And the online incremental learning module updates the full-connection layer weight of the cascade deep classification network according to the new information. Assuming that a certain weight is originally 0.5 in the full connection layer, the weight may be adjusted to 0.6 or 0.4 according to the influence of the correction parameter during the update process. The updated full connection layer weights are synchronized to the standby classification model of the distributed edge node. In a multi-facility diagnostic system, distributed edge nodes are distributed in different places, such as different hospital departments or different regional medical centers. By synchronizing the updated weights, the same diagnosis strategy is adopted by each device when liver ultrasonic image diagnosis is carried out, consistency calibration of the cross-device diagnosis strategy is completed, and accuracy and reliability of the whole diagnosis system are improved.
In one possible embodiment, after the step of completing the cross-device diagnostic strategy consistency calibration, the method further comprises:
step S510, periodically collecting multi-stage ultrasound review data of the target object in a treatment stage, and performing noise suppression and motion artifact compensation processing of the real-time ultrasound scanning signal sequence on each stage of ultrasound review data to generate a standardized follow-up ultrasound image sequence.
For example, it is set that a liver ultrasound review is performed on a patient every two weeks, and the ultrasound data obtained by each review is one period of the multi-period ultrasound review data. For each period of ultrasound review data, a prior noise suppression and motion artifact compensation process for the real-time ultrasound scan signal sequence is performed to generate a normalized follow-up ultrasound image sequence.
When noise suppression and motion artifact compensation processing is performed, wavelet transformation decomposition is performed first for each frame of ultrasonic scanning signal in the first-period ultrasonic review data. Assuming that the size of the liver image corresponding to the ultrasonic scanning signal is 500×600 pixels (the pixel size is determined according to the imaging resolution of the ultrasonic device), the high-frequency noise component and the low-frequency tissue signal component are separated after the decomposition by wavelet transformation. The high frequency noise component presents some signal fluctuations in the high frequency band, relatively low energy and irregular, possibly originating from electronic components of the ultrasound device, minor disturbances of the surrounding environment, etc. The low frequency tissue signal component contains the rough outline information of the main structures such as liver tissue, blood vessels and the like.
The high frequency noise component is then filtered layer by layer using an adaptive thresholding algorithm. For example, for the first layer of the high-frequency noise component, statistical parameters such as standard deviation and mean value of the layer signal are analyzed, and if the standard deviation is 5 (assumed unit), the threshold value of the layer is 15 according to a certain algorithm rule (for example, the standard deviation is multiplied by a coefficient, and the coefficient is assumed to be 3). The signal portions above the threshold are considered to be potentially useful signals to be preserved, and those below the threshold are determined to be noise and filtered out. This adaptive thresholding can effectively remove noise while preserving intermediate frequency signal components associated with vessel wall motion throughout the layer-by-layer filtering process. For vessels in the liver, the frequency of the signals generated by the blood flow is in the intermediate frequency range, for example, the frequency is between 1 and 5MHz (assumption), and the intermediate frequency signals have important significance for the subsequent analysis of the motion state of the vessel wall, the blood flow speed and the like.
And then, carrying out inverse wavelet transformation reconstruction on the filtered ultrasonic scanning signals to generate an ultrasonic image subjected to preliminary denoising. But the images are subject to motion artifacts due to the possible slight breathing or body movement of the patient during the review scan. The tissue displacement vector field between the ultrasonic scanning signals of the adjacent frames is calculated based on an optical flow method, and for the liver image, the moving direction and distance of the liver tissue between the adjacent frames are calculated. Assuming that the time interval between two adjacent frames of images is 0.1 seconds (which is a typical time interval determined by the scanning frame rate of the ultrasonic device), the optical flow method calculates the displacement vector of the liver tissue within the 0.1 seconds. For example, if a specific point on the liver (assuming that the coordinates are (x 1, y 1)) is shifted by (x 2, y 2) from the position in the first frame image to the position in the second frame image, the calculated displacement vector is ((x 2-x 1), (y 2-y 1)). According to the tissue displacement vector field, motion track compensation is carried out on the primarily denoised ultrasonic image, for example, a suspected lesion region in the liver has obvious displacement in the image due to respiratory motion of a patient before the motion track compensation, and after the motion track compensation, the position of the region in the image can more accurately reflect the real position of the region in the liver.
And then, gray scale normalization operation is carried out on the ultrasonic image after the motion compensation, and the pixel intensity is mapped to a preset standard dynamic range. Assuming that the preset standard dynamic range is 0-255 (which is a common image pixel intensity range), in the liver ultrasound image after motion compensation, the gray value range of all pixels in the image is counted first. For example, the gray scale value range of liver tissue may be 50-180 and the gray scale value range of blood vessels may be 30-100 (here, only assumed to be a general range). These raw gray values are then mapped to a standard dynamic range of 0-255 by means of a linear mapping. The purpose of this is to make the images of different phases and the images of different patients comparable in gray scale, facilitating subsequent analysis and processing. And after the gray scale normalization operation, obtaining a normalized ultrasonic image.
And finally, carrying out edge-preserving smoothing treatment on the normalized ultrasonic image by adopting an anisotropic diffusion algorithm, carrying out detail enhancement on the smoothed ultrasonic image by a bilateral filter, carrying out region matching on the ultrasonic image with enhanced details according to a preset anatomical structure template, and cutting out a standardized follow-up ultrasonic image sequence containing the boundary of the complete liver organ.
Step S520, inputting the standardized follow-up ultrasonic image sequence into an updated cascade depth classification network, and outputting follow-up classification probability distribution and spatial topology evolution map of an abnormal region in each period of follow-up ultrasonic image.
In a cascaded depth classification network, standardized follow-up ultrasound images of each phase are analyzed. For example, for a period of liver follow-up ultrasound image, the probability that a region of the liver is an abnormal region is 0.3 (the probability value is calculated according to analysis of image features by the network), which is one data point in the follow-up classification probability distribution. Meanwhile, the spatial topological evolution map can accurately describe the spatial relation change of the abnormal region and surrounding structures such as normal tissues, blood vessels and the like in the follow-up process. For example, a vessel that was originally in the vicinity of the liver at the time of initial diagnosis may change in distance from the lesion region to the vessel during follow-up, and such a change in spatial relationship is recorded in the spatial topological evolution map.
And step S530, carrying out time series alignment on the follow-up classification probability distribution and the initial classification probability distribution, calculating the area shrinkage rate of the abnormal region and the blood flow signal intensity attenuation coefficient, and generating a treatment response quantification index set.
Assuming that the area of the lesion region in the liver in the initial classification probability distribution is calculated as 100 square pixels by image analysis (assuming that one pixel represents an actual length of 0.1mm so that the area is 1 square cm), in the follow-up classification probability distribution of a certain period, the area of the lesion region is calculated as 80 square pixels (i.e., 0.8 square cm), the abnormal region area shrinkage rate is (100-80)/100=0.2, i.e., 20%. For the blood flow signal intensity attenuation coefficient, for example, when the blood flow signal intensity of a certain blood vessel in a lesion region of the liver at the time of initial diagnosis is 80 (assuming intensity units), and at a certain stage in the follow-up process, the blood flow signal intensity of the blood vessel in the lesion region becomes 60, the blood flow signal intensity attenuation coefficient is (80-60)/80=0.25, that is, 25%. In this way, the area shrinkage rate and the blood flow signal intensity attenuation coefficient of the abnormal region in each period are calculated, and a treatment response quantization index set is generated.
Step S540, inputting the spatial topological evolution map into a three-dimensional difference rendering engine, extracting a blood vessel branch morphology change vector and a focus infiltration volume change matrix, and generating a three-dimensional treatment response dynamic model.
In the three-dimensional difference rendering engine, according to information in the spatial topological evolution map, analyzing morphological changes of blood vessel branches in the liver in the follow-up process. For example, a certain blood vessel branch has a diameter of 3 pixels (assuming that one pixel represents an actual length of 0.1 mm) and a length of 20 pixels (i.e., 2 mm) at the time of initial diagnosis, and the diameter becomes 2.5 pixels and the length becomes 18 pixels during follow-up, and a blood vessel branch morphology change vector is obtained by calculating these changes. For the lesion infiltration volume change matrix, it is assumed that the lesion infiltration volume is calculated to be 100 cubic pixels (i.e. 1 cubic centimeter) by three-dimensional analysis of a lesion area in the initial diagnosis, and different lesion infiltration volumes are calculated according to space topology evolution maps of different periods in the follow-up process, for example, 80 cubic pixels (i.e. 0.8 cubic centimeter) at a certain period, and the data are formed into the lesion infiltration volume change matrix. A three-dimensional treatment response dynamic model is generated through the data, and the three-dimensional treatment response dynamic model can intuitively display the three-dimensional structural change condition of the liver in the treatment process.
Step S550, fusing the treatment response quantification index set and the three-dimensional treatment response dynamic model, generating a multi-stage treatment effect comparison report, and performing prognosis pattern matching on the multi-stage treatment effect comparison report and the historical treatment cases in the medical image database.
The multi-stage treatment effect comparison report integrates quantified indexes (such as abnormal region area shrinkage rate and blood flow signal intensity attenuation coefficient) and a three-dimensional structure change model (a three-dimensional treatment response dynamic model). And then performing prognosis pattern matching on the multi-stage treatment effect comparison report and the historical treatment cases in the medical image database. The medical image database stores a great number of historical treatment cases of liver diseases and relevant treatment effect data. And (3) matching each index and model in the multi-stage treatment effect comparison report with the historical treatment cases to find out the historical treatment cases most similar to the current treatment conditions of the patient.
Step S560, screening the optimal historical treatment cases according to the prognosis pattern matching result, extracting the corresponding treatment scheme flow chart and key node monitoring parameters, and generating the personalized treatment monitoring billboard.
For example, by matching the prognosis pattern, it is found that a certain historical treatment case is most similar to the treatment condition of the current patient, and the treatment scheme flow chart of the historical treatment case includes information such as the type, dosage, treatment period of the drug treatment, and examination items required to be performed at different stages in the treatment process. Key node monitoring parameters may include expected ranges of changes in liver function indicators (e.g., glutamic pyruvic transaminase, glutamic oxaloacetic transaminase, etc.) at a certain stage of treatment, expected changes in size of diseased areas of the liver, etc. And extracting the information to generate the personalized treatment monitoring board.
Step 570, comparing the monitoring parameter threshold value in the personalized treatment monitoring billboard with the real-time ultrasonic review data stream in real time, and automatically adjusting the drug dosage parameter and the review time interval in the treatment proposal flow chart when the deviation pre-warning event is triggered, so as to generate the self-adaptive treatment plan update instruction.
For example, the area of a liver lesion area set at a certain stage of treatment in a personalized treatment monitoring board should be reduced to a certain range, and if the area of the liver lesion area is found to not reach the expected reduced range in a real-time ultrasonic review data stream, a deviation early warning event is triggered. At this time, according to preset rules, the drug dosage parameters in the treatment recommendation flow chart are automatically adjusted, for example, the dosage of a certain drug is increased from 10 mg to 15 mg every day, and the review time interval is adjusted, for example, the original one-time every two weeks is changed to one-time every week, so as to generate the adaptive treatment plan update instruction.
Step S580, the self-adaptive treatment plan updating instruction and the multi-stage treatment effect comparison report are logically associated, a treatment closed-loop feedback file is generated, key decision parameters in the treatment closed-loop feedback file are returned to the online incremental learning module, and the subsequent classification network weight updating direction is optimized.
The treatment closed loop feedback file records the logic association relation between the self-adaptive treatment plan updating instruction and the multi-stage treatment effect comparison report, and comprises information such as the basis for adjusting the dosage of the medicine and the rechecking time interval, the expected treatment effect and the like. And returning key decision parameters (such as medicine dosage adjustment quantity, review time interval adjustment quantity and the like) to an online increment learning module, and optimizing the updating direction of the weight of the subsequent classification network by the online increment learning module according to the parameters. For example, if the improvement of the diseased region of the liver is found to be better after the drug dosage is increased, the online incremental learning module adjusts the weights associated with the drug dosage and the treatment effect in the cascaded depth classification network so as to more accurately reflect the relationship in the follow-up diagnosis and treatment advice, thereby improving the accuracy and adaptability of the whole diagnosis and treatment system.
FIG. 2 illustrates a schematic diagram of exemplary hardware and software components of an artificial intelligence based ultrasound image data classification system 100 that may implement the concepts of the present application provided by some embodiments of the present application. For example, the processor 120 may be used on the artificial intelligence based ultrasound image data classification system 100 and to perform the functions of the present application.
The artificial intelligence based ultrasound image data classification system 100 may be a general purpose server or a special purpose server, both of which may be used to implement the artificial intelligence based ultrasound image data classification method of the present application. Although only one server is shown, the functionality described herein may be implemented in a distributed fashion across multiple similar platforms for convenience to balance processing loads.
For example, the artificial intelligence based ultrasound image data classification system 100 may include a network port 110 connected to a network, one or more processors 120 for executing program instructions, a communication bus 130, and a storage medium 140 of different forms, such as a disk, ROM, or RAM, or any combination thereof. Illustratively, the artificial intelligence based ultrasound image data classification system 100 may also include program instructions stored in ROM, RAM, or other types of non-transitory storage media, or any combination thereof. The method of the present application may be implemented in accordance with these program instructions. The artificial intelligence based ultrasound image data classification system 100 also includes an Input/Output (I/O) interface 150 between a computer and other Input/Output devices.
For ease of illustration, only one processor is depicted in the artificial intelligence based ultrasound image data classification system 100. It should be noted, however, that the artificial intelligence based ultrasound image data classification system 100 of the present application may also include multiple processors, and thus the steps performed by one processor described in the present application may also be performed jointly by multiple processors or separately. For example, if the processor of the artificial intelligence based ultrasound image data classification system 100 performs steps a and B, it should be understood that steps a and B may also be performed jointly by two different processors or separately in one processor. For example, the first processor performs step a, the second processor performs step B, or the first processor and the second processor together perform steps a and B.
In addition, the embodiment of the invention also provides a readable storage medium, wherein computer executable instructions are preset in the readable storage medium, and when a processor executes the computer executable instructions, the ultrasonic image data classification method based on artificial intelligence is realized.
It should be noted that in order to simplify the presentation of the disclosure and thereby aid in understanding one or more embodiments of the invention, various features are sometimes grouped together in a single embodiment, figure, or description thereof.