US20200321100A1 - Systems and methods for improved analysis and generation of medical imaging reports - Google Patents

Abstract

Systems and methods for the improved analysis and generation of medical imaging reports are disclosed. In particular, the present disclosure provides systems and methods that may be used for the automated analysis of radiological information, such as medical images and related text statements for discrepancy analysis, accuracy analysis and quality assurance. Systems and methods may include receiving medical images and textual data, generating enhanced medical image data by applying an artificial intelligence module to the received medical images, generating structured text data by applying a natural language processing module to the received textual data, and generating improved medical image reports and/or alerts based on the generated enhanced medical image data and the generated structured text data.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS
• This application is a continuation of U.S. application Ser. No. 16/529,842, filed Aug. 2, 2019, which claims the benefit of U.S. Provisional Application No. 62/713,557, filed Aug. 2, 2018, which is incorporated by reference in its entirety.
TECHNICAL FIELD
• The present disclosure relates to systems and methods for the improved analysis and generation of medical imaging reports.
BACKGROUND
• Medical imaging may involve the use of magnetic resonance imaging (MRI), computed tomography (CT), ultrasound, nuclear medicine, X-ray exams and the like. Medical images are commonly analyzed and interpreted by radiologists. However, in conventional systems the analysis and interpretation of medical images is often a manual process. For example, a radiologist manually locates the areas and regions of interest and prepares a written report. The written report is then sent to the referring physician. Commonly, the written report is integrated into an electronic health record.
• Typically, a radiologists' interpretation of medical images is subjective. The accuracy of a radiologists' interpretation of a medical image is often dependent on factors such as (1) training, (2) experience, (3) expertise, and (4) time available to review a medical image. These factors often impact the accuracy and consistency of a radiologists' interpretation. Further, the time available to review a medical image may decrease as the number of conducted scans continues to increase and outpaces the number of radiologists. Additionally, the improvement in medical imaging technology may produce higher resolution imagery which requires more review time from radiologists, both of which places additional time pressures on radiologists.
• In conventional systems, a radiologists' interpretation is sent to the referring physician and usually no other processes occur on the analyzed images to verify accuracy and quality of interpretation. The images and report are archived and remain unreviewed.
• Further, rarely are the medical images or report analyzed to verify accuracy and quality of a radiologists' interpretation. In typical clinical settings, a manual peer-review (i.e., another peer radiologist reviews the case) is conducted on a small percentage of the annual case workload. Additionally, manual peer-review is conducted not as a measure for detecting missed findings, but as a method for understanding the general level of quality in the radiology department.
• It is estimated that up to 30% of radiologist reports may suffer from errors including misinterpretation (and misdiagnosis) of medical images. Additionally, it is estimated that up to 60% of incidental findings (i.e., an incidentally discovered mass or lesion that is detected in a medical image performed for an unrelated primary objective) may be missed. Further, as radiologists are commonly focused on answering the “reason for study” and interpret only a section of the images, there is a missed potential for early-stage detection in imaging.
SUMMARY
• The present disclosure is related to systems and methods for the improved analysis and generation of medical imaging reports. In particular, the present disclosure provides systems and methods that may be used for the automated analysis of radiological information, such as medical images and related text statements for discrepancy analysis, accuracy analysis and quality assurance. Accordingly, the analysis and generation of medical imaging reports may be improved. Further, the analysis and generation of electronic health records that integrate medical imaging reports may also be improved.
• Embodiments of the present disclosure may include a method for generating improved medical image reports having the steps of: receiving medical images and textual data, generating enhanced medical image data by applying an artificial intelligence module to the received medical images, generating structured text data by applying a natural language processing module to the received textual data, and generating improved medical image reports based on the generated enhanced medical image data and the generated structured text data.
• Embodiments of the present disclosure may also include a method for generating improved medical image reports including the steps of: receiving medical images and textual data, generating enhanced medical image data by applying an artificial intelligence module to the received medical images, generating structured text data by applying a natural language processing module to the received textual data, determining discrepancies between a received medical image report and the generated enhanced medical image data and the generated structured text data, and providing an alert to at least one of a user or a creator of the medical image report responsive to determining a discrepancy.
• Optionally, applying an artificial intelligence module may include applying a convolutional neural network. Additionally, the method may include training at least a regression convolutional neural network, a semantic segmentation convolutional network and a classification convolutional neural network. Applying a natural language processing module to the received textual data may include applying a recurrent neural network. Generating the clinical or non-clinical interface may include the generated enhanced medical image data. In some embodiments, the enhanced medical image data may be integrated into an electronic healthcare record. Generating improved medical image reports based on the generated enhanced medical image data and the generated structured text data may include identifying and storing each finding indicated in the enhanced medical image data, correlating each finding to its corresponding structured text data, identifying corresponding portions for each finding in a radiology report, identifying discrepancies between the corresponding portions and the corresponding structured text data for each finding, and generating improved medical image reports by augmenting the radiology report with the identified discrepancies. The method may also include the step of presenting the generated improved medical image report to a physician for approval. The method may also include the step of presenting an alert to a physician in the case of an identified discrepancy.
• Embodiments of the present disclosure may include a server system for generating improved medical image reports. The server system may be configured to receive medical images and textual data. Further, the server system may include an image processing unit configured to apply artificial intelligence computer vision techniques to the received medical images to generate enhanced medical image data, a text processing unit configured to apply natural language processing to the received textual data to generate structured text data, and a platform configured to generate an automated improved medical image report based on the generated enhanced medical image data and the structured text data.
• Embodiments of the present disclosure may also include a system for generating improved medical image reports including a server system configured to receive medical images and textual data. The server system may include an image processing unit configured to apply an artificial intelligence computer vision techniques to the received medical images to generate enhanced medical image data, a text processing unit configured to apply natural language processing to the received textual data to generate structured text data, and a platform configured to generate an alert responsive to a detected discrepancy between the generated enhanced medical image data, the generated structured text data, and the received medical image and textual data.
• Optionally, the artificial intelligence computer vision techniques may include a convolutional neural network. The server system may include a case manager configured to dispatch data and receive results from the image processing unit and the text processing unit. The server system may also include a platform configured to generate at least one of a clinical or non-clinical interface including the enhanced medical image data. The platform may be configured to incorporate the enhanced medical image data into an electronic healthcare record. In some embodiments, the natural language processing may include at least one recurrent neural network.
• Embodiments of the present disclosure may also include a non-transitory computer-readable medium storing instructions that, when executed on one or more processors, cause the one or more processors to receive medical images, receive textual data, generate enhanced medical image data by applying an artificial intelligence module to the received medical images, generate structured text data by applying a natural language processing module to the received textual data, and generate improved medical image reports based on the generated enhanced medical image data and the generated structured text data.
• Embodiments of the present disclosure may also include a non-transitory computer-readable medium storing instructions that, when executed on one or more processors, cause the one or more processors to receive medical images, receive textual data, generate enhanced medical image data by applying an artificial intelligence module to the received medical images, generate structured text data by applying a natural language processing module to the received textual data, and generate an alert based on a detected discrepancy between the generated enhanced medical image data, the generated structured text data, the received medical images and the received textual data.
• Optionally, generating the enhanced medical image data may include applying a convolutional neural network. Further, applying a natural language processing module to the received textual data may include applying at least one recurrent neural network. The one or more processors may be configured to generate at least one of a clinical or non-clinical interface including the enhanced medical image data. Additionally, the one or more processors may be configured to incorporate the enhanced health-care related image data into an electronic healthcare record. In some embodiments, generating improved medical image reports based on the generated enhanced medical image data and the generated structured text data may include the one or more processors being configured to identify and store each finding indicated in the enhanced medical image data, correlate each finding to its corresponding structured text data, identify corresponding portions for each finding in a radiology report, identify discrepancies between the corresponding portions and the corresponding structured text data for each finding, and generate improved medical image reports by augmenting the radiology report with the identified discrepancies.
BRIEF DESCRIPTION OF DRAWING
• Various embodiments of the present disclosure, together with further objects and advantages, may best be understood by reference to the following description taken in conjunction with the accompanying drawings, in the several figures of which like reference numerals identify like elements, and in which:
• FIG. 1 illustrates a platform built in accordance with an embodiment of the present disclosure.
• FIG. 2A illustrates a component (computer vision processing servers) of the platform illustrated inFIG. 1 in a first state (training).
• FIG. 2B illustrates a component (computer vision processing servers) of the platform illustrated inFIG. 1 in a second state (application), according to some embodiments of the present disclosure.
• FIG. 3A illustrates a component (natural language processing servers) of the platform illustrated inFIG. 1 in a first state (training), according to some embodiments of the present disclosure.
• FIG. 3B illustrates a component (natural language processing servers) of the platform illustrated inFIG. 1 in a second state (application), according to some embodiments of the present disclosure.
• FIG. 4 illustrates a method in accordance with an embodiment of the present disclosure.
• FIG. 5 illustrates a flow diagram illustrating the operation of a platform built in accordance with an embodiment of the present disclosure.
• FIG. 6 illustrates an aspect of a graphical user interface in accordance with an embodiment of the present disclosure.
• FIG. 7 illustrates an aspect of a graphical user interface in accordance with an embodiment of the present disclosure.
• FIG. 8 illustrates an aspect of a graphical user interface in accordance with an embodiment of the present disclosure.
• FIG. 9 illustrates an aspect of a graphical user interface in accordance with an embodiment of the present disclosure.
• FIG. 10 illustrates a computer system for use in accordance with an embodiment of the present disclosure.
DETAILED DESCRIPTION
• The present disclosure is related to systems and methods for the improved analysis and generation of medical imaging reports. In particular, embodiments of the present disclosure may include receiving medical images and textual data and generating improved medical imaging reports by applying artificial intelligence and natural language processing techniques. For example, medical images may include images from medical image archives and scanning devices such as magnetic resonance imaging (MRI), computed tomography (CT), X-Rays and the like. Additionally, textual data may include radiological reports, electronic health records, referring letters, and the like.
• FIG. 1 illustrates a platform built in accordance with the disclosed systems and methods. As illustrated inFIG. 1, aplatform100 may be configured to receive medical images and textual data. For example, theplatform100 may be communicatively coupled to a medical images archive101 and/ormedical scanning device103 so as to receive medical images. Medical images archive101 may include a picture archiving and communication system (PACS) server and the like. In some embodiments, the medical images archive101 may be integrated into an information technology (IT) system within a hospital environment that is configured to collect, store, and distribute imaging studies between scanning devices, and workstations in a clinical environment. Amedical scanning device103 may include radiological equipment such as a CT, MRI, and the like. Medical images received by theplatform100 may be received using the Digital Imaging and Communications in Medicine (DICOM) format and the like.
• Further, theplatform100 may be configured to receive textual data by way of being communicatively coupled tohospital information systems105 and/or reportingsystems107. Thehospital information systems105 and/or reportingsystems107 may provide radiological reports, radiology diagnosis, electronic health records, referring letters, related metadata, information containing medical text statements, and the like. In some embodiments, the textual data may be received using DICOM, SR, Health Level 7 (HL7) protocol, JavaScript Object Notation (JSON), eXtensible Markup Language (XML) formats, using application programming interfaces and the like. In some embodiments, thehospital information system105 may be integrated into an IT system in a hospital that is configured to manage medical records including radiology reports. Such a system may be configured to distribute messages in HL7 format and the like. In some embodiments, the reportingsystems107 may include dedicated software configured to write radiology reports. The reportingsystems107 may be capable of distributing reports through HL7 protocol and/or application programming interface (API) calls.
• Medical images and data may be received by animage acquisition server109 integrated into theplatform100 and configured to receive medical images and data from the medical images archive101 and/ormedical scanning device103 as single DICOM messages. Theimage acquisition server109 may include a DICOM listener component and be configured to filter irrelevant imaging studies and/or series, and identify images associated with a certain imaging series and/or imaging study. For example, a region of interest may include a package of tens to thousands of images, and theimage acquisition server109 may receive the package of images, filter and clean the images, and determine regarding their association to a certain imaging series and/or imaging study. For example, theimage acquisition server109 may look at the metadata and description to filter out a series of images not relevant for analysis by theplatform100.
• Similarly, textual data fromhospital information systems105 and/or reportingsystems107 may be received by atext acquisition server111 also integrated into theplatform100. Thetext acquisition server111 may include a HL7 listener component and be configured to filter irrelevant messages and save reports to a file system. For example, thetext acquisition server111 may review text related to radiology reports and attach a unique identifier (e.g., accession number) to any received text.
• Medical images and textual data may then be transmitted from theimage acquisition server109 andtext acquisition server111 to the datafile system storage113. Image data processed by theimage acquisition server109 and/or textual data processed by thetext acquisition server111 may be stored in a datafile system storage113. Corresponding image data and textual data may be linked and stored in the datafile system storage113. Further, the stored data may be sorted in dedicated sub-folders in accordance with their processing status.
• Theplatform100 may include aprocessing layer115 configured to access the data (e.g., medical image data and textual data) stored in the datafile system storage113. In some embodiments, theprocessing layer115 may include afile listener117, acase manager119, animage processing unit121 and related computer vision (CV)servers125, and atext processing unit123 and related natural language processing (NLP)servers127. Thefile listener117 of theprocessing layer115 may be configured to detect new imaging studies and related textual data being stored in the datafile system storage113. Thefile listener117 may be configured to finalize the receipt of new files and manage the queue and/or order in which new files are processed.
• Thefile listener117 may work in conjunction with thecase manager119. Thecase manager119 may gather all the data (including medical images and textual data) related to a single imaging study (e.g., associated with a single unique identifier or accession number) and route it to appropriate processing units (e.g.,image processing unit121, text processing unit123). Thecase manager119 may route data based on its associated metadata. Thecase manager119 may also receive processed data from theimage processing unit121 andtext processing unit123 and compare and combine results, generate updated findings, and update adatabase129.
• Medical image data may be received by theimage processing unit121 of theprocessing layer115 ofplatform100. Theimage processing unit121 may be configured to analyze the medical image data to perform computer image processing, and utilize one or more computer vision servers configured with artificial intelligence components to automatically detect findings, areas of interest, provide measurements, and other characteristics. Detected findings may include abnormalities, suspected areas as well as other areas of clinical interest, including for example, tumors, lesions, nodules, opacities, irregular blood vessels, blood clots and the like. Theimage processing unit121 may be configured to automatically identify suspected abnormalities, their location and other characteristics.
• In some embodiments, theimage processing unit121 may apply one or more artificial intelligence based algorithms embodied on one or more computer vision (CV)servers125 to the received medical image data to generate enhanced medical image data, which may also be referred to as automatic imaging detection results. The generated enhanced medical image data may include findings detected by theimage processing unit121.
• More particularly, theimage processing unit121 may dispatch images to one or more computer vision (CV)servers125 configured to run convolutional neural networks (CNN) or other computer vision techniques. Images may be dispatched by theimage processing unit121 to the appropriate computer vision (CV)server125 based on their modality (e.g., CT, MRI) and body region (e.g., chest, brain). Prior to applying convolutional neural networks by way of theCV server125, theimage processing unit121 may preprocess the image data. Examples of preprocessing image data include normalizing the pixel information in the image data, the size of voxels, and/or the size of the data provided. Further, preprocessing image data may include segmenting different images. Theimage processing unit121 may also post process results from the convolutional neural network algorithms. In particular, post processing steps may include undoing normalizations to return images to normal sizes. Post processing may also include data cleaning, adjacent component/morphology analysis, and contour operations.
• The computer vision (CV)servers125 may include one or more separate convolutional neural networks. For example, a regression convolutional neural network may be used to estimate parameters such as the centroid coordinates of a pulmonary nodule. Regression convolutional neural networks may be used to create a region of interest for an image. A regression convolutional neural network may be configured to receive an image having a pre-defined size and output a vector of estimated parameters corresponding to a region of interest.
• Additionally, computer vision (CV)servers125 may include a semantic segmentation convolutional neural network. A semantic segmentation convolutional network may be configured to associate each pixel of an image with a class label. A semantic segmentation convolutional neural network may be used to measure lesions in organs. The semantic segmentation convolutional neural networks may utilize pyramid scene parsing and dilated convolutions. In some embodiments, the semantic segmentation convolutional network may be configured to receive an image of any size and output an image of the same size as the input with masks of segmentations.
• Additionally, the computer vision (CV)servers125 may include classification convolutional neural networks that are configured to classify images on a global and/or local level. Examples of global level classifications may include determining whether an image is a CT image, MRI image and the like. Examples of local level classifications may include a classification as to whether a region of interest includes a benign and/or cancerous lesions. In some embodiments, the classification convolutional neural networks may be configured to receive an image with a pre-defined size and output a vector with a field corresponding to each possible classification. Each value in the vector may correspond to a score for the given class. Higher values may be associated with a predicted class.
• In some embodiments, the computer vision (CV)servers125 may work in unison to provide image processing. For example, to measure the ascending and descending aortic caliber, a classification convolutional neural network may be used to classify (and determine) CT slices that contain a thoracic aorta. Further, a regression convolutional neural network may be applied to estimate the center of the thoracic aorta. And still further, a semantic convolutional neural network may be used to determine the diameter of the thoracic aorta.
• Outputs of thecomputer vision servers125 may be passed to theimage processing unit121. Theimage processing unit121 may then generate enhanced medical image data that includes information related to whether an area of interest was located, the characteristics of the area of interest, and the location of the area of interest within the received medical images.
• As discussed above, theimage processing unit121 may include anartificial intelligence component207 embodied within computer vision (CV)servers125 configured to generate enhancedmedical image data211 when provided withmedical image data209.FIG. 2A illustrates thetraining stage201 of theartificial intelligence component207 of theimage processing unit121 and related computer vision (CV)servers125 ofFIG. 1.
• As discussed above, in some embodiments, theartificial intelligence unit207 may include one or more multi-layer convolutional neural networks (e.g., computer vision servers125). Each of the multi-layer convolutional neural networks may be trained upon radiologist marked data.
• For example, training data may include global annotations related to the entire CT scan (e.g., CT scan does or does not contain liver lesions). Training data may also include global annotations related to CT scan slices (e.g., CT scan slice does or does not contain liver lesions, CT scan slice does or does not contain liver). Training data may also include locations of findings (e.g., x, y, z coordinates of centroid pulmonary nodule). Training data may also include 2D segmentation of organs and findings/areas of interest. Training data may also include 3D segmentation of organs and findings/areas of interest.
• In some embodiments, theartificial intelligence unit207 may include one or more convolutional networks configured for lesion detection and measurement. The convolutional network configured for lesion detection and measurement may be trained on annotated slices (e.g., slice does or does not consist of relevant organ). The convolutional neural network may also be trained on slices having the contour of the relevant organ annotated. Further, the convolutional neural network may be trained on slices having the contour of any relevant lesion annotated.
• In some embodiments, theartificial intelligence unit207 may include networks trained to measure thoracic aortas. In such an embodiment, the networks may be trained on annotated CT scans. Annotated CT scans may include information regarding whether a particular CT scan slice is or is not suitable for thoracic aorta measurement. For those slices suitable for thoracic aorta measurement, the annotated CT scans may also indicate the x and y coordinates of the ascending and descending aortas. Slices suitable for thoracic aorta measurement may also have the 2D contour of ascending and descending aortas annotated.
• Once theartificial intelligence unit207 of the computer vision (CV)servers125 are trained, theimage processing unit121 ofFIG. 1 may be applied or operated as illustrated inFIG. 2B. In particular,image data209 may be received by theimage processing unit121 and theartificial intelligence unit207 ofcomputer vision servers125 may apply the trained algorithm(s) to the receivedimage data209 to produce enhancedmedical image data211.
• Turning back toFIG. 1, thecase manager119 may dispatch textual data to thetext processing unit123 of theplatform100. In some embodiments, thetext processing unit123 may be configured to communicate with one or more natural language processing (NLP)servers127 configured to apply the natural language processing algorithms to the received textual data in order to generate structured text data.
• In some embodiments theNLP servers127 may include one or more recurrent neural networks. Thetext processing unit123 may preprocess the textual data. For example, a radiology report may be parsed into sentences and/or sections (e.g., title, sentences), and then each section may be routed to therelevant NLP server127. Further, thetext processing unit123 may be configured to receive the output from each of theNLP servers127 and postprocess the results from theNLP servers127 by reconstructing the radiology report with the output from theNLP servers127.
• In some embodiments, theNLP servers127 may include classification recurrent neural networks (RNNs). The classification RNNs may be configured to classify text into specific classes on a global and/or local level. Examples of global classifications include determining that a report does or does not contain mention of liver lesions. Examples of local classifications includes that a sentence does or does not contain mention of liver lesions. In some embodiments, the classification RNNs may use layered bidirectional long short-term memory (LSTM) architecture.
• In some embodiments, theNLP servers127 may include RNNs configured to provide sentiment analysis. In particular, the RNNs may be configured to classify text as having positive or negative sentiment. For example, the RNNs may indicate whether the radiologists positively or negatively reported liver lesions in the radiology report.
• In some embodiments, a series ofNLP servers127 running various algorithms may be used. For example, a radiology report may be first parsed into sentences. Sentences containing words from a pre-defined set may be identified as “suspicious sentences.” The suspicious sentences may then be passed to aNLP server127 configured to apply a bi-directional LSTM classification neural network. The bi-directional LSTM classification neural network may be configured to classify between sentences that mention a finding and sentences that do not mention a finding. The bi-directional LSTM classification neural network may be configured to output a list of sentences that mention the finding.
• The list of sentences that mention the finding may then be passed to asecond NLP server127 configured to apply a bi-directional LSTM classification neural network that is configured for sentiment analysis. This network may be configured to classify between sentences that mention the findings positively and those sentences that mention the findings negatively. This network may be configured to output a list of sentences that mention the finding positively.
• As illustrated inFIGS. 3A-3B, the natural language processing (NLP)servers127 ofFIG. 1 may include a naturallanguage processing module307 that operates in a training stage301 (seeFIG. 3A) and an operational or application stage303 (seeFIG. 3B).
• Thetext processing unit123 may be configured to receive a radiology report related to a particular identifier (e.g., accession number) from thecase manager119, and may be configured to route radiology reports torelevant NLP servers127 based on their modality (e.g., CT, MRI) and body region (e.g., chest, brain).
• During atraining stage301, the naturallanguage processing module307 may be trained in two stages. In a first stage, the naturallanguage processing module307 may create a vocabulary for each type of textual data (e.g., tweets, emails, radiology reports, metadata). In a second stage, thenatural processing module307 may be trained based on textual data including annotations. In some embodiments, the naturallanguage processing module307 may apply sentiment analysis. In some embodiments, the natural language processing algorithm may be built after analyzing radiology reports and constructing a medical vocabulary data set. In some embodiments, the naturallanguage processing module307 may include natural language processing deep learning classifiers based on a radiological vocabulary.
• For example, during thetraining stage301, the naturallanguage processing module307 may construct a dictionary of “suspicious” words or phrases. This may be a list of words or phrases that are defined by a radiologist as being used to report a specific finding. Then, in a next step, sentences that include the suspicious words or phrases may be extracted from a training set of thousands of radiology reports. The extracted sentences may be annotated based on whether they don't mention, positively mention, or negatively mention the finding. Thenatural processing module307 may then be trained based on the annotated extracted sentences.
• Further, during anoperational stage303, a trained naturallanguage processing module307 may be applied to the receivedtextual data309 in order to generate structuredtext data311. The generatedstructured text data311 may include stratified, categorized and annotated statements regarding patient procedures and conditions.
• Textual data309 received by thetext processing unit123 may include structured and unstructured text statements associated with a patient and/or imaging procedure. For example,textual data309 may include radiology reports of analyzed images, prior radiology reports, referring physician letters and the like.Textual data309 may also include metadata related to patient and/or image procedures. For example, metadata may include DICOM and/or HL7 tags that contain information regarding patient age, referring unit, procedure name, imaging protocol, modality type and other details.
• Referring back toFIG. 1, the enhanced medical image data produced by theimage processing unit121 and the structured text data produced by thetext processing unit123 may be transmitted to and received by thecase manager119. As discussed previously, thecase manager119 may then store the enhanced medical image data and structured text data in arelational database129. Therelational database129 may be configured to store and handle all data related to incoming imaging studies, radiology reports, processing results, processing status, and detected findings. In some embodiments, thedatabase129 may serve as the output of theprocessing layer115. Further, in some embodiments, thedatabase129 may serve as the input for theweb layer131.
• Further, thecase manager119 may be configured to perform a comparative analysis between the enhanced medical image data and the structured text data. In some embodiments, the comparative analysis may involve cataloguing each finding indicated in the enhanced medical image data and correlating it to information derived from the structured text data. The findings may be stored in thedatabase129.
• Theweb layer131 may include aweb server137 andnotification server139.
• Theweb server137 may be configured to manage users, reading groups, clinical and administrative requests. Further, theweb server127 may be configured to query thedatabase129 for studies and findings and push notifications to subscribers based on links between the reports and users in the system.
• For example, theweb server137 may be communicatively coupled to aclinical web client141 and anadministration web client143. Theclinical web client141 may be configured to present a worklist of unreported findings for a certain user and/or reading group. Theclinical web client141 may include a graphical user interface that allows the visualization of different findings. Theclinical web client141 may allow for the filtering and sorting of findings, presenting key images and relevant sections from radiology reports, and allowing the management of various reviewing statuses per each detected finding.
• Theadministrative web client143 may include a user interface configured for use by a system administrator for supporting a clinical team. Theadministration web client143 may allow the presentation of processing status of the system, viewing a list of received studies and their processing status, managing users in the system, changing system configuration and controlling activation of the processing layer.
• As discussed above, theweb layer131 may also include anotification server139. Thenotification server139 may be configured to manage subscriptions for notifications from workstations used by clinical users. For example, thenotification server139 may be configured to listen to the database for new studies and findings and push notifications to subscribers based on links between reports and users in the system.
• Thenotification server139 may be configured to provide notifications via aclinical notification client145. Theclinical notification client145 may be configured as a component in workstations in a clinical environment (e.g., radiology department). In some embodiments, theclinical notification client145 may provide an active user (e.g., radiologist) with on-screen real time notifications on new unreported findings. Notifications may include direct hyperlinks to viewing the finding in theclinical web client141 for quick evaluation immediately after writing the report by the radiologist.
• Using the information stored indatabase129 theplatform100 may be capable of generating automated radiology reports for each finding in the enhanced medical image data set. Further, theplatform100 may receive radiology reports generated by a radiologist. In some embodiments, theplatform100 may compare the automated radiology reports with the radiology reports generated by a radiologist in order to identify discrepancies. Discrepancies may include conditions and findings that were unreported in the radiology reports generated by a radiologist, as well as errors in the radiology reports related to findings.
• In other embodiments, a detected discrepancy may trigger the generation of an alert. The alert may then be displayed on a radiologists' computer screen. The radiologist may then view the radiologist report and one or more notations indicating discrepancies between the computer generated report and the originally determined radiologist report.
• Radiologist reports may include information regarding the type of exam including the date, time, and type of imaging study, clinical history, comparison, technique, findings, and impression. Radiologist reports may be referred to as textual data.
• In some embodiments, theplatform100 may be configured to generate a list of items for which there is a high probability of uncorrelated, unreported or misreported medical findings or conditions, noting a discrepancy between the radiologist's interpretation and the automated radiology reports. These reports may be published by the components of theweb layer131 and provided to the appropriate client:clinical web client141,administration web client143, and/or theclinical notification client145.
• Theplatform100 may be configured to generate a clinical graphical user interface configured for display to a clinician (e.g.,clinical web client141, clinical notification client145). The platform may also be configured to generate a non-clinical graphical user interface (e.g., administration web client143).
• The clinical graphical user interface may be provided to a physician, radiologist, hospital administration, nursing staff, and the like. The clinical interface may provide a user with a list of discrepancies identified between the automated radiology report and the radiology report generated by a physician. In some embodiments, the clinical graphical user interface may be configured to display a copy of the medical image with areas of interest highlighted, along with the original radiology report generated by the physician supplemented by the automated radiology report generated by the systems discussed herein. Additionally, in some embodiments, agreement/disagreement between the automated radiology report and the physician generated original radiologist report may be displayed. For example, in some embodiments, agreement between the radiology report and the automatic image analysis may be presented in green to a user. Additionally, negative reporting of features of interest (e.g., pulmonary nodules) may be presented in red. For example, the system will present the user with a list of items and areas of interest where there is a probability of uncorrelated, unreported or misreported medical findings or conditions, noting a discrepancy between the human physician interpretation and the automatic image detection.
• The non-clinical graphical user interface may be provided by a non-clinical interface to a hospital administrator and the like. The non-clinical graphical user interface may show common trends, stratification and the like. In particular, the non-clinical graphical user interface may allow for the grouping of medical images by condition, findings, protocols, demographic information, discovered conditions and the like.
• FIG. 4 illustrates a method in accordance with embodiments of the present disclosure. The method may include the steps of receiving medical images andtextual data401, generating enhanced medical image data by applying an artificial intelligence module to the receivedmedical images403, generating structured text data by applying a natural language processing module to the receivedtextual data405, and generating improved medical image reports based on the generated enhanced medical image data and the generatedstructured text data407.
• Further, in some embodiments, generating improved medical image reports based on the generated enhanced medical image data and the generated structured text data may include identifying and storing each finding indicated in the enhanced medical image data, correlating each finding to its corresponding structured text data, identifying corresponding portions for each finding in a radiology report, identifying discrepancies between the corresponding portions and the corresponding structured text data for each finding, and generating improved medical image reports by augmenting the radiology report with the identified discrepancies.
• As discussed, the disclosed methods may include the processing of textual data using various text analysis and language processing algorithms to create a stratified, categorized and annotated list of statements regarding the patient and procedure. Further, a comparative analysis may be performed between the medical text statements derived by the platform and the detections in the radiology report. In some embodiments, the systems and methods may present a user with a list of items and areas of interest where there is a probability of uncorrelated, unreported or misreported medical findings or conditions, noting a discrepancy between the human physician interpretation and the automatic image detection. Thus, the disclosed systems and methods may be able to catch discrepancies and possible missed details that could reduce diagnostic errors, misreporting and misdiagnosis rates. The system can also detect findings which were not requested as part of the referring clinical question (referred to as Incidental findings or Incidentalomas).
• FIG. 5 provides a flowchart illustrating the operation of a platform built in accordance with the embodiments discussed herein in a clinical setting. As illustrated, in some embodiments, the workflow may start501 when a patient undergoes amedical imaging exam503. Examples ofmedical imaging exams503 include CT scans, X-rays, MRI, and the like. In anext step505, a radiologist or trained physician may interpret the medical imaging exam and create a textual diagnostic report. After the images are interpreted and a textual diagnostic report is created, the images and reports may be archived in adatabase507 and the report may be communicated to a referring entity orphysician509. In accordance with the disclosed embodiments, the platform may also be provided with a copy of the radiologists' textual diagnostic report and be configured to automatically detect findings in theimages511. The platform may be further configured to automatically detect any discrepancies between the automatically detected findings and theradiology report513. In the event adiscrepancy515 is not found, the process may come to anend519. However, if a discrepancy is found, the platform may provide the radiologist with a summary of the discrepancy. Accordingly, a radiologist may then correct or edit the originally provided radiology report.
• In the illustrated operation, image processing and detection of findings may be initiated as soon as a medical image is obtained. Further, the radiology report may be automatically scanned using the natural language processing elements discussed above.
• FIG. 6 illustrates a graphical user interface in accordance with an embodiment of the present disclosure configured to be displayed to a physician and/or radiologist. In particular, the radiology report generated by a radiologist may be displayed in afirst panel601 of a graphical user interface. Asecond panel603 may be configured to display the structured text automatically generated by the platform described herein. Further, athird panel605 may be configured to display information regarding the size and/or location of the findings. Accordingly, as illustrated inFIG. 6, the platform's automatically generated structured text may be incorporated into a graphical user interface that is configured for radiology notes and/or reporting on radiological images.
• FIGS. 7 and 8 illustrate aspects of a graphical user interface in accordance with an embodiment of the present disclosure. As illustrated inFIG. 7, the graphical user interface may display a medical image (e.g., slice/s from chest CT scan) in afirst panel701. As illustrated, the medical image may be augmented with highlighting and other markers that indicate regions of interest as determined by the image processing unit. Asecond panel703 may display an augmented report. In particular, additional information generated by the structured text and platform may be integrated into thereport705. Further, discrepancies between the structured text and platform may also be integrated into thereport707. In some embodiments,additional information705 anddiscrepancies707 may be color-coded when displayed. Similarly, positive and negative findings may be color-coded when displayed.
• Similarly,FIG. 8 illustrates a medical image (e.g., slice/s from abdomen CT scan) in afirst panel801. Asecond panel803 may display an augmented report. In particular, additional information and/or positive/negative findings generated by the structured text and platform may be integrated into thereport805. Further, discrepancies between the structured text and platform may also be integrated into thereport807.
• FIG. 9 illustrates an aspect of a graphical user interface in accordance with an embodiment of the present disclosure. As discussed above, in some embodiments, a platform such as platform100 (and more particularly, the result processing unit123) may be configured to generate a list of items for which there is a high probability of uncorrelated, unreported or misreported medical findings or conditions, noting a discrepancy between the radiologist's interpretation and the automated radiology reports.FIG. 9 illustrates such a list.
• As illustrated inFIG. 9, in some embodiments aggregated information from several or all analyzed cases and procedures may be displayed. Accordingly, the graphical user interface may be used to analyze common trends, and allow for the stratification and categorization of information into groups and graphs summarizing the information. For example, similar conditions, similar findings, cases categorized by protocols, categorized by demographic information, categorized by discovered conditions, and the like.
• In an alternative embodiment, the disclosed platform may be used to automatically check the quality and accuracy of every radiological exam and interpretation occurring in a hospital or similar institution.
• As discussed above, in some embodiments, the disclosed systems and methods may be integrated into a picture archiving and communication system (PACS) server. A PACS server may be configured for the digital storage, transmission, and retrieval of radiology images. PACS servers may include both software and hardware components and may be configured to directly interface with and/or acquire digital image with imaging modalities. Medical images may be stored and transferred in the PACS system using the standardized, universal format DICOM (Digital Imaging and Communications in Medicine). Further, the PACS server may be configured to transfer images to a workstation for viewing and reporting. Accordingly, a PACS viewer may be communicatively coupled to the PACS server, and include a software that is installed on a corresponding workstation in order to receive and display radiology images. Radiology images may be archived into storage or a database for retrieval at a later time. The storage or database may be communicatively coupled to the PACS server and the PACS system may be configured to manage the storage of these radiology images. In some medical environments, PACS may provide the electronic platform for radiology images interfacing with other medical automation systems such as Hospital Information System (HIS), Electronic Medical Record (EMR), Practice Management Software, and Radiology Information System (RIS). Accordingly, the disclosed systems and methods may be used to improve the functionality of PACS system.
• In some embodiments, the described systems and methods may be applied to medical images of cancer and/or cardiovascular indications. Additional examples of applications of the platform include thoracic aortic dilatation detection, pulmonary nodule detection, abdominal aortic dilatation detection, liver lesion detection and the like. Further examples of illnesses and disease that may be early detected with the disclosed systems and methods include (but are not limited to) liver lesions, pancreatic lesions, renal lesions, adrenal lesions, pulmonary nodules, mediastinal lesions, lymph nodes, thoracic aortic dilatation, abdominal aortic dilatation, pulmonary artery dilatation, pulmonary embolism and the like.
• In some embodiments, a computer vision deep-learning algorithm may be used to analyze medical images. Further in some embodiments, natural language processing deep-language algorithms may be used to analyze textual radiology reports, which may correspond to the medical images. The application of these techniques may be useful in the early detection of disease states including cancer.
• The application of the techniques discussed herein may allow for the detection of formerly missing actionable findings, alert a physician and help physicians provide a more comprehensive diagnosis for the patient.
• As opposed to conventional systems that may rely upon computer vision, the disclosed systems may also utilize natural language processing on related patient notes and the electronic health record.
• In some embodiments, the disclosed systems and methods may be integrated into the backend server, thereby the disclosed systems and methods may not require the adoption of independent server systems. In contrast, conventional systems rely on doctor initiation and/or a specific protocol.
• Further, while in conventional systems, most of the findings reported are false positives or not significant, the disclosed systems are able to utilize information regarding the clinical context and patient history (by way of the electronic health record) and report only on significant findings. Accordingly, user fatigue is reduced, and the provided systems increase the efficiency of providing radiologist reports on medical imagery.
• The disclosed systems and methods provide benefits to patients, radiologists, and/or hospital systems. For example, benefits provided to patients may include higher detection rates on broader areas of pathologies, increased number of patients that go to needed follow up care (screening, consultation), the promotion of early, efficient and cheaper care, especially when detecting small cancer onsets, better outcomes (e.g., morbidity/mortality) due to early detection, detection of pre-cancerous/cardiovascular indications that are often missed and asymptomatic (e.g., lesions, masses, tumors, nodules, ectasia, aneurysm, etc.).
• Radiologists and the hospital system may also be provided with additional benefits. For example, benefits provided to radiologists and the hospital system may include increased detection rates, quality and coverage of the interpretation, and decreased chances of diagnostic error. Further, the disclosed systems and methods can be integrated into existing computing systems, accordingly, the physician workflow is not impacted.
• As used herein, the term “module” may be understood to refer to computer executable software, firmware, hardware, or various combinations thereof. It is noted that the modules are exemplary. The modules may be combined, integrated, separated, or duplicated to support various applications. Also, a function described herein as being performed at a particular module may be performed at one or more other modules and by one or more other devices instead of or in addition to the function performed at the particular module. Further, the modules may be implemented across multiple devices or other components local or remote to one another. Additionally, the modules may be moved from one device and added to another device, or may be included in both devices.
• FIG. 10 illustrates a computer system for use in accordance with an embodiment of the present disclosure. In particular,FIG. 10 is a system diagram for a computing device used in a system built in accordance with an embodiment of the present disclosure.FIG. 10 illustrates a functional block diagram of a machine in the example form ofcomputer system1000, within which a set of instructions for causing the machine to perform any one or more of the methodologies, processes or functions discussed herein may be executed. In some examples, the machine may be connected (e.g., networked) to other machines as described above. The machine may operate in the capacity of a server or a client machine in a client-server network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine may be any special-purpose machine capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken by that machine for performing the functions describe herein. Further, while only a single machine is illustrated, the term “machine” shall also be taken to include any collection of machines that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methodologies discussed herein. In some examples, theplatform100 and server systems ofFIG. 1 may be implemented by the example machine shown inFIG. 10 (or a combination of two or more of such machines).
• Example computer system1000 may includeprocessing device1003,memory1007,data storage device1009 and communication interface1015, which may communicate with each other via data andcontrol bus1001. In some examples,computer system1000 may also includedisplay device1013 and/or user interface1011.
• Processing device1003 may include, without being limited to, a microprocessor, a central processing unit, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP) and/or a network processor.Processing device1003 may be configured to executeprocessing logic1005 for performing the operations described herein. In general,processing device1003 may include any suitable special-purpose processing device specially programmed withprocessing logic1005 to perform the operations described herein.
• Memory1007 may include, for example, without being limited to, at least one of a read-only memory (ROM), a random access memory (RAM), a flash memory, a dynamic RAM (DRAM) and a static RAM (SRAM), storing computer-readable instructions1017 executable byprocessing device1003. In general,memory1007 may include any suitable non-transitory computer readable storage medium storing computer-readable instructions1017 executable byprocessing device1003 for performing the operations described herein. Although onememory device1007 is illustrated inFIG. 10, in some examples,computer system1000 may include two or more memory devices (e.g., dynamic memory and static memory).
• Computer system1000 may include communication interface device1011, for direct communication with other computers (including wired and/or wireless communication), and/or for communication with network. In some examples,computer system1000 may include display device1013 (e.g., a liquid crystal display (LCD), a touch sensitive display, etc.). In some examples,computer system1000 may include user interface1011 (e.g., an alphanumeric input device, a cursor control device, etc.).
• In some examples,computer system1000 may includedata storage device1009 storing instructions (e.g., software) for performing any one or more of the functions described herein.Data storage device1009 may include any suitable non-transitory computer-readable storage medium, including, without being limited to, solid-state memories, optical media and magnetic media.
• Although the present disclosure may provide a sequence of steps, it is understood that in some embodiments, additional steps may be added, described steps may be omitted, and the like. Additionally, the described sequence of steps may be performed in any suitable order.
• While illustrative embodiments have been described herein, the scope thereof includes any and all embodiments having equivalent elements, modifications, omissions, combinations (e.g., of aspects across various embodiments), adaptations and/or alterations as would be appreciated by those in the art based on the present disclosure. For example, the number and orientation of components shown in the exemplary systems may be modified.
• Thus, the foregoing description has been presented for purposes of illustration. It is not exhaustive and is not limiting to the precise forms or embodiments disclosed. Modifications and adaptations will be apparent to those skilled in the art from consideration of the specification and practice of the disclosed embodiments.

Claims (20)

1. A system for providing computer-aided detection of clinical interest areas captured in medical images, the system comprising:

a server system configured to receive the medical images and textual data, wherein the server system comprises:

an image processing unit configured to apply an artificial intelligence computer vision techniques to the received medical images to generate enhanced medical image data;

a text processing unit configured to apply natural language processing to the received textual data to generate structured text data; and

a platform configured to detect clinical interest areas by detecting one or more discrepancies between the generated enhanced medical image data, the generated structured text data, and the received medical image and textual data.

2. The system ofclaim 1, wherein the platform is further configured to generate an alert responsive to the detected discrepancies.

3. The system ofclaim 1, wherein the artificial intelligence computer vision techniques comprises a convolutional neural network.

4. The system ofclaim 1, wherein the server system comprises a processing module configured to dispatch data and receive results from the image processing unit and the text processing unit.

5. The system ofclaim 1, wherein the platform is further configured to generate at least one of a clinical or non-clinical interface including the enhanced medical image data.

6. The system ofclaim 1, wherein the platform is further configured to incorporate the enhanced medical image data into an findings record.

7. The system ofclaim 1, wherein the natural language processing comprises at least one recurrent neural network.

8. A method for generating improved medical image reports comprising:

receiving medical images and textual data;

generating enhanced medical image data by applying an artificial intelligence module to the received medical images;

generating structured text data by applying a natural language processing module to the received textual data;

determining discrepancies between a received medical image report and the generated enhanced medical image data and the generated structured text data; and

generating an improved medical image report including at least one of the generated enhanced medical image data, generated structured text data, and determined discrepancies.

9. The method ofclaim 8, further comprising:

providing an alert to at least one of a user or a creator of the medical image report responsive to determining a discrepancy.

10. The method ofclaim 8, wherein applying an artificial intelligence module comprises applying a convolutional neural network.

11. The method ofclaim 10, comprising:

training at least a regression convolutional neural network, a semantic segmentation convolutional network and a classification convolutional neural network.

12. The method ofclaim 8, wherein applying a natural language processing module to the received textual data further comprises applying a recurrent neural network.

13. The method ofclaim 8, comprising:

generating at least one of a clinical or non-clinical interface including the generated improved medical image report.

14. The method ofclaim 8, comprising:

incorporating the generated improved medical image report into a findings record.

15. The method ofclaim 8, wherein determining discrepancies between a received medical image report and the generated enhanced medical image data and the generated structured text data comprises:

identifying and storing each finding indicated in the enhanced medical image data;

correlating each finding to its corresponding structured text data;

identifying corresponding portions for each finding in a radiology report;

identifying discrepancies between the corresponding portions and the corresponding structured text data for each finding; and

augmenting the radiology report with the identified discrepancies.

16. The method ofclaim 15, further comprising:

presenting the generated improved medical image report to a physician for approval.

17. A non-transitory computer-readable medium storing instructions that, when executed on one or more processors, cause the one or more processors to:

receive medical images;

receive textual data;

generate enhanced medical image data by applying an artificial intelligence module to the received medical images;

generate structured text data by applying a natural language processing module to the received textual data; and

generate an improved medical image report including at least one of the generated enhanced medical image data, generated structured text data, and determined discrepancies.

18. The non-transitory computer-readable medium ofclaim 17, wherein generating the enhanced medical image data comprises applying a convolutional neural network.

19. The non-transitory computer-readable medium ofclaim 17, wherein applying a natural language processing module to the received textual data further comprises applying at least one recurrent neural network.

20. The non-transitory computer-readable medium ofclaim 17, wherein generating an improved medical image report comprises the one or more processors being configured to:

identify and store each finding indicated in the enhanced medical image data;

correlate each finding to its corresponding structured text data;

identify corresponding portions for each finding in a radiology report;

identify discrepancies between the corresponding portions and the corresponding structured text data for each finding; and

augmenting the radiology report with the identified discrepancies.

Images