Abstract

The 2014 i2b2 natural language processing shared task focused onidentifying cardiovascular risk factors such as high blood pressure, highcholesterol levels, obesity and smoking status among other factors found inhealth records of diabetic patients. In addition, the task involved detectingmedications, and time information associated with the extracted data. This paperpresents the development and evaluation of a natural language processing (NLP)application conceived for this i2b2 shared task. For increased efficiency, theapplication main components were adapted from two existing NLP tools implementedin the Apache UIMA framework: Textractor (for dictionary-based lookup) andcTAKES (for preprocessing and smoking status detection). The applicationachieved a final (micro-averaged) F₁-measure of 87.5% on thefinal evaluation test set. Our attempt was mostly based on existing toolsadapted with minimal changes and allowed for satisfying performance with limiteddevelopment efforts.

Graphical abstract

1. Introduction

The 2014 i2b2 (Informatics for Integrating Biology and the Bedside) challengeproposed several different tasks: clinical text de-identification, cardiovascularrisk factors identification, software usability assessment, and novel data uses. Ourefforts focused on the second track, identifying risk factors for heart diseasebased on the automated analysis of narrative clinical records of diabetic patients[1]. The annotationguidelines for the task defined eight categories of information associated withincreased risk for heart disease: 1) Diabetes, 2) Coronary Artery Disease (CAD), 3)Hyperlipidemia, 4) Hypertension, 5) Obesity, 6) Family history of CAD, 7) Smokingand 8) Medications associated with the aforementioned chronic diseases. Eachcategory of information (except family history of CAD and smoking status) had to bedescribed withindicator andtime attributes. Theindicator attribute captures indications of the risk factor in the clinical text.For instance, Diabetes could be identified using a mention of the disease (i.e.“patient has h/o DMII”), or a hemoglobin A1c value above 6.5 mg/dL(i.e. “7/18: A1c: 7.3”) while CAD could be identified using amention (i.e. “PMH: significant for CAD”), or an event (i.e.“CABG in 1999”). The time attribute specifies the temporal relationto the Document Creation Time (DCT). It could take any one of the following values:before DCT, during DCT or after DCT. We refer the reader to [2] for a complete description of the annotationguidelines. For this challenge, we built a natural language processing (NLP)application based on the Apache UIMA (Unstructured Information ManagementArchitecture) [3] and reusingexisting tools previously developed to address similar tasks in previous i2b2challenges. In this paper, we present our approach to extract relevant informationfrom clinical notes, discuss performance results, and conclude with remarks aboutour experience adapting existing NLP tools.

2. Background

Extracting information from clinical notes has been the focus of a growingbody of research these past years [4]. Common characteristics of narrative text used by physicians inelectronic health records (e.g., telegraphic style, ambiguous abbreviations) make itdifficult to access such information automatically. Natural Language Processing(NLP) techniques are needed to convert information from the unstructured text to astructured form readily processable by computers [5,6]. Thisstructured information can then be used to extract meaning and enable ClinicalDecision Support (CDS) systems that assist healthcare professionals and improvehealth outcomes [7]. Among theearliest attempts to develop NLP applications in the medical domain, the LSP(Linguistic String Project) [8], and MedLEE (Medical Language Extraction and Encoding system)[9] were prominentexamples. More recent applications include MetaMap [10] developed by the National Library of Medicineto map terms in biomedical text with concepts in the UMLS (Unified Medical LanguageSystem) Metathesaurus [11].cTAKES [12] was developed atthe Mayo Clinic and is described as “large-scale, comprehensive, modular,extensible, robust, open-source” application based on Apache UIMA. It can beused to preprocess clinical text, find named entities and perform additionaladvanced NLP tasks such as coreference resolution. Textractor [13] is another UIMA-based applicationthat was originally developed at the University of Utah to extract medications,their attributes, and reasons for their prescription from clinical notes.

When extracting information from clinical notes, NLP applications must takelocal contextual and temporal information into account for improved accuracy.Contextual information is important to determine if concepts are affirmed or negated(e.g., ‘denies any chest pain’), or if the subject of theinformation is the patient or someone else (e.g., ‘mother hasdiabetes’). Popular algorithms for negation detection in clinical notesinclude NegExpander [14] andNegEx [15]. Temporalinformation is critical to establish chronological order of events described inpatient notes and to resolve mentions of procedures or laboratory results tospecific time points for accurate analysis [16,17]. The ConTextalgorithm [18] proposed byChapman et. al. is an extension of NegEx that allows analysis of contextualinformation like negation (negated, affirmed), temporality (historical, recent,hypothetical), and experiencer (patient, other). The development of NLP applicationstypically requires significant efforts and relies on annotated clinical text fortraining and testing. Widely accessible and shared annotated corpora in the medicaldomain are still rare, mainly because of strict patient privacy rules. This scarcityhas been an obstacle to developing state-of-the-art NLP approaches for clinical text[19]. To address thisobstacle and enable direct comparison of NLP approaches in the clinical domain, i2b2shared NLP tasks have been organized almost annually since 2006. The challengesstarted with an automated de-identification [20] and smoking status detection [21] challenges. In 2008, the i2b2 challengefocused on identifying information about obesity and 15 co-morbidities[22]. In 2009, the thirdi2b2 challenge [23] wasfocused on identifying medications and associated information such as dosage andfrequency. This was followed by challenges for medical concept extraction, assertionand relations classification in 2010 [24], followed by coreference resolution tasks in 2011[25] and a temporalrelations classification in 2012 [26].

To reduce development efforts, many authors have reused NLP tools orresources such as ConText, sentence boundary detectors and part-of-speech taggersfrom OpenNLP project [27],the Stanford parser [28], orthe Weka machine learning framework [29], but the majority of their applications were still newdevelopments. Reusing larger components or even existing NLP applications couldallow for further development effort reduction. A good example was the applicationdeveloped by Wellner et al. [30] for the 2006 i2b2 de-identification task. It was based on theadaptation of two applications originally designed for recognizing named entities innewswire text. The process involved running two applications out-of-the-box as abaseline and then gradually introducing a few task-specific features, using biasparameters to control feature weights, and adding lists of common English wordsduring development to improve performance. With minimal effort, they were able toobtain very high performance for the task. Although their attempt used applicationsout-of-the-box as baselines, they had to re-train the models with new task-specificfeatures to achieve high performance. Our attempt focused on adapting existing toolsthat were developed to solve similar tasks in the past, and do it without featureengineering and re-training of machine learning models.

3. Methods

4. Results

After development and refinement based on the training corpus (SET1 andSET2), the NLP application processed the testing corpus when made available, and theapplication output was sent to the shared task organizers for analysis. Theapplication output was compared with the reference standard using the evaluationscript provided by the shared task organizers and all extracted informationclassified as true positive (i.e., output matches with the reference standard),false positive, or false negative. Metrics used included recall, precision, and theF₁-measure (details in [1]). The results for each class of information are presented inTable 2. For overall averages, bothmacro- and micro-averages are included. Each separate class-indicator combination isreported using micro-averages only. The evaluation script contained an option tocalculate results separately for each class of information using the –filteroption. It also allowed computing specific class and indicator attribute values suchas the class DIABETES and indicator attribute value ofmentionusing the option –conjunctive. Results for each disease category arepresented formention and each disease-specific indicatorsseparately as in the annotation guideline. The SMOKING category results arepresented asstatus only, and MEDICATION results are aggregated forall the categories correctly identified in the clinical records. All results in thetable were computed for all three values of time attribute for each class and noattempt made to separate ‘before DCT’, ‘during DCT’and ‘after DCT’ results for each class.

As shown inTable 2, the applicationachieved an overall micro-averaged F₁-measure of 87.47% and amacro-averaged F₁-measure of 86.99%. In most disease categories,accuracy was highest for mentions of disease with micro-averaged F1-measures of92.22%, 94.94%, 96.96%, 90.11%, and 99.04%for CAD, family history of CAD, Diabetes, Hyperlipidemia, and Hypertension,respectively. Medications, mentions of Obesity and Smoking status identificationaccuracy reached micro-averaged F₁-measures of 85.85%,86.12% and 86.55%, respectively. Accuracy was lower with otherinformation categories such as laboratory tests, CAD events and symptoms withF₁-measures ranging from 20.56% to %80.

5. Discussion

As presented above, the application accuracy for mentions of the variousdiseases, smoking status, medications and family history was higher than accuracyfor any other indicator type defined in the annotation guidelines (e.g., laboratorytests, CAD events and symptoms). The dictionary lookup approach with terminologicalcontent from the UMLS Metathesaurus for detecting disease mentions was successfulfor this task. Similarly, the smoking status classifier from cTAKES successfullyidentified and classified smoking status information (F₁-measure of about87%) despite the fact that the model was used out-of-the-box, without anytraining on the new corpus for the current i2b2 NLP task. The identification ofmedications and their attributes reached an F₁-measure of about86% when using regular expressions and manually curated lists of terms,demonstrating the feasibility of this approach for the type of narrative notes usedin this shared task. The precision obtained for medications was lower (83%)than recall (89%) and hence affected the final F₁-measure. Thisis mainly due to the way we chose to generate the time attribute by using thecontinuing times scenario (i.e., generating ‘before DCT’,‘during DCT’ and ‘after DCT’ temporal informationtags for every medication detected in the notes). Obviously, there will be falsepositives associated with this approach when medications strictly occur for eitherone or two of the time values in the clinical notes. In addition, since themedication term lists were created manually, some spelling variations and termscould have been missed, therefore producing some false negatives and affectingoverall recall. An example of spelling variation is the term‘nitroglycerine’ in thenitrate group category,which appeared in both corpora as ‘nitroglycerin’. The latter wasnot in the nitrate list used by our application and hence caused some falsenegatives. An example of completely missed terms was sublingual nitroglycerinmentioned as ‘SL NTG’. Among disease mentions, the Hyperlipidemiaclass had the lowest recall (83%) and Obesity had the lowest precision(76%). The former was mostly due to some clinical reports containingannotations for Hyperlipidemia mentions appearing as ‘elevated serumcholesterol’, ‘elevated lipids’ and ‘highcholesterol’ cholesterol? that were missed by our application because ofinaccurate chunking. In addition, we did not have the corresponding CUI codes forsome of them in our dictionary lookup module. There were at least two cases in thetesting corpus where Hyperlipidemia was mentioned directly following a word with nospace in between such as ‘hemodialysis Hyperlipidemia’ which ourapplication missed also. The low precision with Obesity was caused by including theUMLS concept ‘overweight’ in our list of CUIs for Obesity. Although‘overweight’ was used as indicator for obesity in one record in thereference standard corpora, its use produced many false positives since‘overweight’ often does not indicate obesity. There were also falsepositive mentions of Obesity produced by our application in cases where‘obese’ was mentioned without indicating Obesity (e.g.,“abdomen is slightly obese” and “Abdomen: Moderatelyobese”). The other indicators for diseases and risk factors were quitechallenging and our approach using regular expressions at the lexical level was notalways effective. With the exception of hemoglobin A1c laboratory tests (forDiabetes), BMI (for Obesity), and cholesterol LDL (for Hyperlipidemia), theapplication performance was modest with an F₁-measure ranging from21% for the blood glucose indicator up to 65% for the blood pressureindicator. Some of the challenges with these indicators are summarized below,

• Lexical and spelling variations: Some laboratoryindicators for diseases are mentioned with many lexical variations andacronyms.Table 1 shows theregular expressions used to capture blood glucose for diabetes and bloodpressure for hypertension. As shown, glucose can be described with avariety of terms like BG, BS, FS and FG; and blood pressure can bedescribed with terms like BP and b/p. This is an example of some of thelimitations with our approach. and a comprehensive strategy to deal withthis issue to enable better accuracy would be needed.

• Extracting laboratory numerical results accurately:When the application finds matching terms for laboratory or testindicators, it must proceed with extracting associated numerical valuesand compare them to threshold levels for abnormality. Extractingnumerical values may be straightforward when they immediately follow theterm and are expressed as single units such as in the phrase“FSBG was 353”. However, other phrases can be morechallenging like “FG 120–199; now 68–172,although 172 = outlier, mostly in the 70–130”.In this case, ranges of values are expressed with‘–’, and multiple units are expressed withtemporal and frequency modifiers (i.e. ‘now’ and‘mostly’).

• Training data sparseness: The number of trainingexamples available was sometimes too low to allow for the variety neededfor adequate application generalization. For instance, in the case ofcholesterol indicator for Hyperlipidemia, the total number of availableannotations was only 9 in the whole set of 790 training documents. Incontrast, there were about 33 annotations available for the LDLindicator.

• Complex time analysis. Test and laboratoryindicators require more sophisticated time attribute analysis and thisis another limitation of our approach. Unlike chronic disease mentionannotations which were mostly characterized with‘continuing’ time attribute (i.e. before, during andafter DCT), most of the laboratory and vital sign annotations werecharacterized by a variety of time attribute values. For instance,hemoglobin A1c and glucose tests were usually conducted in a prior visitand hence mostly annotated with ’before DCT’ while bloodpressure (BP) was mostly measured during the patient visit and hence hadmostly ’during DCT’ time value. To examine the impact oftime attributes on performance of our application, we followed the“fixed” evaluation procedure described in [32] and produced resultsfor some indicators after replacing the value of time attribute with‘before DCT’ in all annotations from our applicationoutput and in the testing reference standard (seeTable 3). This evaluation considers truepositives, false positives and false negatives for each individualannotation while ignoring the time attribute (i.e. application output isnot penalized for incorrect time values). As shown intable 3, the performance of our applicationimproved when the time component was ignored in the evaluation (comparewith results fromTable 2). Ourdecision to use the most common time attribute values for each of theseindicators caused a loss in precision and recall contributing to loweroverall F₁-measure score.

6. Conclusion

Our rapid approach, adapting resources from existing applications for the2014 i2b2 challenge, allowed for performance similar to other more sophisticatedapplication developed for this task which used additional manual annotations ormultiple machine learning classifiers [1]. We think that existing NLP resources should be reused, andmost can be adapted and used at least as baseline for future tasks in the clinicaldomain. Improvements for future attempts shall focus on a comprehensive strategy totackle spelling errors and variations, acronyms disambiguation, and more refinedtemporal analysis. Use of standard terminologies, as available in the UMLSMetathesaurus, should be the basis for these clinical information extraction tasksas they already contain well-defined concepts associated with multiple terms.Finally, regular expressions and pattern matching can be useful for extractinginformation such as name-value pairs from short phrases (e.g. ‘Cholesterol-LDL 08/26/2091 148’). However, longer phrases containing complex syntacticstructures require the use of advanced parsing techniques to identify constituentsand relations between them. In the future, we plan to explore advanced techniquessuch as dependency parsing or semantic role labeling to reduce errors appearing withlong phrases requiring deeper contextual analysis to be accurately extracted. Forinstance, in the following sentence: “Prior to her bypass surgery on theright leg, she underwent a Persantine MIBI which showed only 1 mm ST depressions andwas considered not diagnostic”; it is important for an application to linkthe negated phrase “was considered not diagotstic” with the nounphrase “Persantine MIBI” to conclude that although the patient hadthe MIBI test performed, the result was not diagnostic and therefore the testindicator (i.e. ‘MIBI’) ruled out CAD.

Footnotes

Contributor Information

Abdulrahman Khalifa, Email: abdulrahman.aal@utah.edu.

Stéphane Meystre, Email: stephane.meystre@utah.edu.

References