Background

Many computerized provider order entry (CPOE) systems include the ability to create electronic order sets: collections of clinically-related orders grouped by purpose. Order sets promise to make CPOE systems more efficient, improve care quality and increase adherence to evidence-based guidelines. However, the development and implementation of order sets can be expensive and time-consuming and limited literature exists about their utilization.

Methods

Based on analysis of order set usage logs from a diverse purposive sample of seven sites with commercially- and internally-developed inpatient CPOE systems, we developed an original order set classification system. Order sets were categorized across seven non-mutually exclusive axes: admission/discharge/transfer (ADT), perioperative, condition-specific, task-specific, service-specific, convenience, and personal. In addition, 731 unique subtypes were identified within five axes: four in ADT (S=4), three in perioperative, 144 in condition-specific, 513 in task-specific, and 67 in service-specific.

Results

Order sets (n=1,914) were used a total of 676,142 times at the participating sites during a one-year period. ADT and perioperative order sets accounted for 27.6% and 24.2% of usage respectively. Peripartum/labor, chest pain/Acute Coronary Syndrome/Myocardial Infarction and diabetes order sets accounted for 51.6% of condition-specific usage. Insulin, angiography/angioplasty and arthroplasty order sets accounted for 19.4% of task-specific usage. Emergency/trauma, Obstetrics/Gynecology/Labor Delivery and anesthesia accounted for 32.4% of service-specific usage. Overall, the top 20% of order sets accounted for 90.1% of all usage. Additional salient patterns are identified and described.

Conclusion

We observed recurrent patterns in order set usage across multiple sites as well as meaningful variations between sites. Vendors and institutional developers should identify high-value order set types through concrete data analysis in order to optimize the resources devoted to development and implementation.

Keywords: order sets, electronic health records, clinical decision support, computerized physician order entry system

Sample

We selected a diverse purposive sample of ten clinical sites with computerized order sets and requested information on each site’s order set usage in the inpatient setting for a period of one year. This sample was designed to include a geographically diverse mix of small and large, community and academic medical centers with a range of CPOE systems (both self-developed and commercially-developed systems with a mix of vendors), case heterogeneity and patient volume (measured by case-mix index, which represents the average diagnosis related group relative weight for a hospital). Seven sites agreed to participate in the project, two did not have usage data available and one declined to participate due to time constraints. The final sample (n=7) is shown inTable 1 and site characteristics are presented in the results section. Data on staffing, case mix index, discharges, and patient days were based on information provided by American Hospital Directory (28).

Dataset

Order set usage data (for 1,914 total order sets) was obtained from the seven sites. Use of an order set was defined as opening and submitting the order set. It did not matter whether the user activated a single item from an order set or every order in a single set – each counted as one use of an order set. Participating sites were asked to provide anonymized logs of inpatient system-wide order set usage for a full year (including time, date and order set name for each instance of use). Information on unused order sets (order sets with zero uses during the study period) was available at only four sites and thus “zero-use” order sets were excluded from analysis. One site was able to provide only six months of data due to the system’s data storage capabilities. Given that order set usage for this site was largely consistent across the two quarters provided, we doubled the order set use counts in the analysis phase to compensate for the shorter data collection period. All other sites provided a full year of order set data. Start and end dates varied across sites.

Classification of Order Sets

Due to granularity and naming mismatches across sites, it was not possible to conduct a cross-site comparison based on order set name alone. In addition, given the wide variation in order set content and large data set, we believed it would be extremely challenging – and not necessarily fruitful – to analyze order set content directly across sites. Thus, we developed an order set classification scheme in order to 1) provide a method of comparing order set usage across sites with varying naming conventions and order set granularity and 2) create unifying basic categories for a large database of order sets with potentially variable content.

Following qualitative assessment of the collected order set usage data, the following five categories were developed based on order set type as determined by the name of each order set (and, when needed, review of the order set content in a small number of cases):

• ADT (admissions/discharge/transfer): Groups of orders related to admission to any hospital service (including general admissions orders), discharge from the hospital or transfer internally or externally. These types of order sets are further divided into subcategories: admission, discharge or transfer.

• Perioperative (pre-operative/post-operative/unspecified): Collections of orders related to preparation for surgery or care following surgery (not necessarily specific to a certain procedure) and any surgery-related order sets with a purpose not otherwise specified. These types of order sets are divided into subcategories: pre-operative, post-operative or unspecified (not-otherwise-specified based on order set name).

• Condition-specific: Order sets pertaining to a specific diagnosis (e.g. myocardial infarction) or symptom (e.g. abdominal pain). For each condition-specific order set, the related condition was recorded.

• Task-oriented: Order sets related to a specific diagnostic (e.g. chest X-ray) or therapeutic procedure (e.g. transfusion) or to administration of a particular medication (e.g. insulin) or other treatment. For each task-specific order set, the related task was recorded.

• Service-specific: Order sets related to a specific hospital service (e.g. ICU). For each service-specific order set, the related service was recorded.

In addition, two other categories were identified on the basis of each order set’s function and origin:

• Convenience: Order sets that catalog laboratory tests, medications or clinical consult orders organized for ordering convenience, but not for any particular clinical purpose. For example, some sites used order sets like “AM Labs” or “Common STAT Labs” that allow common orders to be placed quickly. Unlike regular order sets, users are generally expected to pick only a single or small number of orders from a convenience set, and the convenience set lacks an associated clinical purpose – only some sites had convenience sets, and they were frequently used as workarounds when other mechanisms of organizing orders (e.g. order menus) were not available. As previously mentioned, we do not consider convenience sets to be “true” order sets but define these here for completeness.

• Personal: Order sets created or modified for use by an individual or group of practitioners rather than institution-wide committees.

The categories listed above are not mutually exclusive (e.g., a cardiology admission order set would be both “ADT” and “service-specific”). All order sets surveyed fall into at least one category, with many falling into multiple categories. In order to maintain consistency, rules for classification of recurring types of order sets were devised on an ongoing basis and are shown inAppendix A. When an order set was classified into one of the first five categories, specific details (e.g. the condition, task or service) were also recorded.

In order to further compare usage patterns across sites, we also developed an attribute called “order set signature” which combines multiple order set classifications into a single descriptive term. For example, using this strategy, all order sets classified as “ADT (admit)” and “Service-specific (medicine)” can be grouped across sites into the signature “Admit to Medicine.” We utilized this attribute to group related order sets across sites and create a list of top signatures.

Data Analysis

Once our list of order set types was finalized, classification of order sets was carried out by study staff (JF). Consensus checks were conducted with the primary author (AW) for a random subset of the order sets, high-use order sets and all those with potentially ambiguous categorization. Additional information was requested from study sites on an as-needed basis when an order set name was ambiguous. All data analysis was carried out in Microsoft Excel and SAS 9.2, including calculation of order set counts and category-specific usage statistics.

Sites

Order set usage data was collected from a diverse sample of sites with CPOE. The characteristics of each of the participating sites, including CPOE system, CPOE install year, order set vendor, location, hospital type, teaching hospital status, number of staffed beds (median: 395, average: 431), case mix index (median: 1.61, average: 1.59), discharges per year (median: 18,384, average: 25,021) and patient-days per year (median: 102,421, average: 129,791), are shown inTable 1. Participating sites included a geographically diverse mix of small and large, academic and community hospitals. Five sites had commercial CPOE systems, while two sites had internally-developed systems.

Order Set Types & Usage

Our data set consisted of 1,914 order sets. These order sets were used a total of 676,142 times in a one year period.Table 1 shows the total number of order sets in use at each site as well as the total number of uses during the study period and the average number of uses per set, per discharge and per bed at each site.Table 2 shows the number of order sets and total order set uses by category for each of the seven sites. Order sets from each site were classified into non-mutually exclusive categories as described above. The total by category and average uses per order set are shown on the right-hand side ofTable 2. By count, task- (n=1100) and service-specific (n=956) order sets were the most common, while personal (n=79) order sets were least common. Service-specific order sets contributed the most (53.3%) to overall usage, while personal order sets contributed the least (0.2%). ADT order sets had the highest uses per set (812.5) while personal order sets had the lowest (13.7).

Additional order set usage data is presented inTables 3a–e. For ADT (Table 3a) and perioperative (Table 3b) order sets, total number, total uses, average uses per set and number of sites with each order set type are shown for each subcategory. Admission order sets were the predominant ADT order set subtype by both count and usage. Post-operative sets were the most common perioperative order set subtype by both count and usage.

For condition-specific (Table 3c), task-specific (Table 3d) and service-specific (Table 3e) order sets, the top ten subcategories are shown by number and use as well as the number of sites with each subtype. The top ten conditions, tasks and services by usage accounted for 75.8%, 37.2%, and 63.9% of usage respectively within each category. The complete lists of all conditions (n=144), tasks (n=513) and services (n=67) by overall usage are available inAppendix B. In addition, 472 convenience order sets and 79 personal order sets were identified, accounting for 29.4% and 0.2% of total usage respectively.

Out of all 1,914 order sets, the top 20% of order sets (383 order sets) by use account for 90.1% of total usage. The cumulative distribution of order set usage by site is shown inFigure 2. The top ten individual order sets by usage (excluding convenience and personal order sets) are shown by site inTable 4a. In addition, the top 20 “order set signatures” (described in the methods section) are shown inTable 4b.

Major Usage Patterns

Implications and Future Directions

Our findings clearly show that order sets are a frequently used tool when available as part of CPOE. Given that order sets can be time- and resource-intensive to create, it is important to focus on those high-volume order sets that will be the most used. Our findings indicate that a small proportion of order sets will ultimately be most employed at any given site. Institutions hoping to implement order sets should work to identify high-value order sets based on local needs and focus their attention and resources on this subset.

We hope that sites planning to implement order sets will use these findings to guide the development or purchase of order set content. For those sites that have not yet implemented any order sets, it may be of value to analyze available discharge and billing data to identify areas ripe for the application of order sets. Final selection of order sets to be implemented should be tailored based on local priorities, governance practices (23) and available resources. In addition, we hope that those sites that have already implemented order sets will expand these findings based on their own order set usage data and continue to refine their catalogues.

For smaller sites and those with limited resources for developing order sets, it may be of value to purchase content from an established order set vendor rather than developing content “in-house”. Such vendor-generated order sets can serve as a good starting point for a local catalogue that can, over time, be customized to suit institutional needs. Vendor-generated order sets may offer an alternative to local development although both can require substantial investments of time and resources.

Another potentially valuable approach to encouraging order set use is to facilitate sharing of validated content. Our data suggests that there are order set types that many sites have in common. As such, it may be possible to develop high-value order sets that can be shared widely and then tailored to local needs. HL7 is developing a standard for representing order sets (18), and, if adopted, this approach may enable wider sharing of content. In fact, many EHR vendors already have electronic libraries where customers can share order set content (24,25); however, until the HL7 standard work is completed and the standard is adopted, interoperability of order set content across vendors is not possible; additional obstacles to content sharing include security concerns and economic loss to competitors (25), which may especially hinder vendors from creating sharing capabilities. Nonetheless, we believe it would be both technically feasible and highly valuable to create, through multi-site collaboration, a freely available “starter kit” of high-value order sets that could then be tailored to institutional needs and expanded based on local priorities.

Finally, sites looking to implement new order sets need to realize that post-implementation maintenance is crucial as a large corpus of order sets need periodic review. In order to restrict unintended adverse consequences of order sets, systems need content update and information review (26). As important as multidisciplinary collaboration may be during planning stages of new order sets (27), the same cooperation is needed during the maintenance stages. However, if periodic order set review is difficult for a specific site, it may choose not to develop as many start-up order sets.

Limitations

We examined a purposive sample that included only seven hospitals using CPOE. Our sample was designed to be diverse, but it is not random and is not necessarily representative of hospitals in the United States. In particular, our findings may not be generalizable to sites not represented in our sample (e.g. specialty, VA or children’s hospitals), though many of the themes and analysis techniques we identify are likely to hold.

Considerable differences in order set naming convention and granularity made it challenging to directly compare order sets across sites. However, we believe that the order set classification scheme described above serves as a useful means of unifying order set data across sites. The designed classification scheme is an original approach that has not been employed before; we have provided the scheme inAppendix A so that other researchers may be able to replicate the results. Furthermore, because only one member of the study team classified the order sets, we were not able to calculate a kappa coefficient. In addition, given the size of our database and the data sites were able to provide, analysis of the specific content of each order set and the usage patterns of individual providers was beyond the scope of this investigation. Differences in order set content and provider order set usage warrant further study, as the extent of homogeneity or heterogeneity of content is likely to influence the extent to which such content can be shared and individual provider behavior may be relevant to understanding variable order set usage across and within sites. Finally, we were unable to obtain complete information on order sets with zero uses from all sites, so such sets were excluded from our analysis. However, studying such sets could be useful for discerning patterns of order sets that are infrequently used.

Appendix B-2.

Appendix B-3.

Summary Table.

Research Highlights.

• We have developed a unique order set classification scheme.

• Order sets are widely used although usage statistics vary drastically.

• A small number of order sets accounts for the majority of overall order set usage.

• 1.Bates DW, Leape LL, Cullen DJ, et al. Effect of computerized physician order entry and a team intervention on prevention of serious medication errors. Jama. 1998 Oct 21;280(15):1311–6. doi: 10.1001/jama.280.15.1311. [DOI] [PubMed] [Google Scholar]
• 2.van Rosse F, Maat B, Rademaker CM, van Vught AJ, Egberts AC, Bollen CW. The effect of computerized physician order entry on medication prescription errors and clinical outcome in pediatric and intensive care: a systematic review. Pediatrics. 2009 Apr;123(4):1184–90. doi: 10.1542/peds.2008-1494. [DOI] [PubMed] [Google Scholar]
• 3.Holdsworth MT, Fichtl RE, Raisch DW, et al. Impact of computerized prescriber order entry on the incidence of adverse drug events in pediatric inpatients. Pediatrics. 2007 Nov;120(5):1058–66. doi: 10.1542/peds.2006-3160. [DOI] [PubMed] [Google Scholar]
• 4.Kaushal R, Jha AK, Franz C, et al. Return on investment for a computerized physician order entry system. J Am Med Inform Assoc. 2006 May-Jun;13(3):261–6. doi: 10.1197/jamia.M1984. [DOI] [PMC free article] [PubMed] [Google Scholar]
• 5.Ammenwerth E, Schnell-Inderst P, Machan C, Siebert U. The effect of electronic prescribing on medication errors and adverse drug events: a systematic review. J Am Med Inform Assoc. 2008 Sep-Oct;15(5):585–600. doi: 10.1197/jamia.M2667. [DOI] [PMC free article] [PubMed] [Google Scholar]
• 6.Chan J, Shojania KG, Easty AC, Etchells EE. Does user-centred design affect the efficiency, usability and safety of CPOE order sets? J Am Med Inform Assoc. 2011 May 1;18(3):276–81. doi: 10.1136/amiajnl-2010-000026. [DOI] [PMC free article] [PubMed] [Google Scholar]
• 7.Santolin CJ, Boyer LS. Change of care for patients with acute myocardial infarctions through algorithm and standardized physician order sets. Critical pathways in cardiology. 2004 Jun;3(2):79–82. doi: 10.1097/01.hpc.0000128715.42953.78. [DOI] [PubMed] [Google Scholar]
• 8.Micek ST, Roubinian N, Heuring T, et al. Before-after study of a standardized hospital order set for the management of septic shock. Critical care medicine. 2006 Nov;34(11):2707–13. doi: 10.1097/01.CCM.0000241151.25426.D7. [DOI] [PubMed] [Google Scholar]
• 9.Ballard DJ, Ogola G, Fleming NS, et al. The Impact of Standardized Order Sets on Quality and Financial Outcomes (Vol. 2: Culture and Redesign) 2008 Aug; [cited March 31, 2011]; Available from:http://www.ahrq.gov/downloads/pub/advances2/vol2/Advances-Ballard_12.pdf. [PubMed]
• 10.McAlearney AS, Chisolm D, Veneris S, Rich D, Kelleher K. Utilization of evidence-based computerized order sets in pediatrics. International journal of medical informatics. 2006 Jul;75(7):501–12. doi: 10.1016/j.ijmedinf.2005.07.040. [DOI] [PubMed] [Google Scholar]
• 11.Ozdas A, Speroff T, Waitman LR, Ozbolt J, Butler J, Miller RA. Integrating “best of care” protocols into clinicians’ workflow via care provider order entry: impact on quality-of-care indicators for acute myocardial infarction. J Am Med Inform Assoc. 2006 Mar-Apr;13(2):188–96. doi: 10.1197/jamia.M1656. [DOI] [PMC free article] [PubMed] [Google Scholar]
• 12.Fishbane S, Niederman MS, Daly C, et al. The impact of standardized order sets and intensive clinical case management on outcomes in community-acquired pneumonia. Archives of internal medicine. 2007 Aug 13–27;167(15):1664–9. doi: 10.1001/archinte.167.15.1664. [DOI] [PubMed] [Google Scholar]
• 13.Fleming NS, Ogola G, Ballard DJ. Implementing a standardized order set for community-acquired pneumonia: impact on mortality and cost. Joint Commission journal on quality and patient safety / Joint Commission Resources. 2009 Aug;35(8):414–21. doi: 10.1016/s1553-7250(09)35058-8. [DOI] [PubMed] [Google Scholar]
• 14.Payne TH, Hoey PJ, Nichol P, Lovis C. Preparation and use of preconstructed orders, order sets, and order menus in a computerized provider order entry system. J Am Med Inform Assoc. 2003 Jul-Aug;10(4):322–9. doi: 10.1197/jamia.M1090. [DOI] [PMC free article] [PubMed] [Google Scholar]
• 15.Meleskie J, Eby D. Adaptation and implementation of standardized order sets in a network of multi-hospital corporations in rural Ontario. Healthcare quarterly (Toronto, Ont. 2009;12(1):78–83. doi: 10.12927/hcq.2009.20418. [DOI] [PubMed] [Google Scholar]
• 16.Rothschild AS, Lehmann HP. Information retrieval performance of probabilistically generated, problem-specific computerized provider order entry pick-lists: a pilot study. J Am Med Inform Assoc. 2005 May-Jun;12(3):322–30. doi: 10.1197/jamia.M1670. [DOI] [PMC free article] [PubMed] [Google Scholar]
• 17.Wright A, Sittig DF. Automated development of order sets and corollary orders by data mining in an ambulatory computerized physician order entry system. AMIA Annual Symposium proceedings / AMIA Symposium; 2006. pp. 819–23. [PMC free article] [PubMed] [Google Scholar]
• 18.McClay JC, Campbell JR, Parker C, Hrabak K, Tu SW, Abarbanel R. Structuring order sets for interoperable distribution. AMIA Annual Symposium proceedings / AMIA Symposium; 2006. pp. 549–53. [PMC free article] [PubMed] [Google Scholar]
• 19.Middleton B. The clinical decision support consortium. Studies in health technology and informatics. 2009;150:26–30. [PubMed] [Google Scholar]
• 20.Wright A, Sittig DF, Carpenter JD, Krall MA, Pang JE, Middleton B. Order sets in computerized physician order entry systems: an analysis of seven sites. AMIA Annual Symposium proceedings/AMIA Symposium. 2010;2010:892–6. [PMC free article] [PubMed] [Google Scholar]
• 21.Services CfMM. FY 2010 Final Rule Home Page. 2011 [cited 2011 January 6]; Available from:https://www.cms.gov/acuteinpatientpps/10fr/itemdetail.asp?itemid=CMS1227442.
• 22.Wright A, Bates DW. Distribution of Problems, Medications and Lab Results in Electronic Health Records: The Pareto Principle at Work. Applied Clinical Informatics. 2010;1(1):32–7. doi: 10.4338/ACI-2009-12-RA-0023. [DOI] [PMC free article] [PubMed] [Google Scholar]
• 23.Wright A, Sittig DF, Ash JS, et al. Governance for clinical decision support: case studies and recommended practices from leading institutions. J Am Med Inform Assoc. 2011 Mar 1;18(2):187–94. doi: 10.1136/jamia.2009.002030. [DOI] [PMC free article] [PubMed] [Google Scholar]
• 24.Wright A, Bates DW, Middleton B, et al. Creating and sharing clinical decision support content with Web 2.0: Issues and examples. Journal of biomedical informatics. 2009 Apr;42(2):334–46. doi: 10.1016/j.jbi.2008.09.003. [DOI] [PubMed] [Google Scholar]
• 25.I-Form Library. [cited April 16, 2011]; Available from:http://www.iformlibrary.com/