Introduction

On September 23, 2014, ECRI Institute, with funding from the Jayne Koskinas Ted Giovanis Foundation for Health and Policy, convened an interactive, multi-stakeholder meeting, Partnering for Success, the first of a series of in-person meetings of the Partnership for Health IT Patient Safety.

Specifically, the Partnering for Success meeting was organized to address eight goals:

1. Understand the barriers and challenges of the health information technology (IT) learning system
2. Work on ways to define and categorize health IT safety issues and hazards
3. Agree on the health IT issues to share with the Partnership
4. Agree on characteristics of a successful health IT safety identification, triage, and investigation system
5. List practical ways in which stakeholders can immediately advance health IT safety
6. Share tools and best practices
7. Commit to Partnership goals and participate in follow-up work
8. Inform the national strategy for health IT patient safety

See the full meeting agenda here.

"What we have is an opportunity to really work together," said Jeffrey C. Lerner, PhD, when welcoming participants to the meeting (see About Our Speakers for more information on Partnering for Success presenters). "If we're able to cooperate effectively, we can work in a way that actually ensures patient safety."

Ronni P. Solomon, JD, described the purpose of the Partnership. "We want to make healthcare safer together," she said, and through establishing a nonpunitive learning environment. Solomon emphasized the importance of working together with fellow Partnership participants. "We're also testing a collaborative model," she explained. "We want to achieve robust stakeholder engagement."

The work of the Partnership encompasses three phases:

1. Data collection. Data collection and aggregation across multiple organizations is crucial. The Partnership collects reports of adverse events, near misses, and unsafe conditions using standardized formats as well as nonstandardized data, such as alerts, help desk logs, and claims information. The data provides a foundation for Partnership efforts by revealing contributing factors associated with health IT–related safety issues and by identifying opportunities to use health IT to enhance patient safety. For example, usability, interoperability, and hardware/software were three topics addressed by the multi-stakeholder participants during the interactive breakout sessions. During these sessions, participants were able to share their experiences and solutions and emphasize the importance of gathering this information in a central location.
2. Analysis. Analysis of the information obtained from the data will facilitate improvements in patient safety and in the use and development of the technology. The Partnership includes experts in information technology, patient safety, human factors, systems implementation, and healthcare operations, as well as the Expert Advisory Panel. Health IT system vendors serve as analytic contractors under the Patient Safety and Quality Improvement Act (PSQIA) and will help to analyze the data gathered by the Partnership.
3. Leveraged learning. The knowledge gained through the Partnership will be translated into meaningful practices, resources, and tools. Collaborating organizations will broadly disseminate these learnings via publications, at meetings, and through various professional organizations, many of which are participants in the Partnership and were present at Partnering for Success.

Informing the National Strategy for Health IT Patient Safety

The Partnership for Health IT Patient Safety is applying and building on patient safety principles set forth by the Institute of Medicine (IOM), the Office of the National Coordinator for Health Information Technology (ONC), the Bipartisan Policy Center (BPC), and others to establish a meaningful national framework for health IT safety.

In the 2000 report To Err Is Human: Building a Safer Health System, IOM identified a national agenda for change, specifying actions that entities should take to improve patient safety, including the implementation of nonpunitive systems for reporting and analyzing errors. In 2012, IOM issued the report Health IT and Patient Safety: Building Safer Systems for Better Care, which stated that "to protect America's health, health IT must be designed and used in ways that maximize patient safety while minimizing harm." The report emphasizes that the improvement of health IT safety is a shared responsibility, especially since health IT products are part of a larger sociotechnical system that includes people, organizations, processes, and the external environment. "Safety emerges from the interaction among [these] factors," says IOM. The report underscores the importance of generating, developing, and sharing safety risks and recommends that reports by users be voluntary and that identities of reporters not be discoverable under any circumstance. "User-reported health IT–related adverse events should be collected by a central repository and also be sent to the appropriate vendor," says IOM. (IOM)

The report from ONC published on July 2, 2013, Health Information Technology Patient Safety Action & Surveillance Plan, names two "fundamental objectives": first, to use health IT to improve the safety of patient care, and second, to constantly improve the safety of health IT use. "Achieving these objectives is a shared responsibility," states the report. The plan offers guidance for clinicians, nonclinical staff, patients and caregivers, the government, health IT developers, patient safety organizations (PSOs), accrediting bodies, and more. (ONC "Health . . . Patient Safety")

Likewise, the BPC report An Oversight Framework for Assuring Patient Safety in Health Information Technology identifies a set of principles that should guide strategic planning regarding health IT. These points include recognizing the role of health IT in improving patient care; ensuring that patient safety is a goal shared across the "entire health care system"; understanding that a health IT patient safety framework should be risk-based, flexible, and innovative; and underscoring the importance of reporting health IT-related patient safety issues. (BPC)

In 2014, FDASIA Health IT Report: Proposed Strategy and Recommendations for a Risk-Based Framework was published. This report recommended "the creation of an environment of learning and continual improvement," both to protect patient safety and foster innovation in health IT use. The FDASIA report recommended that such a learning environment should:

1. Identify, report and respond to health IT-related adverse events and near misses;
2. Aggregate and analyze events and near misses to identify patterns and trends;
3. Share information about methodology, practices, policies, and findings in a transparent manner;
4. Support the development and adoption of interventions and mitigations, where appropriate; and
5. Promote system-wide education and learning for stakeholders resulting in a system that is continually undergoing improvement. (U.S. FDA et al.)

The IOM, ONC, FDASIA, and BPC reports all recognize a role for PSOs in gathering adverse event information in a nonpunitive environment under the privilege and confidentiality protections of the PSQIA. The Agency for Healthcare Research and Quality (AHRQ), the federal agency responsible for regulating PSOs and implementing the PSQIA, has published guidance for health IT developers that wish to work to improve patient safety within the framework of the Patient Safety and Quality Improvement Act of 2005. (AHRQ "Frequently")

"We're seeing a host of problems with health IT, and there are some big opportunities to make those things better," said David W. Bates, MD, MSc.

Is collaborative learning effective? Sharing the transportation industry's experience, speaker David Mayer, PhD, emphasized that a multi-stakeholder group can be an effective vehicle to improve the system as a whole without punitive action. "We investigate transportation disasters using [this] really unique nonadversarial collaborative process," he said. Called the Civil Aviation Safety Team (CAST), the group, much like the Partnership, involves multiple aviation stakeholders—ranging from airplane and engine manufacturers to pilots and flight attendants, as well as trade associations and federal regulators—addressing common safety concerns and identifying feasible solutions for all. "The participants in CAST made a conscious decision not to compete on safety," he explained. "You're embarking on a collaboration that's really similar to this, and that's why I'm really excited about what you're doing," he told Partnership participants.

Identification of Health IT Safety Issues

Bates outlined the risks and benefits of health IT use. "Overall, the literature suggests that health information technology clearly appears to improve safety," said Bates. "But, the literature also provides many stories that describe how health IT creates new safety risks. And I would submit that the magnitude of harm and the impact of health IT on patient safety is uncertain; that's because of the heterogeneous nature of health IT. We have very diverse clinical environments, workflows [that are] different from place to place. The evidence in the literature is still relatively limited."

Reports in the literature conflict regarding the efficacy of health IT implementation, said Bates. He cited one study that found an increase in mortality rates after a commercial computerized provider order entry (CPOE) system was introduced (Han et al.) but noted that other organizations later "introduced exactly the same commercial CPOE application [and] actually saw their mortality rates fall." Bates posited that the difference between CPOE implementation success and failure in these scenarios was "related to the way that the application was introduced." One participant noted that effective team support in implementation and problem resolution has advanced the effective use of the technology and changed the focus to a resolution of more specific issues related to the components of the technology. See the discussion Health IT Safety Identification, Triage, and Investigation, later in these Proceedings, for more information.

Other significant risks created by health IT are in the areas of interoperability (e.g., incorrect merging of data), use (e.g., medication selection and patient identification errors), and hardware/software issues (e.g., unexpected downtime, truncated displays, problematic default settings). Each of these areas was addressed specifically by groups of participants in breakout sessions, which are documented later in these Proceedings.

"It's quite clear that health IT can introduce new errors," said Bates. "We have to have better frameworks to describe them." This effort will require new definitions and classification systems, he explained. Likewise, he recommended that organizations develop approaches to identify, track, and engineer errors out of their systems.

Compounding the risk is the fact that organizations have no central data repository to which to report safety issues. "They generally don't get aggregated at the national level," explained Bates. And, he added, patient safety surveillance relies on self-reporting of adverse events, near misses, and hazardous conditions. Because such issues are self-reported, it can be complex to discern if health IT is a factor when analyzing reported events, especially if the event is not designated as such by the reporting organization or individual. If the reporting organization is not attuned to the IT component of voluntarily reported issues, it may not identify significant, relevant health IT-related information when reporting the event for analysis.

Agreement on what precisely constitutes a health IT issue is a necessary foundational step. The challenges in determining what should be considered as a health IT-related event were illustrated in Bates' presentation, which included various case studies of potential health IT-related events. Bates presented several cases that involved health IT to a varying degree, seeking stakeholder perspectives on whether, in their judgment, the case would be considered a health IT safety issue and whether it would be reported as such within their organization and to the Partnership. Specifically, after each case was described, meeting participants were asked to anonymously indicate whether the issue involved was a health IT-related issue and if it should be reported. As shown in the following cases, participants were not always in agreement about whether the example simply involved health IT or was a health IT-related issue. In addition, there were varying opinions regarding who should handle the issue, demonstrating that even for those in the thick of health IT decision making, opinions can differ.

When determining what the current pressing concerns are, participants, through the use of anonymous polling, identified a vast array of differing issues. Throughout the course of the meeting, however, participants seemed to come to agreement regarding potential health IT priorities. See Consensus on Important Health IT Safety Issues for more.

Case 1: Free-Text Field Used for Lab and Medication Orders

Facts: The provider documented care in the free-text fields of the electronic health record (EHR) system, including a medication order and lab order.

Background: The risks in such circumstances are that the orders will not be carried out or will not be carried out in a timely fashion, and that information will not be readily communicated to other providers or trigger the appropriate alerts and fail-safes designed to make care safer. Currently, most EHR systems are not able to read free-text fields and populate data according to information contained in those fields. Yet many providers are familiar with documentation by free text; changing those practices (using the technology appropriately) may create patient safety vulnerabilities until there is adequate recognition of the capabilities and limitations of the technology.

Polling: When asked if this was a health IT issue, only 42% of participants agreed that it was. Half of participants believed it was not, while 8% were uncertain.

Case 2: Incorrect Data Entry and Drop-Down Selection

Facts: In another case, the provider began to type information into a data entry field, which brought the provider to a specific location in the field's drop-down menu. The provider had to choose the correct piece of information within the drop-down menu, but incorrect information was entered.

Background: Errors and unsafe conditions related to auto-population or auto-selection of data and drop-down menus can be a health IT risk. This type of error may be due to multiple factors. In some instances, selections are highlighted and automatically entered unless a provider chooses an item further down in the list. Additionally, "smart" applications recognize standard terms and auto-complete words or phrases with minimal prompting.

Polling: A majority of respondents, nearly 67%, indicated their belief that this was a health IT-related issue, while the rest of respondents were almost evenly divided between being uncertain and believing it was not health IT-related.

Case 3: Lack of Selectivity and Silenced Alerts

Facts: In a third case, a healthcare organization elected to turn off all red-flag (soft) alerts in their EHR system because the software did not allow alert selectivity. During one patient's short-stay procedure, Toradol was ordered, but the patient had an allergy to naproxen (an ibuprofen compound, an allergy to which contraindicates the prescription of Toradol) and experienced a reaction.

Background: Alerts in health IT systems are a complex and often under-considered source of risk. The creation of too many informational alerts can lead to the phenomenon known as alert fatigue, and the creation of either too many or too few alerts can lead to medication errors, order duplication, and other patient safety risks. When implemented deliberately and mindfully, alerts can be beneficial to patient safety and care quality. (ECRI Institute "Implementing") There is currently no standard regarding alert settings; to date, alarm setting decisions are made by individual facilities.

Polling: A majority of participants, 71%, agreed regarding this case, determining the issue to be health IT-related. Of the rest, 19% believed it not to be health IT-related, and 10% were uncertain.

The Partnership hopes to identify areas where alerts would be beneficial, work with software vendors to facilitate use, and determine best practices. This was one area recommended for deeper consideration as a result of participants' discussions during the meeting. (For more, see the discussion Immediate Advancement of Health IT Safety, later in these Proceedings.)

The Importance of Reporting

Another barrier to learning is inadequate reporting. One participant, representing a healthcare organization, shared that her organization was often challenged to convince staff to report health IT safety issues, finding that they were reluctant to report issues for fear that vocalizing an issue meant that they would appear to not be using the technology appropriately or safely. In order to overcome this barrier, the participant highlighted the need to encourage reporting by reminding staff that "it's OK, and we're learning from this." The Partnership's learnings arise from the collection of a vast array of data, information, and stories. It was reinforced throughout the day that each contribution and interaction is important to collaborative learning and, ultimately, advancement.

Barriers to Building a Health IT Learning System

"If you . . . were to take a look at the Institute of Medicine report on this issue . . . , the work of the FDASIA workgroup . . . , the Bipartisan Policy Center report, and all of the other thought leadership pieces out there, you'd see the common theme around this whole issue is about building a learning healthcare system," explained moderator Janet Marchibroda, MBA.

"It really is a whole system that needs to work together," commented one participant.

Another stressed the importance of identifying and addressing barriers early "because [they] can really slow down [the] process" if they are discovered later.

As discussed by Dean Sittig, PhD, and Hardeep Singh, MD, MPH, during the meeting, key barriers to creating a learning system are the absence of widely accepted and useful taxonomies to describe health IT safety concerns and surrounding data and currently underdeveloped mechanisms of effectively reporting, aggregating, and analyzing data.

These concerns resonated with participants. One asked, "How [do we] collect the data in a standardized fashion so we can actually all use the data?" This participant worried that the combination of aging technology and a need to report desensitized or deidentified data would seem complex to practitioners. "I think that's a big barrier for us to overcome," said this participant.

Another participant pointed to the internal barriers to achieving learning for vendors, stating that while "no one wants to create software that has the potential to harm," getting IT professionals to "walk in a physician's shoes" is a challenge.

Limited understanding of clinical staff about health IT issues was also a barrier for another stakeholder, who found difficulties getting staff to recognize health IT issues. In order to overcome this barrier and align a common interpretation of events with health IT issues, one healthcare organization used broad-reaching policies to help prevent unintended consequences of health IT use. These policies included definitions of IT staff member and health IT senior staff roles, a procedure for reporting IT systems issues, a list of information required in a health IT safety report and steps for reporting, a flowchart of the path each issue would take during analysis, and an assessment of the degree of issue criticality and the response required. (Veterans Health Administration/Office of Information)

A participant representing a professional society believes that another barrier to the learning system that the Partnership will need to address is identifying health IT solutions that are "simple and elegant" in order to receive buy-in from the healthcare world and that the group must determine which high-priority health IT issues to tackle.

Developing A Common Language for Health IT Safety Issues

"Measurement is the first step to improvement," said Singh. Speaking about the importance of identifying a standard scheme for classifying health IT-related safety concerns, he noted, "It's very clear that a lot of us are not on the same page." By sharing a common language for classifying events, Partnership stakeholders can more easily measure and assess the shared data and share lessons to improve health IT.

The Partnership uses two standardized approaches to help providers uniformly report patient safety events and hazards: (1) AHRQ's Common Formats for reporting health IT issues and unsafe conditions and (2) Hazard Manager, a management tool and ontology that captures information about health IT hazards before they can cause harm. Hazard Manager was developed with funding from AHRQ. (AHRQ "Health Information") The Partnership may identify opportunities to improve upon these taxonomies in order to collect actionable information to clarify parameters of focus. For more information about these standardized taxonomies, see Standardized Taxonomies for Health IT Issues.

The use of a common vocabulary and proper classification of a health IT issue or event can help to enhance an organization's response to an event, hazardous condition, or near miss. (ONC "Health . . . Adverse")

Said one Partnership participant: "How you classify [an event] will be how you manage it."

Simply knowing that technology was involved allows organizations to better realize the potential risks and safeguards inherent in its use. Sittig proposed five types of health IT-related safety concerns that should be considered when evaluating health IT issues. He described them as follows:

1. The health IT system fails. In these instances, the system either fails during use or is otherwise not working as designed. For example, Sittig recalled a case of a patient who was taking 100 mg of a medication. Because the facility where he was hospitalized stocked only 25 mg pills of the medication, the patient was prescribed four of the 25 mg pills at a time. At discharge, the system merged the outpatient dose of 100 mg and the quantity of pills from the inpatient encounter and indicated that the patient should receive 400 mg of the medication. "It turned out that the problem had happened to 50 other patients, but no one had caught the mistake." These instances of hardware or software not working "need to be talked about, and they need to be fixed as soon as possible," said Sittig.
2. The health IT system works as designed but does not meet the user's expectations. In these situations, there's a mismatch in how the system is designed and how it is used. "Usually we talk about these as usability issues," he said. "That's when we really need to work together" with the vendor. "The developers have a mental model about how it should work, and the users have a [different] mental model."
3. The health IT system is working as designed but is not configured correctly. A good example, said Sittig, is duplicate alerts for pain medication to be taken on an as-needed basis (e.g., when two pain medications are prescribed—one short-acting and one long-acting). In these situations, the computer may look at a duplicate order for pain medication and give a duplicate warning, even though the medication is prescribed as needed. "It's working as designed, but it's really not what we tried to configure."
4. The health IT system is working as designed and configured, but an interaction with another system causes problems. As an example, Sittig described an unintended interaction between a health IT system and an admission, discharge, and transfer (ADT) system. Anytime the patient was transferred, the ADT system discontinued the patient's medications and required the user to reenter the patient's medications. Even when a patient was transferred from one bed to another, the ADT system still classified the encounter as requiring that the patient's medications be discontinued and reentered for the new setting. "It's very difficult to test for this [how different IT systems interact] because you can't imagine how everyone is going to connect all these systems together," said Sittig.
5. Specific health IT safety features or functions were not implemented or were unavailable. In some instances, an event occurs that could have been prevented with a particular health IT system feature. For example, say a hospitalized patient inadvertently receives more than the recommended maximum daily dose of a medication; an alert could stop this from happening. Said Sittig, "These are things we want to have happen. This is the goal of health IT."

Reporting Health IT Issues to the Partnership

Because of uncertainty about the unsafe conditions and risks associated with health IT, the Partnership for Health IT Patient Safety is collecting a variety of reports about health IT-related safety issues to inform the national safety strategy for health IT and to determine which issues on which to focus its efforts. Partnership stakeholders agree that health IT provides numerous benefits, such as supporting clinical decision making, enhancing provider communication, providing access to patient data in a secure environment, engaging patients, and reducing medical errors.

Participants recognize, however, that health IT can create new safety risks if it is not designed appropriately, implemented carefully, and used thoughtfully (ECRI Institute "ECRI Institute PSO Deep Dive"; Sparnon and Marella). Analyzing data about health IT-related safety concerns collected by all the Partnership stakeholders can provide insights into identifying factors that contribute to these concerns and developing strategies to prevent similar problems from arising.

At the meeting, participants sought to identify the types of reports addressing health IT-related safety concerns that should be shared among Partnership stakeholders for leveraged learning. They also considered the format for submitting some of the reports in order to collect the necessary information about a particular health IT-related issue.

Report Types to Share

Solomon identified the multiple sources of data for health IT-related safety concerns collected by the Partnership: adverse event reports, near misses, help desk tickets, system alerts, root-cause analyses, and more. The data is analyzed to identify factors that contribute to the problem and to distill any lessons to improve health IT safety and to share with health IT stakeholders.

Event reports of health IT-related issues collected by healthcare facilities and health IT vendors are valuable sources of data for improving health IT safety, but they are not the only source for information. The Partnership is collecting data from multiple sources, including the following:

• Adverse event and near-miss reports from healthcare organizations
• Help desk requests submitted by healthcare facility staff to their IT departments and their health IT vendor
• Alerts from vendors sent to their health IT system users
• Vendor summary data
• Assessment data
• Root-cause analyses and investigations of health IT-related safety concerns and events conducted by healthcare facilities and vendors
• Published evidence-based research
• Reports of health IT-related events submitted to the U.S. Food and Drug Administration's medical device reporting programs (e.g., Manufacturer and User Facility Device Experience database, MedSun)

One participant noted that medical professional liability claims data can also be an important information source, as has been the case with understanding the factors affecting diagnostic errors and obstetric safety. Through collaborating organizations, the Partnership anticipates adding this type of information to its analysis. Given that there is sometimes a delay before a claim is filed following an alleged event, claims related to health IT issues may take time to emerge as the technology gains widespread use.

This participant reiterated that some malpractice claims related to health IT may not be clearly identified as such by the reporting organization. "We tend to think there's a lot more under the radar that we're not even identifying," the participant commented.

Health IT Issues That Warrant Reporting

Health IT reporting programs, such as the Partnership's reporting initiative, should identify a list of serious safety concerns associated with health IT that must always be reported, recommended Sittig. He gave meeting participants an advance look at a list of eight "must report" health IT safety events that he and coauthor Singh have proposed if a federal center is established to monitor health IT safety. (Sittig et al.) There is no consensus yet on mandating the reporting of certain issues, and valuable learning is often obtained from reports regarding the unanticipated issues.

Regarding these eight health IT safety issues, Sittig said, "These are serious events. Every time they happen, they should be reported." As an example, he listed computerized alerts with high clinician override rates. When an alert is always overridden, "that's saying something is wrong. It's not working," so it should be reported, he said.

The eight suggested reportable health IT safety issues are as follows (Sittig et al.):

1. Unexpected EHR-related downtimes lasting more than eight hours
2. Interruptive alerts that have fired more than 100 times with a 100% override rate
3. Erroneous displays of laboratory test results or medications
4. Roll-backs to an older version of EHR software (e.g., a software upgrade affected the system's function)
5. Instances in which a data backup failed to reload properly
6. Data losses affecting more than 100 patients
7. Software calculation errors affecting more than 100 patients
8. System configuration errors affecting more than 100 patients

This list of must-report health IT events has been proposed for national efforts to monitor health IT safety. Individual healthcare facilities may want to collect types of problems and near misses that exceed the list's scope. For example, while a national program may be interested in limiting reports of software errors to those affecting multiple patients, individual healthcare facilities may want to collect reports of any software calculation error.

Reporting Near Misses: Improvement Prior to Harm

Partnership stakeholders did not want to limit reporting only to events that cause patient harm. They supported the need for reporting and analyzing all types of incidents, including those that do not cause any harm, near-miss incidents, and circumstances that precede an actual event and are caught before anything can happen (i.e., hazardous conditions). Likewise, participants are interested in contributing information about instances in which technology has been used to make care safer. In order to properly analyze all such information and identify best practices, "It will be useful to collect lots of [events] and to use techniques like natural language processing and other approaches to go through them rapidly and classify them in various ways," said Bates.

In fact, many of the event reports submitted to ECRI Institute PSO are near-miss events. Analyzing these reports provides "an opportunity to make improvement" before patients are harmed, said Solomon. "They may not have actually caused harm, but they have the opportunity to cause harm." In addition, the Partnership is examining help desk logs; these can help to identify potential issues before they affect patient safety.

Reporting from All Phases of the Health IT Life Cycle

Reporting of health IT events, issues, and hazards should cover all phases of the technology's life cycle, said Singh. Each phase of the life cycle represents a step in the technology's evolution; consequently, different types of events may arise during each of the three phases and should be captured in the Partnership's collection and analysis of health IT-related issues.

There are three phases to health IT; each phase can affect patient safety differently, said Singh. As a result, organizations must heed the incidents arising during each phase to ensure health IT safety. The phases are as follows (Sittig and Singh "Electronic Health Records"):

Phase One: Safe IT implementation. During the early stages of health IT's life cycle, the focus is on implementing the technology and ensuring its safe use. Events during this first stage typically involve implementation issues. Are the system interfaces fully functional? Is the software configured to accomplish tasks as expected? As an example of a phase one event, a computer glitch in a medication ordering system mistakenly prescribed a male impotence drug for 900 smokers seeking an antismoking medication; the electronic formulary selected a list of the most popular medications, and the wrong medication was inadvertently selected.

Phase Two: Using IT safely. As the healthcare staff adjust to use health IT, phase two dominates. Events during this stage involve the unsafe or inappropriate use of the technology by people and their organizations. For example, has the organization set up too many computerized alerts, leading clinicians to overlook critical alerts because of information overload? This is a "classic example" of a phase two problem, said Singh. In one study, prescribers reported receiving a median of 63 alerts per day; nearly 87% of respondents perceived this to be excessive, nearly 70% believed they could not effectively manage these alerts, nearly 55% reported the potential for test results to be missed, and nearly 30% reported having missed a result that led to a delay in care because of the excessive alerts (Singh et al.).

Phase Three: Using IT to monitor and improve safety. At this stage, health IT reaches its intended goal to improve and monitor safety. Events during phase three are triggered when the system detects medical errors and gaps in patient care, such as identifying patients who have not received follow-up for abnormal cancer screening tests. "We're not all there yet because we're mostly struggling with phase one and phase two," said Singh. Another factor to consider: improvement and monitoring may be performed very differently and have different indicators and outcomes for individual organizations. The final goal of effective, improved patient care—rather than just meeting quality indicators without really doing the work—must always be kept foremost in mind. (See Figure 2. Health IT Life Cycle for more.)

Reporting across the Continuum of Care

Partnership participants also recognized that reporting of health IT safety concerns should extend across the continuum of care and include events from hospitals, physician practices, long-term care facilities, and other settings connected to a health IT system. As the patient moves from one healthcare setting to another, the patient's data is likely entered into different health IT systems. Currently, the interoperability of these systems is imperfect, which can limit clinicians' understanding of the patient's continuity of care across the continuum. Participants in the day's meeting indicated that interoperability is one of their primary concerns regarding health IT safety.

"We need to think about the longitudinal patient journey as patients receive care in different settings," said Singh. Reporting of health IT interoperability concerns across the continuum can provide insights into the safety issues. "We don't have the data because we're so dispersed with our IT systems and fragmented," he said.

Health IT Safety identification, Triage, and investigation

"What are the key characteristics of a successful health IT safety identification, triage, and investigation system?" asked Marchibroda. Participants considered the necessary components of a process for identifying, triaging, and investigating health IT safety issues. Terhilda Garrido, MPH, ELP, of Kaiser Permanente, stressed the importance of working with health IT vendors and health IT stakeholders when identifying and managing health IT safety events.

"We absolutely need our health IT vendors to be working with organizations as we problem solve," Garrido emphasized to Partnership stakeholders. "Neither of us has the sole expertise or understanding in such a complex system of how to both identify and remediate these things." See Kaiser Permanente's Systematic Approach to Solving Health IT Concerns for a description of the healthcare system's approach.

Another health system noted that it retained a core group of its 180-person health IT implementation team to continue to meet weekly to review current issues from a quality and safety perspective. When the health IT system was phased in at the various hospitals, the team staffed a command center to monitor reports as they were coming in and tracked their resolution. "There's a portion of the team that still exists . . . to look at the issues that are coming in [and address] how do we deal with those issues, what kind of changes do we need to make," noted the participant.

Throughout the meeting, participants identified various characteristics of an effective health IT safety reporting program, including the following:

• Make reporting of health IT safety concerns easy for users; otherwise, users are less likely to report problems they encounter even though they have been told that the information is needed to improve health IT safety. "We should be able to set this up so that reporting is really simple," recommended Bates.
• Recognize the limitations encountered by some settings in reporting health IT safety issues. Physician practices, for example, typically do not have an IT department for reporting their concerns. Instead, they will report the problems they encounter to the system vendor, said one meeting participant.
• Establish a nonpunitive environment, free of any finger-pointing, for reporting and identifying problems with health IT. As one meeting attendee said, "You bring everybody to the table, and everybody can contribute to solutions."
• Adopt a multidimensional approach to analyzing health IT-related events to understand the many factors that can contribute to health IT safety issues. One approach is to use the sociotechnical model developed by Sittig and Singh and described in Sociotechnical Model for Health IT Event Investigation.

Immediate Advancement of Health IT Safety

When asked by an audience member about how the Partnership members can break down barriers and keep stakeholders engaged, one panel member used an appropriate analogy: "How do you eat an elephant? Piece by piece." Recounting a personal anecdote about a physician who was so excited about his new EHR system that he couldn't stop talking about it at a party, this participant posited that working individually with champions at different healthcare organizations will help to ensure buy-in and success in making health IT safer for patients.

Another panel member commented that while there are issues with health IT that need to be addressed, it has had a positive effect on many different aspects of healthcare. One way this participant's organization has been able to foster innovation and bring new ideas to the table is by offering member facilities monetary "risk reduction awards" through a contest wherein individuals submit proposals for patient safety improvements; almost two-thirds of the submissions for the first set of awards were related to health IT.

To facilitate and encourage reporting, Partnership participants were given two tools to promote health IT safety at their organizations: a "thank you" card to give out to staff who submit reports and a flyer to inform other stakeholders of the Partnership. Another resource for organizations to help identify issues with health IT that was mentioned frequently were the nine ONC's SAFER guides, which are available at http://www.healthit.gov/safer/safer-guides.

At the meeting, participants worked together in breakout sessions to share and recommend potential strategies that can help to strengthen health IT safety and usage. These groups focused on use and user error, interoperability issues, and hardware/software issues. The goals of the breakout sessions were to identify the characteristics of each of the three focus topics, to create discussion questions to help in the identification of the issues, to determine the best way to report the issue, to identify follow-up actions based on stakeholder experience, and to determine how learning from the actions taken can be disseminated.

The use and user error breakout groups sought to define the line between an error due to the technology and an error made by the user when using the technology (e.g., entering incorrect data). Here, training and user accountability are key. Additionally, understanding which systems should be standardized and which systems are best left to be configured by the individual organization is also important. These groups discussed the ways in which system design or appearance can help or hinder system use. The goal, agreed breakout session participants, is for the system to make the correct action be the easiest one to take.

The interoperability breakout groups debated how to identify interoperability issues, methods of reporting interoperability issues, and to whom such issues should be reported. Here, ensuring that data is timely and reliable and that access is possible across the continuum is vitally important. Concern that errors can be easily multiplied as systems share and communicate information in different platforms is a challenge, as data needs to be accessed in a multitude of systems. Participants agreed, however, that reporting such issues should be simple for practitioners because the ease of reporting increases the likelihood that issues will be reported and errors will be corrected early. 

The hardware/software groups agreed that the identification of health IT safety issues is complex and difficult and that issues go underreported. System downtimes are a challenge. Processes need to be in place to accommodate care and obtain information about patients during what could be an extended period of outage. Moreover, providers are challenged in populating that information back into the record once a system is again operable. Understanding the system vulnerabilities is important. Thus, the groups discussed the value of simulation testing before systems are implemented, as well as that of testing systems after each upgrade. The importance of evaluating hardware and software installations from a high-reliability perspective was also discussed.

Each breakout group suggested several strategies to combat identified health IT safety issues. See Potential Health IT Improvement Strategies for strategies that were identified as a result of these breakout sessions.

Disseminating Tools and Best Practices

Sharing information, including tools and best practices, was a common theme at the Partnering for Success meeting. Many of the representatives of healthcare organizations were interested in what others were doing to combat issues such as aligning the inpatient and outpatient EHR systems and determining how best to use heath IT system alerts and alarms, as evidenced by discussion among the participants. At the beginning of the day, Solomon remarked that ECRI Institute has been involved in reporting for 40 years and has provided a collaborative for sharing and learning, so ECRI Institute understands that "if it's not a nonpunitive [reporting system], you don't get much data." (See Appendix B: ECRI Institute Health IT Safety Resources for additional information and tools.)

"How can organizations avoid these [health IT] issues?" asked Bates. "Well, there are some best practices, like in the IOM report, like in the SAFER guidelines, but they are not used pervasively today. And it's been hard to learn from the experiences of others. There's some sharing within users of individual vendors, and the vendors have been good about sharing stories, but that's uneven, and across vendors, we haven't done so well. So one of the things we need to do . . . is we need to identify some best practices and then spread them."

Tejal Gandhi, MD, MPH, CPPS, led a panel discussion on how the Partnership can build a health IT learning system and how organizations can create and share health IT safety information.

One participant uses e-mail, learning days, and symposiums to disseminate information among the healthcare facilities within the organization, but the most successful strategy that has been used is to convene groups with shared interests (e.g., obstetric safety) to improve care. The participant also discussed the importance of assembling various executives from each medical center to make up the patient safety committee, thereby supporting buy-in from each of the executive's facilities.

An IT vendor noted that its goal is to prevent health IT issues before they occur, and the major undertaking is to get IT workers on the vendor side to look at and test software as users would. In order to accomplish this, the organization shares customer stories, "even if they're painful," and other information in order to help staff understand what it's like for healthcare workers to use the system and the importance of changes to the system.

"There is a phenomenal appetite for this data," commented one participant. "We hear it all the time in our own organization. . . . So just even having that platform where we are publishing and [sharing] evidence-based information [is beneficial], because people are really groping in the dark right now for things. Everybody wants to know [if] someone else has tackled this and made some progress with it." This is a benefit of the Partnership; it is transparent, and issues and learning are shared among participants.

Participants believe that the Partnership will be useful in getting people to work together more effectively. For example, one participant recounted how the surgical departments at a healthcare organization were all experiencing a similar health IT issue and had the same vendor; yet in contacting the vendor, they each received different feedback on the issue and were approaching it differently. Knowing others were addressing the issue at the time would have been useful. "It was just a big 'aha' moment," said this participant; it led to a decision to approach the problem together in order to identify the best solution. Likewise, identifying shared problems and working on them centrally through the Partnership can lead to a more efficient and effective outcome.

Indeed, another participant noted that having such data available as a resource is valuable on its own. "It would help provide that evidence base for changing practices. . . . To be able to [use data from the Partnership] and have the ability to say 'it's not just us,'" would be invaluable, this participant explained, as would feeding back into the system to share the learnings.

Another participant hopes to use such information to educate and train staff. This vendor representative believes that the data will "give them the tools to understand how [the system] could be used or how the software could be used to the best of our ability to prevent [issues] from occurring." She further believes that the Partnership can be beneficial by eliminating the fear of reprisal that many vendors have about speaking out about health IT issues.

Hospital participants detailed the struggles they have implementing EHR and other health IT systems into their facilities. Participants agreed that criteria for health IT "must have" features and settings would be useful to help ensure that EHR implementation is performed safely. Participants felt that vendors likely know what is working and not working, but there's a discomfort in telling a hospital how to set up the system. One suggestion was voiced for an Amazon-like "suggestion" regarding how to set up EHR systems (e.g., if you are a 50-bed rural hospital, here is what most facilities like you have chosen to do).

Participants discussed the difficulty of reaching out blindly to other hospitals for suggestions on how to implement EHRs. One participant pointed out that "crowdsourcing" this information can be problematic because some outliers may actually be more advanced than most other facilities. But participants agreed that having a forum to discuss what issues occurred during implementation would be helpful to others.

For additional best practices, see Kaiser Permanente's Systematic Approach to Solving Health IT Concerns.

Commitment to Goals and Follow-Up

The Partnering for Success meeting provided many points of discussion; however, Singh pressed the group for specifics: "What are three things we can think about over the next year?"

In response, many participants volunteered to form working groups to tackle the issues identified at the Partnering for Success meeting. Workgroups will study why the reported events occurred and identify best practices for preventing their recurrence. Topics being considered by the Partnership's Expert Advisory Panel include the following:

• Auto-completion of text in critical data entry fields
• Copy-paste or cut-paste in progress notes
• Limiting the amount of records able to be opened simultaneously
• Determination of items available in drop-down lists and conformation of item selection
• Elimination of unacknowledged communication
• Use of Tall Man lettering (Institute for Safe Medication Practices)
• Elimination of "renew all" or "transfer all" functions
• Reduction of over-alerting
• Elimination of automatic end times for certain medication schedules
• Establishment of standard recent test result on-screen location
• Identification of methods to address wrong-patient chart entries
• Identification of staff training and training verification strategies for health IT systems

Looking Forward

One participant shared her excitement about the model for shared learning sought by Partnership stakeholders. "For us, this really does represent a different way of thinking and a different way of doing work together across the health IT stakeholder community," she said. "We're excited to be in the same room with providers, academics, professional organizations, and thought leaders. . . . We really look forward to doing things differently and working with this group."

"I think it truly is the innovation that we're all in the room together, working together," Solomon said to close the day. "This is the group that's going to make something happen."