In May 2018, a team from the U.S. Department of Veterans Affairs (VA) Veterans Health Administration (VHA) earned recognition as a finalist for ECRI Institute's 12th Health Devices Achievement Award. The team established a rigorous process for:

• Investigating the cause of treatment errors associated with the use of a blood glucose monitor (BGM)
• Researching possible solutions, which in this case involved assessing various configurations of the BGM's results screen
• Testing the various alternatives to identify a configuration that minimized the risk of future adverse events

The project team included representatives from the VA Pittsburgh Healthcare System (VAPHS), the VA National Center for Patient Safety (NCPS), and the VA Hudson Valley Health Care System (VAHVHCS).

The Health Devices Achievement Award recognizes innovative and effective initiatives undertaken by member healthcare institutions to improve patient safety, reduce costs, or otherwise facilitate better strategic management of health technology. For details about the winning submission and other finalists, see The Health Devices Achievement Award: Recognizing Exceptional Health Technology Management.

ECRI Institute congratulates all the project team members:

• From VAPHS: Jamie Estock, Holly Curinga, Audrey Gallagher, and Monique Y. Boudreaux-Kelly, PhD; the team also included former VAPHS Fellows Ivan-Thibault Pham and Benjamin Sprague, MD
• From NCPS: Katrina Jacobs and Tandi Bagian
• From VAHVHCS: Jeanette Acevedo and Marilyn Brammer
   

The Challenge

To reduce treatment errors associated with one model of BGM by identifying the optimal configuration for the device's results screen. The project involved:

• Designing a study to assess the various configuration alternatives
• Conducting a randomized, crossover simulation study
• Sharing the findings with the device manufacturer, FDA, and other healthcare facilities
   

The Landscape

1. VHA team members became aware of adverse events reported to FDA in which healthcare professionals misinterpreted information displayed on the results screen of a particular BGM and subsequently delivered inappropriate treatment. In two recent adverse events, the blood glucose result was displayed as an abbreviation intended to signify an out-of-range result, rather than as a numeric value. The healthcare professionals misunderstood what the abbreviation signified and responded with treatment that was not appropriate for the patient's condition. One of the incidents resulted in the patient's death. Both of these events involved the same model of monitor.

2. The BGM in question can be configured by the healthcare facility to display a critical blood glucose level in one of six different ways. The team theorized that certain configurations could be more prone to misinterpretation than others. It initiated a study to identify a configuration that would result in the fewest treatment errors.

3. Because VHA is the largest integrated healthcare system in the United States—it treats more than two million diabetic patients who require point-of-care blood glucose testing—and because many VHA facilities use the BGM in question, the outcome of the study could have far-reaching patient safety implications.
 

The Process

The VHA project involved three phases: a heuristic evaluation of the BGM's six results screen configurations, two pilot tests of the study materials and procedures, and a simulation study during which two results screen configurations were compared.

1. Heuristic evaluation

a) To identify which of the six results screen configurations were most—or least—likely to lead to errors in use, the team assessed each of the configurations against the following usability heuristics (see the References tab for sources of additional information on these heuristics):

(1) Speak the user's language

(2) Avoid outcomes that would surprise the user

(3) Reduce memory load

(4) Avoid obscure error codes

(5) Use standard meanings of words

(6) Use terminology consistently

(7) Limit abbreviations to those that are universally recognized

b) The team found that all six results screen configurations, which it labeled A through F, violated at least one usability heuristic.

(1) One of the configurations violated all seven of the usability heuristics. That configuration, referred to in the study as configuration F, had been involved in the two recent adverse events.

(2) Another configuration, configuration D, violated six of the usability heuristics. That configuration had been involved in another reported adverse event.

2. Pilot tests

a) Two concurrent pilot tests were conducted as a means to:

(1) Select which of the two problematic display configurations—D or F, as determined during the heuristic evaluation—would be compared in a randomized, crossover study with a configuration (configuration B) that the team hypothesized would result in fewer treatment errors

(2) Refine the study's test methods and procedures, including the computer-based simulation, clinical scenarios, and data collection tools

b) Study participants were registered nurses (RNs) and licensed practical nurses (LPNs) who use the BGM at least once every two weeks to evaluate blood glucose readings or perform blood glucose tests.

c) Participants were presented with two scenarios; in each, the participant was asked to interpret blood glucose readings and make treatment decisions for a simulated patient.

(1) In both cases, the simulated patient had a blood glucose level of 32 mg/dL and subtle symptoms of hypoglycemia (specifically irritability).

(2) The BGM display configuration was the only meaningful difference between the two scenarios.

(3) Treatment errors were recorded in two circumstances: if the participant did not treat the hypoglycemic patient or if the participant treated the patient incorrectly (i.e., by administering insulin instead of glucose).

(4) After the simulation, the researchers captured the participants' subjective feedback about the two results screen configurations in a free-response pros/cons survey.

d) During the pilot tests, more treatment errors were made with configuration F than with configuration D; thus, configuration F was chosen for the simulation study.

3. Simulation study

a) Researchers conducted a randomized, crossover simulation study to compare two results screen configurations. The configurations differed in how they displayed critical blood glucose readings:

(1) Configuration B provided a numeric reading of the blood glucose result.

(2) Configuration F provided an abbreviation representing an out-of-range result.

b) The study involved 66 nurses from two VA medical centers—one a large, high-complexity medical center and one a small, low-complexity medical center.

(1) The participant profile was the same as that used in the pilot studies.

(2) All participants had completed annual training on the BGM. One distinction, however, was that participants at facility 1 did not have prior training on the meaning of the range abbreviation used in configuration F, while the participants at facility 2 did.

c) As in the pilot tests, participants were presented with two simulated clinical scenarios. The BGM's results screen configuration was the only meaningful difference between the two scenarios.

d) The simulation incorporated:

(1) First-person point-of-view videos of blood glucose testing

(2) Conversations with simulated patients and healthcare professionals

(3) Access to relevant resources, such as the simulated patient's electronic healthcare record and the BGM's reference guide

e) The presentation order of the scenarios was randomized to control for order effects related to the presentation of configuration F and configuration B.

f) Treatment errors were recorded as described above for the pilot tests, and the participants' subjective feedback about what they liked and did not like about the two display configurations was captured via a free-response paper-and-pencil survey.
 

The Results

Interpreting the results from the study, the VHA team concluded the following:

1. Displaying a numeric blood glucose result eliminated potentially life-threatening treatment errors caused by confusing range abbreviations.

a) Eleven percent of participants made a treatment error when presented with configuration F.

b) None of the participants made a treatment error with configuration B.

2. Prior training on configuration F's range abbreviation reduced the number of treatment errors, but did not eliminate them. Of the participants with prior training on configuration F's range abbreviation, 7% still made a treatment error with configuration F.

3. Device users preferred to see a numeric value for the blood glucose reading. In the post-simulation survey, 53% of participants reported, without being asked, a preference for the glucometer to display a numeric value over a range abbreviation.

To put its findings into action, the VHA team:

1. Developed configuration recommendations to reduce the risk of treatment errors resulting from confusing range abbreviations on the model of BGM tested.

2. Communicated these recommendations to the VA National Center for Patient Safety (NCPS), which adopted them and developed VHA policy regarding these BGMs.

3. Further, NCPS discussed the findings with both FDA and the device's manufacturer.

a) In January 2018, the manufacturer released a firmware upgrade that included the removal of two alarm codes.

b) VHA views the removal of the two alarm codes as a positive step toward implementing its recommendations for BGM manufacturers, as outlined below.
 

Key Takeaways

1. Training alone cannot prevent life-threatening treatment errors from occurring when a medical device's messages are prone to being misinterpreted.

a) To reduce the risk of such errors with its BGMs, VHA now recommends that the device's reportable range be set to 10-600 mg/dL and that the device be configured to display critical results as a numeric value.

b) VHA's testing found that these settings, which match the manufacturer's default configurations, are less likely to be misinterpreted by healthcare professionals.

2. Additionally, healthcare facilities do not, by themselves, have the ability to eliminate all instances of confusing information displayed on the BGM. The VHA team has encouraged its BGM manufacturer—and it encourages all device manufacturers—to give greater consideration to usability heuristics when designing device displays and messages. For instance:

a) The BGM in question displays "out of critical range" in response to a blood glucose result that is critical.

(1) VHA's study found that 15% of participants reported, without being asked, that they were confused by the "out of critical range" message.

(2) Users could interpret this message as meaning that the result is not critical, because it is "out of" the critical range.

b) The BGM also displays numeric alarm codes for other conditions.

(1) Nine percent of VHA's study participants reported, without being asked, that they were confused about the numeric alarm codes.

(2) Users could mistake the alarm code for a blood glucose reading.

c) In both of these cases, the messages are inherent to the device design. Yet both violate usability heuristics by not using the standard meaning of words and by using obscure error codes.