Quality Requirements and Standards

In 2010, an assessment study of 27 different glucose meters was conducted. That same year, new papers in the literature advocated for tighter quality requirements. An FDA-sponsored public hearing on glucose meter quality also recommended tighter quality requirements. How will these 27 different glucose meters fare if quality standards are tightened?

October 2010
Sten Westgard, MS

In March of 2010, the FDA held a public meeting about the quality of blood glucose meters (BGM). To outside observers, the thrust of the meeting was that current quality standards for BGMs were too lenient and that stricter guidelines needed to be implemented.

Later in the year, Brad Karon, George Klee and James Boyd performed simulation studies with historical patient data and determined that, if Tight Glycemic Control was a possibility, the BGMs would need to meet an allowable total error of 15%. [Glucose Meter Performance Criteria for Tight Glycemic Control Estimated by Simulation Modeling, Brad S Karon, James C. Boy, and George G. Klee, Clinical Chemistry 2010;56:7; 1091-1097]

The question that arises from this recommendation is, how many BGMs on the market have the potential to meet this tighter standard?

As fate would have it, a comprehensive study of a multitude of BGMs was conducted in 2010:

System Accuracy Evaluation of 27 Blood Glucose Monitoring Systems According to DIN EN ISO 15197, Guido Freckmann, MD, Annette Baumstark, PhD, Nina Jendrike, MD, Eva Zechomack, MD, Serge Kocher, PhD, Jacques Tshiananga, MPH, Frank Heister, PhD, and Cornelia Haug, MD, Diabetes Technology and Therapeutics, 2010;12;3:221-231.

Not only did this study evaluate BGMs on the basis of whether they met current quality standards, but they recorded the data in such a way that we can evaluate whether any of these BGMs will meet possible future quality standards.

The study

Freckmann et al prepared their evaluation study according to the methodology described in DIN EN ISO 15197-2003. It was conducted in 2008 at the Institute for Diabetes-Technology in Ulm, Germany.

"The accuracy of 27 BG monitoring systems was assessed by comparison with a reference method. For each BG monitoring system 200 results were obtained from 100 subjects according to the standard DIN EN ISO 15197:2003."

These 200 results were carefully partitioned so that results covered a broad range of the test method:

"The standard ISO 15197 specifies that the BG concentrations of the blood samples shall be distributed as follows: 5% <50 mg/dL, 15% >50 to <80 mg/dL, 20% >80 to <120 mg/dL, 30% >120 to <200 mg/dL, 15% >200 to <300 mg/dL, 10% >300 to <400 mg/dL, and 5% >400 mg/dL."

The accuracy was measured against two different methods:

"Reference measurements were performed with two different methods for all BG monitoring systems; (1) glucose oxidase (YSI 2300 STAT Plus glucose analyzer, YSI Life Sciences, Yellow Springs, OH), and (2) hexokinase (Hitachi 917, Roche Diagnostics GmbH, Mannheim, Germany). The accuracy of the glucose oxidase method was verified measuring NERL Glucose Standards (Thermo Fisher Scientific, East Providence, RI), verified against National Institute of Standards and Technology (NIST) (Gaithersburg, MD) reference material. The accuracy of the hexokinase method was verified measuring NIST Standard Reference Material 965a."

In other words, the study checked the reference methods against higher-level reference materials, providing additional assurance that these methods would provide measurements that can be considered more authoritative.

The assessment criteria

The study assessed whether or not results fell within 15, 10, and 5 mg/dL at concentrations less than 75 mg/dL, and whether or not results fell with 20%, 15%, 10%, or 5% at concenstrations above 75 mg/dL. In addition, the results were plotted on Clarke Error Grids (described below).

The Clarke Error Grid is one of the oldest assessment tools for glucose method assessment. [Clarke WL, Cox D, Gonder-Frederick LA ,Carter W, Pohl SL: Evaluating clinical accuracy of systems for self-monitoring of blood glucose. Diabetes Care 10:622–628,1987]. This tool uses a grid with five different zones of performance:

Zone A: Accurate
Zone B: Not accurate, but no error in treatment
Zone C: Not accurate, potential to overtreat
Zone D: Not accurate, potential failure to detect
Zone E: Not accurate, potential wrong treatment decision

[graphic source: http://en.wikipedia.org/wiki/File:Clarkeerrorgrid.gif]

The most commonly expressed requirements for a Clarke Error Grid is that 95% of values should fall within Zone A+B. Thus, less than 5% of the results are permitted to fall in zones C, D, and E.

Subsequent glucose meter standards have more or less followed the precedent of the Clarke Error Grid, setting goals for 95% of the results. This is not without controversy - one perspective is that there should be goals set for where 100% of the results.

One key aspect of most of the requirements is that there are two different requirements, one requirement for the low end of the range, typically <75 mg/dL or <100 mg/dL where measurement is particularly challenging, and another requirement for measurements above that threshhold.

The ISO 15197 standard sets the goal of 95% of results should fall within 15 mg/dL of the true value < 75 mg/dL, and within 20% of the true value for values  > 75 mg/dL. [International Organization for Standardization. ISO 15197 in vitro diagnostic test systems—Requirements for blood-glucose monitoring systems for self-testing in managing diabetes mellitus. 2003.]

The FDA (1998) guidance to glucose monitoring devices  sets the goal of 95% of results should fall within 20 mg/dL of the true value < 75 mg/dL, and within 20% of the true value for values  > 75 mg/dL. [Center for Devices and Radiological Health, US Food and Drug Administration. Draft guidance document: Review criteria for assessment of blood glucose monitoring in vitro diagnostic devices using glucose oxidase, dehydrogenase or hexokinase methodology, February 28, 1997. http://www.fda.gov ]

The American Diabetes Association, in 1987, recommended that glucose meter results should fall within 15% of reference systems. [Consensus Development Panel: Consensus statement on self-monitoring of blood glucose. Diabetes Care 1987; 10:95-99.] In 1996, the ADA reduced the goal to 5%, but no one has expected meters to meet this standard.

Finally, in the most recent paper on the subject, Karon, Klee and Boyd used simulation studies based on a databse of patient data results to model the appropriate standard. Their study reached the conclusion that 95% of the glucose meter results should fall within 15% of the true value. [Glucose Meter Performance Criteria for Tight Glycemic Control Estimated by Simulation Modeling, Brad S Karon, James C. Boy, and George G. Klee, Clinical Chemistry 2010;56:7; 1091-1097] This recommendation is similar to the consensus that appears to be emerging from the FDA public hearing on blood glucose meters.

The assessment

The study already assessed whether or not the 27 BGMs met the standards of the Clarke Error Grid and the ISO 15197 standard. In this article, we're attempting to determine if these BGMs will meet the tighter 15% standard.

On the low end, there is some difficulty in exactly determining this fact, since 15% of a result below 75 mg/dL is 11.25. The study only tracked whether or not results fell within 15 or 10 mg/dL. In this case, we accepted that if results fell within 15 mg/dL, they were meeting the 15% guideline. Thus, our assessment may be too lenient.

Below is a summary table of the 27 BGMs and whether or not they met the criteria of the different quality standards.

Conclusion

The study authors offered this conclusion:

"In summary, we found more than 40% of BG monitoring systems with a CE label that did not meet the minimal accuracy requirements of the standard DIN EN ISO 15197. As inaccurate BG monitoring systems bear the risk of false treatment decisions by the diabetes patient and subsequent possible severe health injuary, manufacturers should regularly and effectively check the quality of BG meters and test strips."

It is interesting to note that the Clarke Error Grid is the most forgiving assessment tool. This probably shouldn't come as a surprise, since the grid was published more than 20 years ago, back during the days of more primitive devices.Nevertheless, those BGMs that failed the Clarke Error Grid also failed the ISO 15197 standard and tighter 15% standard.

The bigger finding is that there are a lot of BGMs on the market that do not provide enough accuracy. Furthermore, the CE label is no absolute guarantee of quality for BGMs now or in the future. Even with a more lenient judgment on the lower end of the range, only nine BGMs have the potential to meet a tighter 15% quality requirement. If we are trying to improve the accuracy and usefulness of BGMs, it appears we will need serious improvements over the current generation of instrumentation.