Tools, Technologies and Training for Healthcare Laboratories

Evaluation of 3 Glucose monitoring systems

With all the controversy raging over the FDA Draft Guidelines for new glucose meters, let's remind ourselves of the performance that we see currently on the market. In this example, we'll evaluate three different glucose monitoring systems and see if they meet today's goals and possibly tomorrow's tighter goals...

MAY 2014
Sten Westgard, MS

[Note: This QC application is an extension of the lesson From Method Validation to Six Sigma: Translating Method Performance Claims into Sigma Metrics. This article assumes that you have read that lesson first, and that you are also familiar with the concepts of QC Design, Method Validation, and Six Sigma. If you aren't, follow the link provided.]

While the debate rages on about the latest proposed FDA draft guidelines for glucose meters, it's useful to remind ourselves about the actual state of glucose meter performance. In January 2014, a study was published that looked at three different blood glucose monitors (BGMs):

Evaluation of the Wisecheck Glucose Monitoring Systems. Seonhee Kwon, Ha Nui Kim, Chi Hyun Cho, Jang Su Kim, Chae Seung Lim, Lab Med Online January 2014 Vol 4(1): 15-21. 

While the title suggests this study was exclusively devoted to one glucose device, actually a total of three devices were evaluated: the ACCU-CHEK Performa and the SD GlucoLink were also evaluated in the same study.

The Precision and Comparison Data

Precision studies were carried out according to the EP15-A2 guideline using Bio-Rad Meter Trax Controls and Total imprecision was calculated from 20 measurements. The Comparison study was performed according to CLSI EP9-A2 guideline, comparing 150 patient samples against the Toshiba 200FR glucose method.

Instrument Level
CV%
Bias%
Wisecheck 50.5 mg/dL 2.98% 5.93%
117.55 mg/dL 2.41% 0.99%
268.25 mg/dL 1.88% 3.92%
ACCU-CHEK Performa 64.1 mg/dL 2.31% 3.42%
153.5 mg/dL 1.86% 1.25%
382.75 mg/dL 2.34% 3.26%
SD GlucoLink 58.75 mg/dL 3.98% 3.37%
132.15 mg/dL 4.31% 8.67%
313.1 mg/dL 3.32% 11.11%

Note that the current consensus appears to specify that glucose methods should have less than 3% CV. But POC devices are another story - typically their precision is not considered as important.

Determine Quality Requirements at the decision level

Now that we have our imprecision and bias data, we're almost ready to calculate our Sigma-metrics. We're just missing one key thing: the analytical quality requirement.

For glucose, different organizations have set different quality goals.

Source
Quality Requirement
CLIA PT
Target value ± 6 mg/dL
  [± 0.333 mmol/L]
or 10%
(whichever is greater)
Rilibak (Germany)
Target value ± 15%
RCPA Target value ± 1.0 mmol or ± 10%
(whichever is greater)
Ricos et al. biologic database, desirable specification
5.5% (plasma)
6.9% (serum)

Furthermore, there are additional quality goals specified by ISO:

  • The 2003 ISO 15197 sets forth these criteria: ≥ 95% of the samples, the difference of glycemia between POCT and reference method should be within ± 0.8 mmol/L for values < 4.2 mmol/L and within ± 20% for values ≥ 4.2 mmol/L.
  • The 2011 ISO 15197 sets forth these criteria: ≥ 95% of the samples, the difference of glycemia between POCT and reference method should be within ± 0.9 mmol/L for values < 5.5 mmol/L and within ± 15% for values ≥ 5.5 mmol/L.

Not to mention that the FDA's new draft guidelines propose still different goals:

  • That 99% of all values fall within ± 10% of the reference method for glucose concentrations > 70 mg/dL, and within ± 7 mg/dL at glucose concentrations <  70 mg/dL. Further, to avoid critical patient management errors, no individual result should exceed ± 20% of the reference method for samples >70 mg/dL or ± 15 mg/dL <70 mg/dL.(That is, 100% of all values fall within ± 20%

As you can see, even for a measurand that's been around for a very long time, we seem to be heading toward more disagreement, not less, on how good performance and quality should be.

What's even more interesting is that none of these quality requirements currently apply, since these are POC devices. For some reason, with POC glucose, an entirely different quality requirement is generally accepted. While a core laboratory glucose measurement in the US is judged by a CLIA standard of around 10% allowable total error, for POC devices, that allowable error doubles to 20%. The FDA draft guidance is only for manufacturers and will only apply to devices in the future.

In recent years, it has been pointed out that glucose meters are not even hitting the 20% goal and many recommendations have been made to tighten the requirements to 15% allowable total error. But that has not been made official yet in the US, particularly since the FDA draft guidelines seem to have raised a storm of opposition, and even with the ISO 15197 standard, while the tighter goal of 15% has been proposed, this is not yet fully accepted by the marketplace.

Where does this leave us? We'll start with using the 20% allowable total error, one of the widest possible goals, just to see if the glucose meters can hit that target, then we'll take a look at 15% and 10%.

Calculate Sigma metrics

Now all the pieces are in place. Remember, this time we have three levels, so we're going to calculate three Sigma metrics.

Remember the equation for Sigma metric is (TEa - bias) / CV.

Example calculation: for a 20% quality requirement, at the level of 50.5 mg/dL, given Wisecheck's 2.98% CV and 5.93% bias:

(20 - 5.93) / 2.98 = 14.07 / 2.98 = 4.72

Sigma-metric for 20% goal

Instrument Level CV% Bias%

Sigma-metric
for 20% goal

Sigma-metric
for 15% goal

Sigma-metric
for CLIA/10% goal

Wisecheck 50.5 mg/dL 2.98% 5.93%  4.72  3.04 2.00
117.55 mg/dL 2.41% 0.99%  7.89  5.81  3.74
268.25 mg/dL 1.88% 3.92%  8.55  5.90  3.24
ACCU-CHEK Performa 64.1 mg/dL 2.31% 3.42%  7.18  5.01 2.85
153.5 mg/dL 1.86% 1.25%  10.08  7.39  4.70
382.75 mg/dL 2.34% 3.26%  7.16  5.02  2.88
SD GlucoLink 58.75 mg/dL 3.98% 3.37%  4.18  2.92  1.72
132.15 mg/dL 4.31% 8.67%  2.63  1.47  0.31
313.1 mg/dL 3.32% 11.11%  2.68  1.17  negative

 

If we look at the current 20% goal, Wisecheck and ACCU-CHEK Performa have very good metrics. The ACCU-CHEK is performing better than six sigma across the whole range. But even at the widest goal, the SD GlucoLink doesn't perform as well. As the goals tighten down, however, the performance of each device is rated more harshly. At 15% allowable total error, the ACCU-CHEK is still excellent to world class, while the Wisecheck slips to just barely acceptable, and the SD GlucoLink falls into unacceptability. If we impose the same CLIA goal use in the core laboratory methods (approximately 10%), then none of these devices performs adequately. These devices are not giving us "lab quality" results.

Summary of Performance by Sigma-metrics Method Decision Chart and OPSpecs chart

Given so much data, it may be easier to visually assess the performance with Method Decision Charts. Because we have three different quality requirements, we'll graph the data on three different Method Decision Charts.

Wisecheck and 20% goal

Here we can see that with the 20% specification for total error, the ACCU-CHEK has all points in the bull's-eye, and the Wisecheck has almost all of them.

Wisecheck and 15% goal

With the 15% goal, performance begins to slip outside the bull's-eye for both Wisecheck and ACCU-CHEK, but they are still in acceptable zones of performance.

Wisecheck and the CLIA goal (around 10%)

When the laboratory standard of 10% allowable total error (actually this is the CLIA goal, which is les stringent below 60 mg/dL), none of the methods can hit the bull's-eye. We can't expect lab quality results out of these methods. What's most concerning is how far off the mark SD GlucoLink appears to perform. Not only is imprecision an issue, bias is a problem, too.

Usually in our Sigma-metric studies, we follow up with an OPSpecs chart to determine what QC will be necessary to properly monitor these devices. But with glucose monitoring devices, these are usually waived, so there is no regulatory mandate to perform any increased amount of QC, even if it's necessary. As long as the laboratory follows the manufacturer's instructions for QC, the method is presumed to be in control and the results are assumed to be acceptable. Given the highly competitive marketplace, every manufacturer has incentive to recommend the least amount of QC and maintenance possible. So we have a situation where devices are performing poorly, but there is no mechanism to force the users to improve their QC practices in response. That may explain why the FDA is tightening up guidelines for future devices, so this type of performance won't be a problem for laboratories anymore.

Conclusion

The authors' conclusion is probably reflective of the current 20% standard for glucose monitoring devices:

"The Wisecheck glucometer showed good precision...and correlation with the reference method. It provided rapid and reliable measurements of blood glucose levels and seemed appropriate for use in diabetes management. "

Our conclusion might differ: While the Wisecheck is basically adequate if we only impose the most lenient goal for quality, this would not be a device we would choose if tighter quality requirements were imposed. Indeed, from the comparison, of the three devices, it looks like ACCU-CHEK might be a wiser choice.