Tools, Technologies and Training for Healthcare Laboratories

Glucose Goals

 Continuing in our series of performance and goal comparisons, we evaluate the quality requirements and method performance of glucose.

Glucose Quality: What's the right Goal? What's the actual performance?

Sten Westgard
MARCH 2014

In 2013, we took a look at a number of electrolyte and basic chemistry analytes, like chloride, calcium, sodium, potassium, and albumin and total protein to make assessments of which goals were achievable, and which methods were performing well. This month we're turning to a different analyte: glucose.

What's the Right Goal for Glucose?

On a more specific level, even for laboratories that are attempting to assess and assure quality, the question of goals, targets and requirements are a challenge. The main question for labs is, "What is the Right Goal for this Test?"  While the 1999 Stockholm hierarchy identifies an approach, the ideal quality goals - goals developed based on the evidence of how the test results are actually used by clinicians - are rare indeed. Even for well-studied assays that have been used in medicine for decades, there is still a wide disparity on the actual goal that should be used for quality.

Analyte Acceptance criteria / quality requirements
CLIA Desirable
Biologic
Goal
RCPA Rilibak

Spanish
Minimum
Consensus

GLUCOSE ± 10% ± 6.9% ± 0.4 mmol/L ≤ 5.0 mmol/L;
± 8% > 5.0 mmol/L;
 ± 15.0% ± 11%

This is a case where most of the goals are very similar: between 6.9% to 15%, with a median of around 10%. No so coincidentally, the new FDA draft guidance is asking manufacturers of glucose meters to achieve the following goals:

FDA Draft Guidance for glucose meter clearance requirements:

  • 99% of all values fall within ± 10% of the refernce method for glucose concentrations > 70 mg/dL and within ± 7 mg/dL for glucose concentrations < 70 mg/dL
  • No individual result should exceed 20% of the reference method > 70 mg/dL and outside 15 mg/dL for glucose concentrations < 70 mg/dL.

It's notable to see that these glucose meter specifications are beginning to converge with the core laboratory specifications. A few years ago, the 20% goal was the only one that was mentioned with glucose meters. Now that number is beginning to narrow. Also, notice that the glucose meter specifications identify not only a goal, but an acceptable success rate (either 99% or 100% success), which can then be turned into a target Sigma-metric. If you want to achieve 99% success on the Sigma scale, you want to have performance of 3.9 Sigma or higher. If you specify 100% as the success rate, that's higher than Six Sigma.

The new FDA Draft guidelines are similar in approach to the specifications set down in the ISO 15197:2003 and 15197:2013 criteria:

  • (2003 specifications) 95% of all values fall within ± 20% of the reference method for glucose concentrations > 75 mg/dL, and within ± 15 mg/dL for glucose concentrations below 75 mg/dL
  • (2013 specification) 95% of all values fall within ± 15% of the reference method for glucose concentrations > 100 mg/dL, and within ± 15 mg/dL for glucose concentrations below 100 mg/dL

You can see that the ISO 15197 requirements are more forgiving, particularly at the lower end of the range (an allowable error that is double that of the FDA recommendation), and with a lower success rate (only 3.1 Sigma)

Where can we find data on method performance?

Here is our standard caveat for data comparisons: it's really hard to get a good cross-instrument comparison study. While every lab has data on their own performance, getting data about other instruments is tricky. If we want true apples-to-apples comparisons of performance, moreover, we need to find a study that evaluates different instruments within the same laboratory under similar conditions. Frankly, those studies are few and far between. Not many labs have the time or the space to conduct head-to-head studies of different methods and analyzers. I've seen a lot of studies, but few gold-standard comparisons.

Proficiency Testing surveys and External Quality Assessment Schemes can provide us with a broad view, but the data from those reports is pooled together. Thus, we don't get individual laboratory standard deviations, we get instead the all-group SD or perhaps the method-specific group SD. As individual laboratory SDs are usually smaller than group SDs, the utility of PT and EQA reports is limited. If we want a better idea of what an individual laboratory will achieve with a particular method, it would be good to work with data from studies of individual laboratories.

In the imperfect world we live in, we are more likely to find individual studies of instrument and method performance. A lab that evaluates one instrument and its methods, compares it to a "local reference method" (i.e. usually the instrument it's going to replace). While we can get Sigma-metrics out of such studies, their comparability is not quite as solid. It's more of a "green-apple-to-red-apple" comparison.

Given those limitations, here are a few studies of major analyzer performance that we've been able to find in recent years:

  • Evaluation des performances analytiques du systeme Unicel DXC 600 (Beckman Coulter) et etude de la transferabilite des resultats avec l’Integra 800 (Roche diagnostics), A. Servonnet, H. Thefenne, A. Boukhira, P. Vest, C. Renard. Ann Biol Clin 2007: 65(5): 555-62
  • Validation of methods performance for routine biochemistry analytes at Cobas 6000 series module c501, Vesna Supak Smolcic, Lidija Bilic-Zulle, elizabeta Fisic, Biochemia Medica 2011;21(2):182-190
  • Analytical performance evaluation of the Cobas 6000 analyzer – special emphasis on trueness verification. Adriaan J. van Gammeren, Nelley van Gool, Monique JM de Groot, Christa M Cobbeart. Clin Chem Lab Med 2008;46(6):863-871.
  • Analytical Performance Specifications: Relating Laboratory Performance to Quality Required for Intended Clinical Use. [cobas 8000 example evaluated] Daniel A. Dalenberg, Patricia G. Schryver, George G Klee. Clin Lab Med 33 (2013) 55-73.
  • The importance of having a flexible scope ISO 15189 accreditation and quality specifications based on biological variation – the case of validation of the biochemistry analyzer Dimension Vista, Pilar Fernandez-Calle, Sandra Pelaz, Paloma Oliver, Maria Josa Alcaide, Ruben Gomez-Rioja, Antonion Buno, Jose Manuel Iturzaeta, Biochemia Medica 2013;23(1):83-9.
  • External Evaluation of the Dimension Vista 1500 Intelligent Lab System, Arnaud Bruneel, Monique Dehoux, Anne Barnier, Anne Bouten, Journal of Clinical Laboratory Analysis 2012;23:384-397.
  • Evaluation of the Vitros 5600 Integrated System in a Medical Laboratory, Baum H, Bauer I, Hartmann C et al, poster PDF provided at Ortho-Clinical Diagnostics website. Accessed December 10th, 2013.
  • Evaluation of the VITROS 5600 Integrated System - Validation and Comparison Studies. Chen LS, Sakpal M, Kwong T. poster PDF provided at Ortho-Clinical Diagnostics website. Accessed March 23, 2014.
  • Sigma metrics used to access analytical quality of clinical chemistry assays: importance of the allowable total error (TEa) target. Hens K, Berth M, Armbruster D, Westgard S. Clin Chem Lab Med 2014 (in press)

These studies in turn can give us multiple data points (for instance, CV for high and low controls) for some instruments and we can attempt to evaluate these instruments' ability to achieve different quality requirements.

Can any instrument hit the Desirable Biologic TEa Goal for Glucose?

Here is a Method Decision Chart using the "Ricos Goal" of 6.9%.

[If you're not familiar with Method Decision Charts and Six Sigma, follow the links here to brush up on the subject.]

 Glucose performance based on Desirable Biologic Goal

As you can see, this is a quality requirement that not many methods can hit in the bull's-eye. No method can hit it consistently. This Ricos goal might be better suited for research and design of the next generation of glucose methods. It doesn't look like today's methods can achieve this standard.

Can any instrument hit the CLIA TEa goal for glucose?

Here is a method decision chart for a goal of 10% total allowable error.

 Comparison of glucose performance, CLIA goal

This goal is wider, and now some methods are hitting the bull's-eye. Those methods that couldn't hit the Ricos goal are still challenged to hit the CLIA goal. Recall that the FDA draft guidance for glucose meters is that 99% of results fall within 10% of the reference method value, or about 3.9 Sigma. So everything in the "good" zone and better meets that FDA guidance. Since we assume core laboratory methods should perform better than glucose meters, it's reassuring to see that this is in fact true.

Can any instrument hit the Rilibak interlaboratory comparison goal for glucose?

Here's a Method Decision chart for the 15% Rilibak goal:

Glucose comparison of performance, Rilibaek goal

Since the Rilibak goal is the most generous, it's not surprising that nearly all methods can hit this target in the bull's eye.

Can core laboratory methods get 100% of their results within 20% of the reference or true value?

Finally, here is a method decision chart for 20% allowable total error:

Glucose performance comparison, 20% FDA draft goal

As you can see, nearly every method hits the center of the target here. Given that the FDA wants 100% of results to fall within 20% of the reference method value, again it's reassuring to see that core laboratory methods are already where the FDA wants glucose meters to eventually be.

Now, what about glucose meters? Can current meters hit the FDA draft goals?

Looking at a recent evaluation of 12 glucose meters, we can try to predict if they would pass the proposed FDA draft quality requirements. We have to make some compromises to our criteria, since the study was looking at whether or not the meters passed the ISO 15197:2003 and/or ISO 15197:2013 requirements. As we saw earlier, neither of those requirements is exactly similar to what the FDA has proposed. However, the ISO 15197:2003 version is the closest, since it makes a distinction at glucose values above and below 75 mg/dL (very close to the FDA draft guidance focus on the level of 70 mg/dL). However, the allowable error is not nearly as close; ISO 15197:2003 allows 10 mg/dL difference whereas the FDA draft guidance proposes only 7 mg/dL. So the acceptability of the meters might be overstated by using that ISO 15197:2003 standard for the low end.

 

Meter % of results falling within
10 mg/dL below level 75 mg/dL
Meets 99% goal? % of results falling within
20% above 75 mg/dL
Meets 100% goal? Possible FDA
Verdict?
 Accu-Chek Aviva 97%  No  99% No Fail
 BGStar 100% Yes 99% No Fail
 Contour XT 100% Yes 100% Yes Pass
 GE100 100% Yes  100% Yes Pass
 GE200 100% Yes 99% No Fail
 GL40 95% No 98% No Fail
mylife Pura 98% No 99% No Fail
mylife Unio 100% Yes 100% Yes Pass
Omnitest 3 100% Yes 96% No Fail
OneTouch Verio Pro 74% No 99% No Fail

[ Note that there were two other meters that remained nameless because they didn't pass either of the ISO 15197 standards.]

So we can see that a majority of the meters studied here probably would not meet the new FDA draft guidance goals for performance, but 3 meters possibly could. The trick of it is that any model currently on the market won't need to submit itself to re-clearance. Meters currently on the market won't be forced to withdraw - the FDA draft guidance goals will only apply to new meters that are attempting to enter the market. This quick snapshot here indicates that some meters are already likely to meet the new criteria - so the goals suggested by the FDA are not out of reach. If some current meters can hit the FDA draft goals, the next generation of meters should be able to hit them, too.

Another point to be made is the comparison of meters and core lab methods. While the vast majority the core lab methods can hit the FDA draft goals, only a few of the meters can. There is still a difference between performance at the point-of-care and in the core laboratory. When meters claim "lab quality" results, the buyer should beware and assure that this is true, not assume that a glucose result from a meter is interchangeable with a glucose result from the core laboratory.

Conclusion

With quality requirements, there is no one source that is completely perfect and practical. Laboratories need to be careful when selecting quality goals, making sure that they choose the goals that they are mandated to meet (In the US, labs must be able to achieve CLIA targets), but also making sure that their goals are practical and in harmony with the actual clinical use of the test in their health system.

It appears here that the CLIA goal has the best ability to differentiate method performance. With the Ricos goal, everyone looks bad. With the Rilbaek goal, everyone looks good. The new FDA draft guidance appears to be tightening the goals so that meters perform more like core laboratory methods.

Remember, from our perspective, a goal that no one (no method) can hit is not useful, except perhaps to instrument researchers and design engineers (for them, this becomes an aspirational target, to design the next generation of methods to achieve that ultimate level of performance). Neither is it helpful to use a goal that every method can hit - that might only indicate we have lowered the bar too much, and actual clinical use could be tightened and improved to work on a finer scale.

If you liked this article, you might also like