Tools, Technologies and Training for Healthcare Laboratories

Evaluation of a blood glucose device

Glucose meters are coming under renewed scrutiny - off-label use with critcally ill patients, infection issues, and new draft guidelines for tighter requirements for FDA clearance. We take a look at a current glucose meter to see if current performance is meeting today's goals - and possibly tomorrow's...

Evaluation of a blood glucose self-monitoring system

MARCH 2014
Sten Westgard, MS

[Note: This QC application is an extension of the lesson From Method Validation to Six Sigma: Translating Method Performance Claims into Sigma Metrics. This article assumes that you have read that lesson first, and that you are also familiar with the concepts of QC Design, Method Validation, and Six Sigma. If you aren't, follow the link provided.]

This Sigma analysis looks at a number of papers from 2011 to 2013 which examined the performance of a point-of-care (POC) glucose device:

Evaluation of OneTouch Verio, a new blood glucose self-monitoring system for patients with diabetes. Littmannn K, Petersen ERB, Pussinen et al. Scan J Clin and Lab Inves 2013;73:286-292. (PDF)

SKUP analysis report OneTouch Verio, http://skup.nu/GetFile.ashx?fileid=474 SKUP/2011/86

A Comprehensive Evaluation of the Performance of the Test Strip Technology for OneTouch Verio Glucose Meter Systems, Bailey T, Chang A, Rosenblit PD et al, Diabetes Technology and Therapeutics 2012 Volume 14:8;701-709

The Imprecision and Bias Data

Below is the table of imprecision and bias for three levels:

OneTouch Verio
Level
CV% Bias %
Glucose <7 mmol/L (<126.1 mg/dL) 2.3% 4.5%
7 - 10 mmol/L (126.1 - 180.2 mg/dL 3.1% 2.9%
>10 mmol/L (> 180.2 mg/dL) 3.5% 0.1%

Looking at the raw numbers, you may find it difficult to judge the method performance. From experience, you might be able to tell when a particular CV is high or low. But the numbers themselves don't tell the story.

If we want an objective assessment, we need to set analytical goals - specify quality requirements - and use those to calculate the Sigma-metric. 

Determine Quality Requirements

Now that we have our imprecision and bias data, we're almost ready to calculate our Sigma-metrics. But we're missing one key thing: the analytical quality requirements.

In the US, traditionally labs look first to CLIA for guidance. But for glucose, there are two standards that are acceptable. In the US, CLIA holds glucose at the core laboratory to a tighter standard than at the point-of-care. Indeed, for much of the glucose range, the core lab methods are expected to meet basically a 10% quality requirement, while point-of-care blood glucose meters need only meet a 20% quality requirement. Given that almost all other soruces of quality requirements have more stringent goals, the minimum achievement of these POC devices should be able to meet the CLIA specifications for total allowable error (TEa).

However, the FDA issued some new draft guidelines for glucose meters in early 2014, and it's of interest to determine whether or not this meter would pass these possible future guidelines. Since the device is already on the market, it won't be necessary to meet this standards, but as a laboratory user, it is good to know if the device you are using will meet future standards or if you will want to replace it with something that meets those standards in the future.

Method CLIA Goal
Glucose

Target value ± 6 mg/dL
or ± 10% (greater)
(for Core Lab methods)

20%

(for POC devices)

FDA Draft Guidance for glucose meter clearance requirements:

  • 99% of all values fall within ± 10% of the reference method for glucose concentrations > 70 mg/dL (3.88 mmol/L) and within ± 7 mg/dL for glucose concentrations < 70 mg/dL
  • No individual result should exceed ± 20% of the reference method > 70 mg/dL and outside ± 15 mg/dL for glucose concentrations < 70 mg/dL.

Calculate Sigma metrics

Now the pieces are in place. Remember the equation for Sigma metric is (TEa - bias) / CV.

Example calculation: for glucose at the lower level range below 7.0 mmol/L, the current POC goal is 20%. We also know from the study that at that level, the Verio has 2.3% imprecision and a 4.5% bias:

(20 - 4.5) / 2.3 =  15.5 / 2.3 = 6.7 Sigma

If we were to apply the CLIA 10% goal:

(10 - 4.5) / 2.3 =  5.5 / 2.3 = 2.39 Sigma

If we were to apply the FDA draft guidance:

We would need the meter to hit 3.9 Sigma at the CLIA goal of 10% and above 6 Sigma for the POC goal of 20% at all levels

ll that in industries outside healthcare, 3.0 Sigma is the minimum performance for routine use. 6.0 Sigma and higher is considered world class performance.We'll simplify the table below and calculate all the Sigma-metrics.

OneTouch Verio
Level
CV% Bias %
POC
Sigma
CLIA Core Lab
Sigma
2014
FDA Draft

Glucose <7 mmol/L (<126.1 mg/dL) 2.3% 4.5% 6.7 2.49 Fail
7 - 10 mmol/L (126.1 - 180.2 mg/dL 3.1% 2.9% 5.5 2.3 Fail
>10 mmol/L (> 180.2 mg/dL) 3.5% 0.1% 5.7 2.8 Fail

While under current POC standards, this meter is more than acceptable, it looks like it would not be cleared if the new FDA draft guidelines were imposed.

In a separate study, It was found that for healthcare professionals, below 75 mg/dL (or 4.2 mmol/L), only 93.5% of results fell within 10 mg/dL of the reference method result, but above 75 mg/dL, 100% of results fell within 20% of the reference method. This matches the table of data that we see here - the method is better at achieving success at a 20% target than it is for 10% target..

Summary of Performance by Sigma-metrics Method Decision Chart

If the numbers are too much to digest, we can put this into a graphic format with a Six Sigma Method Decision Chart. We'll use a normalized OPSpecs chart here so we can show both the performance meeting the 10% goal and the 20% goal.

 Normalized Method Decision Chart, OneTouch Verio[tm]

So, if we simply judge this using today's 20% POC glucose goal, this is an excellent to world class method. If we want to judge it against the same standards as a core laboratory method, it's not acceptable. If we want it to meet the possible future FDA quality goals, it probably won't pass.

If we had this method today, what would we do for QC? In this case, we plot the method data on an OPSpecs (Operating Specifications) chart. However, keep in mind that since this is a POC device, there might be little flexbility in the QC that can be done on the device.

Normalized OPSpecs Chart, OneTouch Verio[tm]

If we're simply using today's POC goal of 20% allowable total error, we can use 3s limits and run 3 controls. If we're going to hold this method to "lab quality" standards, we're goingto need possibly double the number of controls and use all the "Westgard Rules" we can muster.

Conclusion

The studies conclude that this method is acceptable, and under current guidelines, it is. In fact, compared to many other glucose meters on the market right now, this has superior performance. But the possible future FDA requirements will be more stringent, and if this device was to be submitted under those future targets, it might not gain clearance.