Tools, Technologies and Training for Healthcare Laboratories

Three POC Glucose Devices for Neonatal ICU

As part of our continuing series of evaluations, we take a look at three point-of-care (POC) blood glucose analyzers. We use a paper that evaluates them for use in a specific critical setting: a neonatal intensive care unit. 

OCTOBER 2013
Sten Westgard, MS

[Note: This QC application is an extension of the lesson From Method Validation to Six Sigma: Translating Method Performance Claims into Sigma Metrics. This article assumes that you have read that lesson first, and that you are also familiar with the concepts of QC Design, Method Validation, and Six Sigma. If you aren't, follow the link provided.]

Our last look at POC devices that measure glucose was a few years ago. While we've covered reviews of multiple blood glucose monitors (BGMs), we came across a recent study that was focused on the use of POC BGMs in a neonatal intensive care unit (NICU):

Prospective evaluation of three point of care devices for glycemia measurement in a neonatal intensive care unit. Corinne Stadelmann Diaw, Nicolas Piol, Jocelyne Urfer, Dominique Werner, Matthias Roth-Kleiner, Clinica Chimica Acta 425 (2013): 104-16. 

The Precision and Comparison Data

"Two within-run imprecision tests were conducted, one in an aqueous and the second in a whole blood matrix. As aqueous matrices, the control solutions (low and high glucose concentration), provided by the manufacturers for their specific device were used. For the whole blood matrix, the following protocol was used: 25 ml of venous whole blood was drawn from a healthy volunteer donor and directly heparinized. The sample was kept three days at room temperature, leading to glycemia close to zero....The blood was then separated in 3 aliquots and spiked with different volumes of a concentrated glucose solution... in order to generate final glucose concentrations of 2.2 mmol/L, 3.3 mmol/L and 4.5 mmol/L, as assessed by the reference method....Twenty measurements werer performed with each device on each of the five test solutions (2 aqueous and 3 whole blood solutions), leading to n=60 for each company and test solution."

This is not an ideal imprecision study - collecting only within-run imprecision data usually results in unrealistic optimistic estimations of precision. But we will proceed with this analysis (you'll soon see why).

Instrument Level POC A
CV%
POC B
CV%
POC C
CV%
Aqueous Low 1.92 mmol/L (34.2 mg/dL) 9.2% 4.2% 3.6%
Aqueous High 14.33 mmol/L (258.2 mg/dL) 4.2% 3.9% 1.6%
Whole blood Low 2.2 mmol/L (39.6 mg/dL) 13.3% 5.3% 3.3%
Whole blood Middle 3.3 mmol/L (59.5 mg/dL) 16.1% 5.3% 3.3%
Whole blood High 4.10 mmol/L (81.1 mg/dL) 11.5% 3.9% 2.8%

Note that the current consensus appears to be around having methods with less than 3% CV. But POC devices are another story - typically their precision is not considered as important.

Accuracy (bias) was assessed against a gold gold standard hexokinase method, with requirements set forth in the ISO 15197 standard, both the 2003 version and the proposed revision 2011 version:

The 2003 ISO 15197 sets forth these criteria: ≥ 95% of the samples, the difference of glycemia between POCT and reference method should be within ± 0.8 mmol/L for values < 4.2 mmol/L and within ± 20% for values ≥ 4.2 mmol/L.

The 2011 ISO 15197 sets forth these criteria: ≥ 95% of the samples, the difference of glycemia between POCT and reference method should be within ± 0.9 mmol/L for values < 5.5 mmol/L and within ± 15% for values ≥ 5.5 mmol/L.

The study also calculated a "Bias of measured mean to reference value" for all the whole blood samples, which we can convert into a % bias.

Instrument Level POC A
CV%

POC A
Bias%

POC B
CV%
POC B
Bias %
POC C
CV%
POC C
Bias %
Whole blood Low 2.2 mmol/L (39.6 mg/dL) 13.3% 10.9% 5.3% 34.1% 3.3% 15.0%
Whole blood Middle 3.3 mmol/L (59.5 mg/dL) 16.1% 14.8% 5.3% 27.3% 3.3% 9.4%
Whole blood High 4.10 mmol/L (81.1 mg/dL) 11.5% 8.9% 3.9% 21.8% 2.8% 7.3%

If you're wondering if those numbers are concerning, you're right. That's a lot of bias between the POC devices and the reference method.

Now remember we have precision data for the whole blood matrices - so we can pair up the bias at those levels, but we won't be able to use the imprecision data from the aqeous controls, since there isn't any bias data to pair up with them:

 

Determine Quality Requirements at the decision level

Now that we have our imprecision and bias data, we're almost ready to calculate our Sigma-metrics. We're just missing one key thing: the analytical quality requirement.

For glucose, different organizations have set different quality goals.

Source
Quality Requirement
CLIA PT
Target value ± 6 mg/dL
  [± 0.333 mmol/L]
or 10%
(whichever is greater)
Rilibak (Germany)
Target value ± 15%
RCPA Target value ± 1.0 mmol or ± 10%
(whichever is greater)
Ricos et al. biologic database, desirable specification
5.5% (plasma)
6.9% (serum)

As you can see, even for a measurand that's been around for a very long time, there is not good agreement on how good quality should be.

What's even more interesting is that none of these quality requirements apply, since these are POC devices. For some reason, with POC glucose, an entirely different quality requirement is generally accepted. While a core laboratory glucose measurement in the US is judged by a CLIA standard of around 10% allowable total error, for POC devices, that allowable error doubles to 20%. In recent years, it has been pointed out that glucose meters are not even hitting this wider goal and many recommendations have been made to tighten the requirements to 15% allowable total error. But that has not been made official yet in the US, and even with the ISO 15197 standard, while the tighter goal of 15% has been proposed, this is not yet fully accepted.

Where does this leave us? We'll start with using the 20% allowable total error, one of the widest possible goals, just to see if the glucose meters can hit that target.

Calculate Sigma metrics

Now all the pieces are in place. Remember, this time we have three levels, so we're going to calculate three Sigma metrics.

Remember the equation for Sigma metric is (TEa - bias) / CV.

Example calculation: for a 20% quality requirement, at the level of 2.2 mmol/L (39.6 mg/dL), given method POC A's 13.3% imprecision, 10.9% bias:

(20 - 10.9) / 13.3 = 9.1 / 13.3 = 0.68

Instrument Level POC A
CV%

POC A
Bias%

POC A
Sigma

POC B
CV%
POC B
Bias %
POC B
Sigma
POC C
CV%
POC C
Bias %
POC C
Sigma
Whole blood
Low
2.2 mmol/L
(39.6 mg/dL)
13.3% 10.9% 0.68 5.3% 34.1% n/a 3.3% 15.0% 1.5
Whole blood
Middle
3.3 mmol/L
(59.5 mg/dL)
16.1% 14.8% 0.32 5.3% 27.3% n/a 3.3% 9.4% 3.2
Whole blood
High
4.10 mmol/L
(81.1 mg/dL)
11.5% 8.9% 0.96 3.9% 21.8% n/a 2.8% 7.3% 4.5

Recall that in industries outside healthcare, on the short-term scale, 3.0 Sigma is the minimum performance for routine use and 6.0 Sigma is considered world class quality. We're looking at the long-term scale for this Sigma-metric calculation, which is 1.5s higher (the short-term scale builds in a 1.5s shift, to allow for "normal process variation"). So we could go as low as 1.5 for the bare minimum acceptability. Still, what this is telling us is that we're not achieving even that level of performance when we apply the widest standard goal for POC glucose meters A and B. POC meter B has a bias that simply exceeds the quality requirement outright; basically, that device is just not getting any agreement with the answers coming out of the core laboratory. The one bright spot is POC meter C. At the middle and high levels, the Sigma's are in the acceptable range, even good.

Summary of Performance by Sigma-metrics Method Decision Chart and OPSpecs chart

Because we have different quality requirements, but we are still interested in assessing performance of the same method, we'll use normalized Method Decision Charts and Normalized OPSpecs charts.

3 POC glucose meters in NICU

Here we can see the 20% specification for total error has differentiated the performance of these three devices. Method A's main problem is that it has too much imprecision. Method B's main problem is that it has too much bias. Method C, in relative terms, is the most acceptable.

Using QC Design tools (such as EZ Rules 3 ), we can express the method performance on an OPSpecs (Operating Specifications) chart and determine what we would need to do for QC if we were to use any of these POC methods:

3 POC glucose meters in NICU - QC Design

What the OPSpecs chart tells us is that neither A nor B are controllable. We can't run enough controls to accomodate the amount of error present in the methods and still achieve acceptable error detection for a goal of 20%. The quality assurance capability simply isn't there. Even for C, which is a better method, we would need to consider using a full set of "Westgard Rules" or possibly 4 controls per run in order to control the middle and high levels - the lowest level is not possible for us to control.

What's worse is that since these are waived devices, there is no regulatory mandate to perform any increased amount of QC. As long as the laboratory follows the manufacturer's instructions for QC, the method is presumed to be in control and the results are assumed to be acceptable. Given the highly competitive marketplace, every manufacturer has incentive to recommend the least amount of QC and maintenance possible. So we have a situation where devices are performing poorly, but there is no mechanism to force the users to improve their QC practices in response.

The authors were equally frustrated with the performance of these devices:

"To our surprise, none of the three BGM fulfilled the ISO 15197:2003 accuracy criteria on the whole range of glycemia measurements. The situation was quite similar with the new proposed ISO-criteria 15197:2011." [ed. emphasis added]

Conclusion

The dissatisfaction with blood glucose meters has been accumulating in recent years. More and more labs and health systems are realizing that there are severe limitations to the accuracy and clinical utility of these near-patient devices. Given that the demands for the accuracy of these measurements are even higher for neonatal ICU's, a health system should think carefully about whether they want to trust a POC BGM measurement when it comes to an infant's health status. 

The authors' conclusion mirrors our findings:

"None of the three tested POCT devices reached the goal of current or proposed new ISO 15197 accuracy criteria over the whole range of glycemia values....Before being used in neonataology, we suggest that all new BGM, although validated in the adult population, need a specific and careful evaluation in newborn infants in order to assess their limitations. "