Tools, Technologies and Training for Healthcare Laboratories

Field evaluation of POC lipid and glucose analyzers

A recent online publication combined a laboratory and field evaluation of two POC testing devices for lipids and glucose. Are we seeing good POC performance in real-world settings?

Laboratory and Field Evaluation of Two POC Lipid and Glucose devices

DECEMBER 2013
Sten Westgard, MS

[Note: This QC application is an extension of the lesson From Method Validation to Six Sigma: Translating Method Performance Claims into Sigma Metrics. This article assumes that you have read that lesson first, and that you are also familiar with the concepts of QC Design, Method Validation, and Six Sigma. If you aren't, follow the link provided.]

This analysis looks at a paper from a 2013 online-first publication of the Annals of Clinical Biochemisrty which examined the performance of two point-of-care (POC) lipid and glucose devices:

A combined laboratory and field evaluation of the [name withheld] and [name withheld] point-of-care testing lipid and glucose analyzers, Simon J Whitehead, Clare Ford, and Rousseau Gama, Annals of Clinical Biochemistry, online first published July 23, 2013.

This study was conducted at the Wolverhampton City Primary Care Trust in the United Kingdom. Rather than focus on the name of the POC instruments, we're going to focus on the POC performance. The devices could test for glucose, total cholesterol (TC), and high-density lipoprotein cholesterol (HDL-C).

The Imprecision and Bias Data

The imprecision data used in the study was collected thus:

"Inter-batch imprecision (n=12) was assessed (9am and 5 pm daily for six consecutive days) using (i) lithium heparinized (Li-Hep) venous whole blood samples (TC and HDL-C only) and (ii) venous whole blood samples (TC,HDL-C and glucose). Lithuim-Heparin venous blood was collected...and either stored as whole blood at 4 C for the duration of the experiment or immediately centrifuged  (2380 g for 5 min) and the separated plasma divided into 200 ul aliquots and stored at -80 C until used.""

As for bias, the study compared the POC devices against the instrument used in the central core laboratory, a Roche Modular:

"Accuracy of the POCT, TC, HDL-C and glucose assays was estimated by comparison with Roche laboratory methods which served as a reference....Paired POCT and laboratory measurements were collected from 273 subjects ([POC A]: n=167; [POC B]: n=106). Mean laboratory TC, HDL-C and glucose measurements for the patient populations used to evaluated the [POC A] and [POC B] analysers differed by <10%....Bland-Altman and Deming regression plots [were completed]."

The nice part of this study is that it takes 2 POC devices and compares them against the central laboratory. Thus, the laboratory has a choice about which device to choose - which one is a better fit with the laboratory performance.

Below is the table of imprecision and bias for the three levels of each analyte:

POC Method A: Li Hep Plasma
Level
CV% Slope Y-int
Bias %
Total Cholesterol (TC)
mmol/L
4.2 4.9% 1.1 -1.46 24.76%
6.8 5.3%       11.47%
HDL-C, mmol/L 1.1 8.7% 1.01 -0.18 15.36%
1.7 10.3%     9.59%
Glucose 5.7 4.0% 1.38 -1.97 3.44%
11.9 3.9%     21.45%
POC Method A:
Li Hep Whole Blood
Level
CV% Slope Y-int
Bias %
Total Cholesterol (TC)
mmol/L
3.4 6.4% 1.10  -1.46 32.94%
5.7 6.7%        15.61%
HDL-C, mmol/L 0.9 7.8% 1.01  -0.18 19.0%
1.5 6.4%     11.0%

 That's the first POC device. Below is the performance of the second POC device:

POC Method B: Li Hep Plasma
Level
CV% Slope Y-int
Bias %
Total Cholesterol (TC)
mmol/L
4.0 3.3% 1.02 -0.2 3.0%
7.3 2.4%       0.74%
HDL-C, mmol/L 1.2 2.7% 1.0 -0.13 10.83%
1.7 2.5%     7.65%
Glucose 5.9 2.1% 1.6 -2.34 20.34%
11.7 3.7%    40.0%
POC Method B:
Li Hep Whole Blood
Level
CV% Slope Y-int
Bias %
Total Cholesterol (TC)
mmol/L
4.5 4.1% 1.02  -0.2 2.44%
7.6 2.4%       0.63%
HDL-C, mmol/L 1.3 4.9% 1.0  -0.13 10.0%
1.9 5.2%    6.84%

As usual, we have a lot of numbers We have two levels of controls, so we have imprecision estimates for both of those levels. Using the Deming regression equation, we can estimate the bias at each of those levels. Then we also have the two sample types, plasma and whole blood ["The inter-batch imprecision of the POCT glucose assays could not be assessed using whole blood due to the instability of the analyte in that matrix."]

Looking at the raw numbers, you may find it difficult to judge the method performance. From experience, you might be able to tell when a particular method CV is high or low. But the numbers themselves don't tell the story.

If we want an objective assessment, we need to set analytical goals - specify quality requirements - and use those to calculate the Sigma-metric. 

Determine Quality Requirements

Now that we have our imprecision and bias data, we're almost ready to calculate our Sigma-metrics. But we're missing one key thing: the analytical quality requirements.

In the US, traditionally labs look first to CLIA for guidance. While this study was not conducted in the US, it may be advisable to use these goals since we're working at the point-of-care, where more error is expected, and indeed tolerated. For example, in the US, CLIA holds glucose at the core laboratory to a tighter standard than at the point-of-care. Indeed, for much of the glucose range, the core lab method is expected to meet a 10% quality requirement, while the point-of-care blood glucose meters need only meet a 20% quality requirement. Given that almost all other soruces of quality requirements have more stringent goals, the minimum achievement of these POC devices should be able to meet the CLIA specifications for total allowable error (TEa).

Method CLIA Goal
HDL-C ± 30%
Total Cholesterol (TC) ± 10%
Glucose

Target value ± 6 mg/dL
or ± 10% (greater)
(for Core Lab methods)

20%

(for POC devices)

Calculate Sigma metrics

Now the pieces are in place. Remember the equation for Sigma metric is (TEa - bias) / CV.

Example calculation: for Total cholesterol at the level 4.2 mmol/L, the CLIA goal is 10%. We also know from the study that at that level, the POC A device has 4.9% imprecision and a 24.76% bias:

(10 - 24.76) / 4.9 = - 14.76 / 4.9 = negative

So the POC at the lower level is NOT delivering excellent performance. The bias, in fact, exceeds the allowable total error. This means that the comparative method, the Roche Modular P, and the POC A method are simply not in agreement. They will produce results that may have clinically significant differences between them.

Recall that in industries outside healthcare, 3.0 Sigma is the minimum performance for routine use. 6.0 Sigma and higher is considered world class performance.We'll simplify the table below and calculate all the Sigma-metrics.

 

POC Method A: Li Hep Plasma
Level
CV% Bias %
Quality Requirement
(TEa%)
Sigma-metric
Total Cholesterol (TC)
mmol/L
4.2 4.9% 24.76% 10% negative
6.8 5.3% 11.47% negative
HDL-C, mmol/L 1.1 8.7% 15.36%  30%   1.68
1.7 10.3% 9.59%  1.98
Glucose 5.7 4.0% 3.44% 20%   4.14
11.9 3.9% 21.45%  negative
POC Method A:
Li Hep Whole Blood
Level
CV% Bias %

Quality Requirement
(TEa%)
 Sigma-metric
Total Cholesterol (TC)
mmol/L
3.4 6.4% 32.94%  10%   negative
5.7 6.7%  15.61%  negative
HDL-C, mmol/L 0.9 7.8% 19.0%  30%   1.41
1.5 6.4%  11.0%  2.97

 That's the first POC device. What about the second device?

POC Method B: Li Hep Plasma
Level
CV% Bias %

Quality Requirement
(TEa%)
 Sigma-metric
Total Cholesterol (TC)
mmol/L
4.0 3.3% 3.0% 10%  2.12
7.3 2.4% 0.74%  3.86
HDL-C, mmol/L 1.2 2.7% 10.83% 30%  7.1
1.7 2.5%  7.65%  8.9
Glucose 5.9 2.1% 20.34% 20%  negative
11.7 3.7% 40.0%  negative
POC Method B:
Li Hep Whole Blood
Level
CV% Bias %

Quality Requirement
(TEa%)
 Sigma-metric
Total Cholesterol (TC)
mmol/L
4.5 4.1% 2.44% 10%  1.84
7.6 2.4% 0.63%  3.9
HDL-C, mmol/L 1.3 4.9% 10.0% 30%  4.08
1.9 5.2% 6.84%  4.45

Yes, there are still a lot of numbers here, and many of them are not good.

For many of the assays on POC A, we've blown our error budget. In other words, when the two methods are that discordant (Sigma is below 0), we can't really benchmark the performance. Unless the POC and core lab are reconciled (possibly recalibrated), the bias is so large that the results of the two methods are just too different to be useful. Indeed, the discrepancy is more likely to generate confusion, delay (while more tests are performed to determine which result is "real"), and in the worst case, incorrect diagnoses and decisions.

Of the two devices, POC B has higher Sigma-metrics, and even has a bright spot - the HDL-C method performs at better than 6 Sigma on plasma samples.

Summary of Performance by Sigma-metrics Method Decision Chart

If the numbers are too much to digest, we can put this into a graphic format with a Six Sigma Method Decision Chart. Here's the chart for CLIA specifications for allowable total error. And at this point, we're going to reveal the names of the devices.

 Here's the first POC device:

2013-POCA-Sigma-Normalized-MEDX

Here's the second POC device:

2013-POCB-Sigma-Normalized-MEDX

Here's where the graphic display helps reveal issues with performance. You can see a big difference in the bias and precision of the assays of the two POC devices.

You can see that the HDL-C method for Cholestech LDX is in the "bull's-eye" but that other methods don't fare so well.

Now the question becomes, what would the laboratory do if this instrument was in routine operation? What QC would be necessary to assure the level of quality required for use of the tests? In this case, we usually plot the method data on an OPSpecs (Operating Specifications) chart. However, since both of these devices are at the point-of-care, there is very little flexbility in the QC that can be done on the devices. While Sigma-metrics might dictate that POC A use full "Westgard Rules" and multiple controls, the on-board QC may be much less. Furthermore, both the CardioChek is a waived device, which means that all customers need to do is follow the manufacturer's advice on performing QC. Even if the manufacturer provides inadequate QC recommendations, the customer can follow them and be in compliance.

In other words, if you already have this device in routine operation, it's a bit late to worry about QC. The devices aren't capable of running enough QC to compensate for the performance.

Conclusion

The study authors concluded the following:

"[I]n order to make POCT a suitable choice for delivery of the NHS Health Checks, it is essential that  the analytical performance of the POCT analyser is fit for purpose and both the laboratory and users are aware of pre-analytical factors and governance issues. Cost is an additional key consideration when planning a POCT based NHS Health Checks service. The CardioChek is significantly cheaper than the LDX in terms of both capital and running costs and this may be an important factor in procurement decision making."

I believe that is a rather optimistic conclusion given the analytical performance. The LDX shows significantly better analytical performance than the CardioChek. According to CLIA quality requirements, using a Six Sigma standard, the fitness for purpose of these POC devices is in question. Certainly, these methods have gotten their CE, FDA approval, and CLIA waived status, but those regulatory bars may have been set too low. There are good POC methods and devices out there, but the performance demonstrated by these devices is concerning.

847.714.3904 [Cell]