Tools, Technologies and Training for Healthcare Laboratories

Performance of a POC Chemistry Analyzer

Small point-of-care chemistry analyzers continue to enter the market. Is the latest model an acceptable substitute for core lab quality results?

Evaluation of a new POC chemistry analyzer

JULY 2014
Sten Westgard, MS

[Note: This QC application is an extension of the lesson From Method Validation to Six Sigma: Translating Method Performance Claims into Sigma Metrics. This article assumes that you have read that lesson first, and that you are also familiar with the concepts of QC Design, Method Validation, and Six Sigma. If you aren't, follow the link provided.]

This analysis looks at a 2013 study in the Journal of Laboratory Medicine and Quality Assurance which examined the performance of  new point-of-care (POC) chemistry analyzer:

Performance Evaluation of the [Name Withheld] Point-of-Care Chemistry Analyzer, Tae-Dong Jeong, Woochange Lee, Sail Chun, and Won-Ki Min, J Lab Med Qual Assur 2013;35:70-80.

The study is in Korean, however all the numbers are clearly stated in tables and graphs with English captions. We can extract the data and make Sigma-metric calculations.

The Imprecision and Bias Data

The imprecision data used in the study was collected following the CLSI EP5-A2 protocol and using Bio-Rad Liquid Assayed Multiqual Level 1 and 3 for the chemistries and Bio-Rad Diabetes level 1 and 2 for HbA1c.

As for bias, the study compared the POC device against the main chemistry instrument, a Roche Cobas 8000 for the chemistry tests, and against the Cobas Integra 800 for the HbA1c.Using the

Below is the table of total imprecision and bias calculated at each level for each analyte:

POC Chemistry Analyzer
Level
CV% Bias %
Cholesterol 125.58 2.0% 1.2%
322.41 3.1% 1.8%
Triglycerides 90.30 3.7% 1.1%
349.07 3.1% 3.2%
HDL 36.89 7.0% 2.9%
80.01 7.2% 2.5%
BUN 10.81 6.9% 4.3%
82.35 3.6% 1.1%
Creatinine 1.14 3.4% 3.3%
7.96 3.1% 2.5%
Amylase 31.09 5.6% 13.8%
255.36 1.9% 4.4%
HbA1c 5.84 4.2% 2.6%
10.55 3.8% 1.0%

As usual, we have a LOT of numbers. We have imprecision at two levels and calculated bias at each of those levels.

Looking at the raw numbers, you may find it difficult to judge the method performance. From experience, you might be able to tell when a particular method CV is high or low. But the numbers by themselves don't easily tell the story.

If we want an objective assessment, we need to set analytical goals - specify quality requirements - and use those to calculate the Sigma-metric. 

Determine Quality Requirements

Now that we have our imprecision and bias data, we're almost ready to calculate our Sigma-metrics. But we're missing one key thing: the analytical quality requirements.

In the US, traditionally labs look first to CLIA for guidance. While this study was not conducted in the US, it may be advisable to use these goals since we're working at the point-of-care, where more error is expected, and indeed tolerated. 

Analyte
Source Allowable Total Error (TEa) %
Cholesterol CLIA 10%
Triglycerides CLIA 25%
HDL CLIA 30%
BUN CLIA +/- 2 mg/dL or 9% (whichever is greater)
Creatinine CLIA +/- 0.3 mg/dL or 15% (whichever is greater)
Amylase CLIA 30%
HbA1c CAP/NGSP 6%

CLIA provides some of these goals in a dual format, with a units-based goal for the lower end of the range, and a %-based goal for the upper end of the range. Given that we are working with averages of imprecision and bias, we chose to use the %-based CLIA quality requirements for those analytes (glucose, creatinine).

Calculate Sigma metrics

Now the pieces are in place. Remember the equation for Sigma metric is (TEa - bias) / CV.

Example calculation: for this analyzer, the goal for cholesterol is 10%. We also know from the comparison study that there is a 1.2% bias and an imprecision of 2.0% at a level of 125.58 mg/dL:

(10 - 1.2) / 2.0 = 8.8 / 2.0 = 4.0

So the chemistry analyzer is delivering good performance for cholesterol at this level.

Recall that in industries outside healthcare, 3.0 Sigma is the minimum performance for routine use. 6.0 Sigma and higher is considered world class performance (short-term scale).

Now we'll calculate all the Sigma-metrics:

Cobas c111
Level
CV% Bias %
Sigma-metric
Cholesterol 125.58 2.0% 1.2%  4.4
322.41 3.1% 1.8%  2.7
Triglycerides 90.30 3.7% 1.1%  7.0
349.07 3.1% 3.2%  9.1
HDL 36.89 7.0% 2.9%  3.9
80.01 7.2% 2.5%  3.8
BUN 10.81 6.9% 4.3%  2.1
82.35 3.6% 1.1%  2.2
Creatinine 1.14 3.4% 3.3%  6.8
7.96 3.1% 2.5%  4.0
Amylase 31.09 5.6% 13.8%  2.9
255.36 1.9% 4.4%  13.5
HbA1c 5.84 4.2% 2.6%  0.8
10.55 3.8% 1.0%  1.3

Now we have even more numbers, and some of them are good, while others are not good at all. Even for some assays, the low end might be poor (Amylase) while the high end is fantastic.

Summary of Performance by Sigma-metrics Method Decision Chart

If the numbers are too much to digest, we can put this into a graphic format with a Six Sigma Method Decision Chart. Here's the normalized chart for the CLIA goals.

POC chemistry analyzer Method Decision Chart

Here's where the graphic display helps reveal issues with performance. You can see that the new device has one method completely in the bull's-eye (Triglycerides) and after that it's a mixed bag. Amylase is "half-in, half-out". Creatinine is achieving Excellent performance on average. Cholesterol and HDL are basically marginal, BUN is poor, and the HbA1c result is off the chart.

Now, how would you QC this device if you had it in your laboratory? Let's look at a Normalized OPSpecs chart:

2014 POC chemistry analyzer

For Triglycerides, we could use simple 3s limits and two controls. If we were only interested in the high end performance of Amylase, that same QC rule could be used. For Creatinine, we probably need some set of "Westgard rules". For Cholesterol and HDL, we need a very robust set of "Westgard Rules" that uses 8 to 10 total control measurements. Since it's unlikely that the POC analyzer is even built to have that capacity within a single run, this means we'll need to use data from the present run plus the data of several previous runs. Which means when we find an error it might indicate multiple previous runs were out of control. For BUN and HbA1c, even if we apply the maximum "Westgard Rules" and all the controls we can afford, that's still not going to cut it. If errors occur, it will take us a longer time to detect them (or they will need to get much larger before we're able to detect them) and consequently we can expect more defects and uncertainty in those lab results, which will be passed on to the clinician and patient.

Conclusion

The authors concluded that since the within-run and total-run coefficients of variation were below 10%, these methods were acceptable. That's obviously not the same standard we would apply in the core lab. But here's part of their conclusion:

"The [POC Chemistry Analyzer] showed suitable analytical performance with respect to precision and linearity and demonstrated a good correlation with automated chemistry analyzer. With the additional benefits of a short turnaround time and ease of use, the [POC Chemistry Analyzer] is an acceptable POCT chemistry analyzer."

As the Sigma-metric review indicates, there are a lot of imperfections with this device. Words like "suitable" and "acceptable" are vague and subjective. The calculations tell a different story. Most core lab chemistry tests can achieve a far better performance, so this POC device may deliver faster results, but they are not necessarily better ones.

For those thinking about the coming IQCP regulations in the US, where the use of some as-yet-undefined Risk Analysis process will allow laboratories to reduce the frequency or effort of QC for POC devices, this study should serve as another credible refutation. Perhaps for triglycerides performance is good enough to consider easing the QC effort, but many of these analytes need more QC, not less. The evidence simply doesn't support the idea that this instrument could reduce QC to once a week, once a month, or rely solely on electronic QC. The current performance is already too risky - diminishing the monitoring of these methods will only increase patient risk.

847.714.3904 [Cell]