Tools, Technologies and Training for Healthcare Laboratories

Evaluation of 5 POC CRP tests

An analysis of 5 different point-of-care (POC) analyzers for C-Reactive Protein (CRP). Many years ago, assays were considered to be too imprecise to support their clinical use. Are the latest instruments any better?

Evaluation of five POC CRP methods

FEBRUARY 2013
Sten Westgard, MS

[Note: This QC application is an extension of the lesson From Method Validation to Six Sigma: Translating Method Performance Claims into Sigma Metrics. This article assumes that you have read that lesson first, and that you are also familiar with the concepts of QC Design, Method Validation, and Six Sigma. If you aren't, follow the link provided.]

This analysis looks at a paper from a 2013 publication of the Scandinavian Journal of Clinical & Laboratory Investigation which examined the performance of five point-of-care (POC) devices that measure C-reactive protein:

Analytical performance, agreement and user-friendliness of five C-reactive protein point-of-care tests, Minnaard MC, Van de Pol AC, Broekhuizen BDL et al, SJ Clin Lab Invest 2013;73:627-634.

The Imprecision and Bias Data

The imprecision data used in the study was collected thus:

"[A] protocol was used based on the clinical and laboratory standards institute EP5 and EP9 guidelines. CVs of each CRP assay were quantified testing 20 aliquots of the low (~20 mg/L) and high (~100 mg/L) level plasma pools."

As for bias, the study compared the POC devices against the instrument used in the central core laboratory, an Olympus AU 2700:

"From each of the eight serum pools with increasing concentrations 10 aliquots were tested to assess agreement between the POC test and the laboratory reference test using Bland-Altman plots."

Below is the table of imprecision and bias for the two levels of each instrument:

POC Method A: Li Hep Plasma
Level
CV% Bias %
POC Afinion 23 4.6% 1.3%
115 7.3% 12.78%
NycoCard Reader II 25 16.9% 12.0%
114 6.0%  9.39%
Eurolyser Smart 700/340 34 30.5% 22.94%
136 18.0% 0.66%
QuikRead go 25 8.3% 9.2%
106 4.0%  8.4%
QuikRead 101 19 9.3% 34.1%
92 6.3%  3.48%

As usual, we have a lot of numbers We have two levels of intrerest, so we have imprecision estimates for both of those levels. Using the Bland-Altman comparison, the study provided estimates of bias at both the low and high levels.

Looking at the raw numbers, you may find it difficult to judge the method performance. From experience, you might be able to tell when a particular method CV is high or low. But the numbers themselves don't tell the story.

If we want an objective assessment, we need to set analytical goals - specify quality requirements - and use those to calculate the Sigma-metric. 

Determine Quality Requirements

Now that we have our imprecision and bias data, we're almost ready to calculate our Sigma-metrics. But we're missing one key thing: the analytical quality requirements.

In the US, traditionally labs look first to CLIA for guidance. While this study was not conducted in the US, it may be advisable to use these goals since we're working at the point-of-care, where more error is expected, and indeed tolerated. For example, in the US, CLIA does provide any total allowable error guidance for CRP. If you look at the biological variation database (sometimes called the Ricos goals), that has a desirable total allowable error set at 56.6% One of the reasons this is rather high is because there is a lot of within-subject biologic variation for this analyte.

Calculate Sigma metrics

Now the pieces are in place. Remember the equation for Sigma metric is (TEa - bias) / CV.

Example calculation: for the Afinion at the level of 23 mg/L, the Ricos goal is 56.6%. We also know from the comparison study at that level, the Afinion has a 1.3% bias against the Olympus AU 2700 and an imprecision of 4.6%:

(56.6 - 1.3) / 4.6 = 55.3 / 4.6 = 12.02

So the Afinion at the lower level is delivering world class performance.

Recall that in industries outside healthcare, 3.0 Sigma is the minimum performance for routine use. 6.0 Sigma and higher is considered world class performance.We'll simplify the table below and calculate all the Sigma-metrics.

POC Method A: Li Hep Plasma
Level
CV% Bias %
Sigma-metric
POC Afinion 23 4.6% 1.3% 12.02
115 7.3% 12.78% 6.0
NycoCard Reader II 25 16.9% 12.0% 2.64
114 6.0%  9.39% 7.87
Eurolyser Smart 700/340 34 30.5% 22.94% 1.1
136 18.0% 0.66% 3.11
QuikRead go 25 8.3% 9.2% 5.71
106 4.0%  8.4% 12.05
QuikRead 101 19 9.3% 34.1% 2.41
92 6.3%  3.48% 8.43

Yes, there are still a lot of numbers here, and many of them are good.

For many of the insrtuments, the performance is good either at the low end or the high end, but not both. The problems are mostly due to excessive imprecision, but there are also some methods that have high bias.

Of the five devices, the Afinion has the best overall performance at both low and high levels.

Summary of Performance by Sigma-metrics Method Decision Chart

If the numbers are too much to digest, we can put this into a graphic format with a Six Sigma Method Decision Chart. Here's the chart for Ricos goal for desirable allowable total error.

2014-CRP-Comparison-Ricos-566MEDx

Here's where the graphic display helps reveal issues with performance. You can see that the Afinion is the only device with performance at both high and low level in the "bull's-eye."

Now the question becomes, what would the laboratory do if one of these instruments was in routine operation? What QC would be necessary to assure the level of quality required for use of the tests? In this case, we usually plot the method data on an OPSpecs (Operating Specifications) chart. However, since these devices are at the point-of-care, there may be little flexibility in the QC that can be performed. While Sigma-metrics might dictate that a particular method requires the full "Westgard Rules" and multiple controls, the on-board QC may be much less than that.

2014-CRP-Comparison-Ricos-566OPSpecs

With the Afinion and the QuikRead go, the use of the 1:3s control rule with just two controls would be sufficient for QC. The other instruments have at least one level that is in a zone that is going to require an extensive set of "Westgard Rules" with N of 4, double the typical amount. So for those instruments, it might cost twice as much QC effort to provide the appropriate quality assurance.

Conclusion

The study authors concluded the following:

"The results of this study show that there is considerable variation in agreement betwteen the different CRP POC test in comparison with a laboratory reference method, yet that most POC systems perform adequately."

I believe that is a rather optimistic conclusion given the analytical performance. While the Afinion and the QuikRead go do show world class performance, the verdict on the other instruments is mixed. Also of note is the stark contract between the core laboratory method (the Olympus AU2700) that had CVs of just 1.9% and 1.1% at low and high levels, respectively, and the POC methods. The considerable variation is a diplomatic way to say that some methods are better than others. Discerning laboratories and physicians would do well to pick out the best of this crop and avoid the more concerning instruments that have lesser performance.

847.714.3904 [Cell]