Tools, Technologies and Training for Healthcare Laboratories

Evaluation of Two HbA1c POC Analyzers

As part of our continuing series of evaluations, we take a look at two point-of-care (POC) HbA1c analyzers. As standards on performance have tightened, have POC devices kept up? Has their improved performance matched the higher demands? 

November 2012
Sten Westgard, MS

[Note: This QC application is an extension of the lesson From Method Validation to Six Sigma: Translating Method Performance Claims into Sigma Metrics. This article assumes that you have read that lesson first, and that you are also familiar with the concepts of QC Design, Method Validation, and Six Sigma. If you aren't, follow the link provided.]

Our last look at POC devices that measure HbA1c was back in 2010. Clinical Chemistry in Laboratory Medicine published a study in 2011 that looked at 2 devices. We decide to take this data and calculate Sigma-metrics. Here's the more recent study:

Evaluation of the Performance of two HbA1c point-of-care analyzers. Sanchez-Mora C, Rodriguez-Oliva MS, Fernandez-Riejos P, Matro J, Polo-Padillo J, Goberna R , Sanchez-Margalet V, Clin Chem Lab Med 2011;49(4):653-657.

The Precision and Comparison Data

"For intra-assay variability, the same sample was repeated 20 consecutive times. The samples used for intra-assay variability had HbA1c values of 9.5%... and 4.5% as determined by the laboratory method."

Instrument Level (%HbA1c)
CV%
Instrument A 4.5% 2.66%
9.5% 1.95%
Instrument B 4.5% 2.97%
9.5% 3.1%

Note that CLSI and current guidelines for Diabetes set the maximum allowable CV at 5%, with 3% as the desired performance. So performance has met those thresholds.

Bias / trueness was assessed using by comparing the two POC methods to the laboratory method, which was an ARKRAY HA 8160, an HPLC method. 53 samples from diabetic patients were compared, grouped across 4 ranges from 4% to >10%. Linear regression statistics were used.

Instrument n
Regression Slope
Y-intercept Type
Instrument A 53 1.02 +0.03 NGSP units
Instrument B 53 1.13 -0.84 NGSP units

We are going to use the regression equation to calculate the bias at both levels where imprecision was measured.  This helps to align our bias data with our imprecision data.

Just to review this calculation, here's a layman's explanation of the equation:

NewLevelNewMethod = (slope * OldLevelOldMethod ) +Y-intercept

Then we take the difference between the New and Old level, and convert that into a percentage value of the Old level.

Example Calculation: Given Instrument A at level 4.5%, comparing to the NGSP assigned values:

NewLevelA = (1.02 * 4.5) + 0.03

NewLevelA = 4.59 + 0.03

NewLevelA = 4.62

Difference = 4.62 - 4.5 = 0.12

Bias% = 0.12 / 4.5 = 2.7%

Now remember we have precision data for two levels - so we can use the regression equation and calculate bias for each level:

Instrument Level (NGSP)
Total CV% Bias%
A 4.5% 2.66% 2.7%
9.5% 1.95% 2.3%
B 4.5% 2.97% 5.7%
9.5% 3.1% 4.2%

Determine Quality Requirements at the decision level

Now that we have our imprecision and bias data, we're almost ready to calculate our Sigma-metrics. We're just missing one key thing: the analytical quality requirement.

For HbA1c, different organizations have set different quality goals. Despite the importance of this test, and the sheer volume of these tests being run, CLIA doesn't set a quality requirement. So it's not easy to decide what to pick - we'll choose a few options and calculate the metrics for all of them.

Source
Quality Requirement
CLIA PT
No quality requirement given
Rilibak (Germany)
Target value ± 18%
CAP PT 2011 Target value ± 7%
Ricos et al. biologic database, desirable specification
4.3%

The details of these sources and quality requirements are discussed in Dr. Westgard's essay. The important thing to note here is that there is a pretty big difference between the requirements.

A few years ago, we were talking about quality requirements of 10 to 12%, and clinical decision intervals of 14%. The requirements have gotten more demanding, and at the same time clinicians are tightening their interpretation of the test results. In this case, we're going to use several of the quality requirements and calculate a few different Sigma-metrics. It will up to the individual laboratory to determine which quality requirement is right for their use - and thus, which Sigma-metric is appropriate.

Calculate Sigma metrics

Now all the pieces are in place. Remember, this time we have two levels, so we're going to calculate two Sigma metrics.
(And then we'll make it more complicated by using multiple goals)

Remember the equation for Sigma metric is (TEa - bias) / CV.

Example calculation: for a 7% quality requirement, at the level of 4.5% HbA1c, given method A's 2.66% imprecision, 2.7% bias:

(7 - 2.7) / 2.66 = 4.34 / 2.66 = 1.6

Instrument A Sigma-metric Performance
Quality Requirement
Level Total CV% Bias% Sigma-metric
7% (CAP 2011) 4.5% 2.66% 2.7% 1.6
9.5% 1.95% 2.3% 2.4
4.3% (Ricos et al) 4.5% 2.66% 2.7% 0.6
9.5% 1.95% 2.3% 1.0
18% (Rilibak) 4.5% 2.66% 2.7% 5.8
9.5% 1.95% 2.3% 8.1

Recall that in industries outside healthcare, on the short-term scale, 3.0 Sigma is the minimum performance for routine use and 6.0 Sigma is considered world class quality. We're looking at the long-term scale for this Sigma-metric calculation, which is 1.5s higher (the short-term scale builds in a 1.5s shift, to allow for "normal process variation"). So we could go as low as 1.5 for the bare minimum acceptability. Still, what this is telling us is that we're not achieving great performance when we apply the standards of CAP or the desirable specification for total error. The Rilibak goal for performance, which is very generous, is the one goal that the methods can achieve.

Instrument B Sigma-metric Performance
Quality Requirement
Level Total CV% Bias% Sigma-metric
7% (CAP 2011) 4.5% 2.97% 5.7% 0.4
9.5% 3.1% 4.2% 1.5
4.3% (Ricos et al) 4.5% 2.97% 5.7% negative
9.5% 3.1% 4.2% 0.6
18% (Rilibak) 4.5% 2.97% 5.7% 4.1
9.5% 3.1% 4.2% 5.1

For instrument B, higher bias and higher imprecision make it even harder to achieve the minimum level of performance. Only when the Rilibak rules are applied can this method achieve good or excellent performance.

Summary of Performance by Sigma-metrics Method Decision Chart and OPSpecs chart

Because we have different quality requirements, but we are still interested in assessing performance of the same method, we'll use normalized Method Decision Charts and Normalized OPSpecs charts.

2012-HbA1c-Normalied Method Decision Chart, POC method A

Here we can see the deisrable specification for total error based on biologic variation (Ricos goal) is not achievable. If we judge the quality by Rilibak standards, however, Method A is very good.

2012-HbA1c-Normalized Method Decision Chart for POC Method B

Here, with Method B, CAP and Ricos goals are not possible to hit. Again, the method can hit the Rilibak quality goals.

For both A and B, it appears that CAP and Ricos goals are not possible to hit, at least not with a Six Sigma level of performance.

Using QC Design tools (such as EZ Rules 3 ), we can express the method performance on an OPSpecs (Operating Specifications) chart and determine the changes we might make to our QC procedures for this method:

2012-HbA1c-Normalized OPSpecs chart for Method A

Here, if Rilibak goals are being applied, this method can be controlled with simple a 1:3s control procedure and 2 control measurements.

2012-HbA1c-Normalized OPSpecs chart for Method B

If we were to base our QC decisions on this data, for Rilibak requirement, we would need a robust set of "Westgard Rules."

For both methods, neither biologic variation goals nor the CAP goals are within reach. That is, if those goals are important, even the most robust set of "Westgard Rules" will not detect all the errors in the first run - the error will persist over multiple runs before the QC procedure will be able to detect the error.

Conclusion

There's a lot of talk about how point-of-care devices have advanced to the point where they are now providing "lab quality." This study shows that, for these two methods at least, lab quality is not being provided. (If you want to see what lab quality for HbA1c looks like, check out this study).

It's worth considering that HbA1c at the point-of-care might require different standards for quality. With glucose, for instance, we know that the quality demanded at the core laboratory is tighter (10%) than at the point-of-care (20%, or possibly coming soon, 15%). With HbA1c, paradoxically, CLIA has never set standards for performance, at the point of care or in the core laboratory. There's a need for the other organizations to think seriously about what performance is needed from these methods.