Tools, Technologies and Training for Healthcare Laboratories

Sigma-metrics of Six HbA1c methods

The August 2014 issue of the Journal of Clniical Chemistry had not one, but two different articles assessing the quality of different HbA1c methods. In addition to the Lenters-Westra and Slingerland, Woodworth et al performend a Sigma-metric analysis - this time "patient-weighted" - of their own. Do the metrics agree?

November 2014
Sten Westgard, MS

[Note: This QC application is an extension of the lesson From Method Validation to Six Sigma: Translating Method Performance Claims into Sigma Metrics. This article assumes that you have read that lesson first, and that you are also familiar with the concepts of QC Design, Method Validation, and Six Sigma. If you aren't, follow the link provided.]

The 2014 August issue of Clinical Chemistry was a double-treasure of HbA1c studies: not one, but two separate analyses of multiple HbA1c methods. While Dr. Westgard discusses the Letners-Westra and Slingerland study, here's a simple Sigma-metric analysis of the other study:

Utilization of Assay Performance Characteristics to Estimate Hemoglobin A1c Result Reliability. Woodworth A, Korpi-Steiner N, Miller JJ, Rao LV, Yundt-Pacecho YP, Kuchipudi L, Parvin CA, Rhea JM, Molinaro R, Clin Chem 2014;60(8):1073-1079.

The Imprecision and Bias Data

"Forty NGSP secondary reference laboratory (SRL) target value–assigned samples (Dr. Randie Little, University of Missouri, performed testing and provided samples
for this study for a fee; NGSP SRL) were sent to each laboratory and stored at 80 °C until analysis.... Precision for each assay was determined using the CLSI
EP5-A2 protocol. Respective laboratory Hb A1c QC materials (both low QC and high QC) were assayed in duplicate twice per day (morning and afternoon) for a
total of 20 days. [Bias was] determined according to the CLSI EP9-A2 protocol. Eight of 40 NGSP SRL samples were thawed each day and tested in duplicate over a period of 5 days."

Instrument Level (%HbA1c)
CV% Bias% Level (%HbA1c) CV% Bias%
Bio-Rad Variant II 5.09% 1.43% 4.99%  9.74%  1.33%  2.00%
Bio-Rad Variant II Turbo 5.18% 2.97% 0.08%  10.07%  1.81%  0.10%
Tosoh G8 5.75% 1.28%  3.99%  9.6%  0.80%  4.98%
 Capillarys 2 5.24% 1.60%  0.33%  7.93%  1.33%  0.01%
 Integra 800 5.61% 2.40%  5.76%  9.9%  1.18%  4.07%
DCA Vantage lot 1 5.31% 1.88% 0.34%  10.31%  2.65%  2.72%
DCA Vantage lot 2 5.23% 1.93% 0.37%  10.49%  1.81%  1.73%

We have measurements in the normal and diabetic levels, but what we're probably most interested is the performance at around 6.5% HbA1c, since that is where the diagnosis of diabetes is made.

 

Determine Quality Requirements at the decision level

Now that we have our imprecision and bias data, we're almost ready to calculate our Sigma-metrics. We're just missing one key thing: the analytical quality requirement.

For HbA1c, different organizations have set different quality goals. Despite the importance of this test, and the sheer volume of these tests being run, CLIA doesn't set a quality requirement. So it's not easy to decide what to pick - we'll choose a few options and calculate the metrics for all of them.

Source
Quality Requirement
CLIA PT
No quality requirement given
Rilibak (Germany)
Target value ± 18%
CAP/NGSP PT 2014 Target value ± 6%
Ricos et al. biologic database, desirable specification
3.0%

The details of these sources and quality requirements are discussed in Dr. Westgard's essay. The important thing to note here is that there is a pretty big difference between the requirements. For methods in US laboratories in 2014, the most relevant goal is the CAP/NGSP requirement of 6.0%.

A few years ago, we were talking about quality requirements of 10 to 12%, and clinical decision intervals of 14%. The requirements have gotten more demanding, and at the same time clinicians are tightening their interpretation of the test results. In this case, we're going to use several of the quality requirements and calculate a few different Sigma-metrics. It will up to the individual laboratory to determine which quality requirement is right for their use - and thus, which Sigma-metric is appropriate.

Calculate Sigma metrics

Now all the pieces are in place. Remember, this time we have two levels, so we're going to calculate two Sigma metrics.
(And then we'll make it more complicated by using multiple goals)

Remember the equation for Sigma metric is (TEa - bias%) / CV.

Example calculation: for a 6% quality requirement, at the level of 5.09% HbA1c, given Bio-Rad Variant II's 1.43% imprecision, 4.99% bias:

(6 - 4.99) / 1.43 = 1.01 / 1.43 = 0.7 Sigma

Instrument Level (%HbA1c)
CV% Bias% Sigma-metric Level (%HbA1c) CV% Bias% Sigma-metric
Bio-Rad Variant II 5.09% 1.43% 4.99% 0.7  9.74%  1.33%  2.00% 3.0
Bio-Rad Variant II Turbo 5.18% 2.97% 0.08% 2.0  10.07%  1.81%  0.10% 3.3
Tosoh G8 5.75% 1.28%  3.99% 1.6  9.6%  0.80%  4.98% 1.3
 Capillarys 2 5.24% 1.60%  0.33% 3.5  7.93%  1.33%  0.01% 4.5
 Integra 800 5.61% 2.40%  5.76% 0.1  9.9%  1.18%  4.07% 1.6
DCA Vantage lot 1 5.31% 1.88% 0.34% 3.0  10.31%  2.65%  2.72% 1.2
DCA Vantage lot 2 5.23% 1.93% 0.37% 2.9  10.49%  1.81%  1.73% 2.4

 

Recall that in industries outside healthcare, on the short-term scale, 3.0 Sigma is the minimum performance for routine use and 6.0 Sigma is considered world class quality. We're looking at the long-term scale for this Sigma-metric calculation, which is 1.5s higher (the short-term scale builds in a 1.5s shift, to allow for "normal process variation"). So we could go as low as 1.5 for the bare minimum acceptability. Still, what this is telling us is that we're not achieving great performance when we apply the standards of CAP or the desirable specification for total error.

The study computerd a "patient-weighted" Sigma-metric.

Instrument Low Sigma-metric High Sigma-metric "Patient-weighted" Sigma-metric Standard Average Metric
Bio-Rad Variant II 0.7 3.0 1.57 1.85
Bio-Rad Variant II Turbo 2.0 3.3 2.29 2.65
Tosoh G8 1.6 1.3 1.43 1.45
Capillarys 2 3.5 4.5 3.90 4.0
Integra 800 0.1 1.6 0.36 0.85
DCA Vantage lot 1 3.0 1.2 2.36 2.1
DCA Vantage lot 2 2.9 2.4 2.84 2.65

 The "Patient-weighted" metric is close to a simple average of the low and high metrics. There may not be much additional utility it offers to laboratories. It may be far more useful to simply choose the level where medical interpretation is most important, and calculate the Sigma-metric at that level.

Summary of Performance by Sigma-metrics Method Decision Chart and OPSpecs chart

We can make visual assessments of this performance using Sigma-metric Method Decision Charts.

2014-6 HbA1c methdos MEDX

Here we can see that no method is hitting the bull's eye, and a majority of the methods seem to be missing the target. Only the Capillarys 2 seems to perform well.

Now what about QC? How do we monitor and control these methods? For that, we need an OPSpecs chart:

2014-6 HbA1c methods OPSpecs

 

Here, most of the methods are not controllable. That is, even with the full "Westgard Rules" we probably won't be catching errors when they first occur - it will take a while before we pick them up. Only the Capillarys method seems to be controllable with QC, a set of "Westgard Rules." The study authors reached the same conclusion, but used a different rule of thumb to determine QC intensity, rules, and frequency:

"Except for the Capillarys 2 Flex Piercing, the patient-weighted [Sigma] metric for all platforms investigated at a TEa of 6% was <3, indicating that maximum QC (3 levels, 3 times per day) should be performed to achieve the necessary error detection."

 It's unlikely that many laboratories view running 9 controls a day as a practical approach to monitoring QC. More economical to find a better method than to have to double or triple the number of controls being run every day.

Conclusion

HbA1c is one of the assays where a great deal of progress has been made in standardization. Quality requirements have been tightened, methods have been improved. And even Lenters-Westra studies have shown improvements in method performance over the last 4-5 years. But this study shows for sure that there is still plenty of room for improvement.