Tools, Technologies and Training for Healthcare Laboratories

Analysis of 4 HbA1c instruments by Sigma-metrics

IFCC EFLM have recommended that HbA1c methods be assessed using Sigma-metrics. We're starting to see papers now that incorporate Sigma-metrics into the instrument assessment. What are they telling us about today's latest HbA1c instruments? Are they good enough for use with patients?

 Sigma-metric Analysis of an 4 HbA1c instruments

Sten Westgard, MS
August 2017

[Note: This QC application is an extension of the lesson From Method Validation to Six Sigma: Translating Method Performance Claims into Sigma Metrics. This article assumes that you have read that lesson first, and that you are also familiar with the concepts of QC Design, Method Validation, and Six Sigma metrics. If you aren't, follow the link provided.]

This analysis uses data from a recent study in Laboratory Medicine:

Jose M. Maesa, Patricia Fernandez-Riejos, Catalina Sanchez-Mora, Maria de Toro-Crespo, Concepcion Gonzalez-Rodriguez, Application of Six Sigma Model to Evaluate the Analytical Quality of Four HbA1c Analyzers, Clin Lab 2017;63:.

This study evaluated the performance of four high performance liquid chromatography (HPLC) analyzers: the Menarini HA-8180V, Bio-Rad D-100, Tosoh G8, and Bio-Rad Variant II Turbo. [Please note, all these instruments are registered trademarks of their respective manufacturers]

The Imprecision, Bias and Sigma-metric Data

"For the same 20 consecutive days, we proceed to measure the two levels of quality control (high and low) provided by manufacturers in each of the analyzers....With values obtained, we calculated imprecision (standard deviation and CV) and bias for each quality control level and instrument...."

 Two levels of controls were measured, but we further concentrated the Sigma-metric assessment on just one decision level per test, which is highlighted in yellow.

HbA1c HPLC methods 
Instrument Information (DCCT, %HbA1c)
TEST control
Target Values Bias Units % Bias % CV
Menarini HA-8080V Glyco HbA1c control  5.8 0.1 0.9 1.2
10.7 0.13 1.2 1.4
Bio-Rad D-100
 
Lyphocheck Diabetes control
 
5.25 0.02 0.3 1.2
9.7 0.1 1.0 1.7
Tosoh G8 Tosoh HbA1c control  5 0.12 2.4 0.8
9.9 0.08 0.8 0.5
Bio-Rad
Variant II Turbo
 
Lyphocheck Diabetes control
 
5.25 0.22 0.22 4.9
9.7 0.25 0.25 3.6

 So that's what we start with, and from this we can start calculating Sigma-metrics... once we agree on an allowable analytical total error (commonly abbreviated as TEa%). That, unfortunately, is not an easy thing to agree on. In the paper, TEa from 1 to 10 are used and a variety of Sigma-metrics are calculated. 10 different metrics for each data point. It makes the graphs a bit confusing. But there are three dominant suggestions for TEa that exist:

  • The CAP/NGSP program has drawn down the TEa% to 6.0%. 
  • The desirable specification for TEa using the 2014 biological variation database (commonly called the "Ricos goals") is only 3.0%.
  • The Lenters-Westra specifications for TEa for % HbA1c is 6.7%.

In these scenarios, what are the different metrics? 

HbA1c HPLC methods 
Instrument Information (DCCT, %HbA1c)
TEST control
Target Values Bias Units % Bias % CV CAP/NGSP Sigma Ricos Sigma Westra Sigma
Menarini HA-8080V Glyco HbA1c control  5.8 0.1 0.9 1.2 4.4 1.82 5.01
10.7 0.13 1.2 1.4 3.5 1.28 3.96
Bio-Rad D-100
 
Lyphocheck Diabetes control
 
5.25 0.02 0.3 1.2 4.8 2.3 5.3
9.7 0.1 1.0 1.7 2.96 1.19 3.38
Tosoh G8 Tosoh HbA1c control  5 0.12 2.4 0.8 4.7 0.8 5.6
9.9 0.08 0.8 0.5 10.2 4.3 11.6
Bio-Rad
Variant II Turbo
 
Lyphocheck Diabetes control
 
5.25 0.22 0.22 4.9 0.4 neg 0.5
9.7 0.25 0.25 3.6 1.0 0.1 1.2

Confusing, I know. It's a lot of numbers. What might be considered acceptable according to the CAP/NGSP standards, may not be acceptable according to the Ricos desirable goals.

Summary of Performance by Sigma-metrics Method Decision Chart

We can make visual assessments of this performance using a Normalized Sigma-metric Method Decision Chart:

2017 4 HbA1c methods Ricos TEa Sigma method decision chart

When we use the Ricos desirable TEa goal for HbA1c, there's not a lot of acceptable performance. The method performance for the Bio-Rad Variant II Turbo is actually off the charts here. But most of the methods are not that much better.

2017 4 HbA1c methods CAP TEa method decision chart

When the CAP/NGSP TEa goal is used, 3 of the 4 methods are acceptable. The Bio-Rad Variant II Turbo is still off the charts due to poor performance.

2017 4 HbA1c methods Lenters-Westra TEa Sigma Method Decision chart

The Westra TEa doesn't change any performance decision from what we saw with CAP/NGSP goals.

What we can conclude quite quickly is that (1) Ricos desirable goals are unrealistic for HbA1c, and (2) we can rule out the Bio-Rad Variant II Turbo. It doesn't come close to meeting any of analytical performance specifications.

Summary of QC Design by Normalized OPSpecs chart

 What do the different Sigma-metrics mean when it comes to designing QC? How many controls are necessary and how many "Westgard Rules" are needed for a 4, 5, ot 6 Sigma method?

We can use the OPSpecs chart to guide our QC planning. We'll skip the application of the Ricos goal, however, since that was not possible to achieve.

2017 4 HbA1c methods CAP TEa OPSpecs chart

For the moment, we'll just work with the "worst-case" scenario - we'll select a QC procedure that works for both levels of performance. [In a different scenario, we might only choose the decision level aroudn 6.5%, which is where the critical diagnosis for diabetes is made]. With the CAP/NGSP goal, the HA-8180V can use just 3 of the "Westgard Rules" with N=2. The D-100 will need all the "Westgard Rules" and 8 controls. The Tosoh G8 will need even more than that. Even though the Tosoh G8 performance is a lot better than the Variant II Turbo, it still will require intensive lab efforts to keep in control.

2017 4 HbA1c methods Westra TEa OPSpecs chart

Herre we can see even this slightly larger TEa goal makes a big difference in the choice of QC rules. Now the HA-8180V can use 1:3s and N=2 (no "Westgard Rules"!) And the D-100 can reduce to just 4 "Westgard Rules" and 4 controls per run. The Tosoh G8 is good enough now that the "Westgard Rules" and 8 controls will keep this method in control - there won't be a need for even more extraordinary (Herculean) efforts.

 

Conclusion

The authors conclude "Applying the Stockholm consensus and its subsequent Milan review to the results: the maximum level in quality requirements for HbA1c is an allowable total error (TEa) = 3%, G8...is a poor result, and HA-8180V and D-100 are both in region 1 [Sigma]...which is an unacceptable result."

This conclusion, for once, seems a bit harsh to us. Usually we see authors that are too kind, suffering from a bias toward reaching positive conclusions. It's nice to see that sometimes the authors are unsparing in their evaluations. However, the 3% Ricos desirable TEa may be so demanding that NO method on the market can achieve acceptable performance. Rather than halt all HbA1c testing around the world while we wait for better engineering, perhaps we should consider other goals. Both the NGSP, CAP, and Erna Lenters-Westra have concluded that a TEa goal around 6.0 to 6.7% is appropriate for the clinical use of the test.

[Notice that even in their conclusion, the authors omitted discussion of the Bio-Rad Variant II Turbo. It simply was off the charts poor, no matter which of these goals was applied.]

Ern Lenters-Westra, in the HbA1c method validation work her laboratory performs,  has a detailed set of specifications for HbA1c performance. It's worth using those to make one final check of method performance.

The [Lenters-]Westra Rules:

  • As already noted, TEa for DCCT % Hba1c should be 6.7%
  • CV in CLSI EP-5 should be at 48 mmol/mol and 75 mmol/mol < or equal to 3% in SI units or < or equal to 2% in DCCT units
  • Bias in CLSI EP-9 at 48 mmol/mol and 75 mmol/mol compared with the mean of at least 3 certified secondary reference measurement procedures should < or equal to 2 mmol/mol or < or equal to 0.2% DCCT units.
  • The IFCC/EFLM TF-HbA1c [task force for HbA1c standardization] has set default risk levels of 2σ for routine laboratories and 4σ for laboratories performing clinical trials.

If we apply all these criteria, does that change our acceptance of the methods?

HbA1c HPLC methods 
Instrument Information (DCCT, %HbA1c)
TEST control
Target Values Bias Units % Bias % CV Westra Sigma
Menarini HA-8080V Glyco HbA1c control  5.8 0.1 0.9 1.2 5.01
10.7 0.13 1.2 1.4 3.96
Bio-Rad D-100
 
Lyphocheck Diabetes control
 
5.25 0.02 0.3 1.2 5.3
9.7 0.1 1.0 1.7 3.38
Tosoh G8 Tosoh HbA1c control  5 0.12 2.4 0.8 5.6
9.9 0.08 0.8 0.5 11.6
Bio-Rad
Variant II Turbo
 
Lyphocheck Diabetes control
 
5.25 0.22 0.22 4.9 0.5
9.7 0.25 0.25 3.6 1.2

Again, the "Westra Rules" show that the bias and the imprecision of the Variant II Turbo is unacceptable, so it's no wonder that the resulting Sigma-metric is also unacceptable as well. Three of the four methods are acceptable with the Westra criteria and the TF-HbA1c guidance on acceptable Sigma levels. But the implementation challenges of applying proper QC design may be such that only a method like the Tosoh G8 is desirable.