Quality Requirements and Standards

The third installment discussing the quality of HbA1c devices in 2014, based on the new study by Lenters-Westra and Slingerland in Clinical Chemistry. In this part, Dr. Westgard looks at data from the second reagent lot tested, as well as provides an analysis using IFCC units and reference methods.

Another Teaching Moment: POC HbA1c Performance

Part 3. Sigma Calculations

James O. Westgard
September 2014

The new report on the performance of POC HbA1 by Lenters-Westra and Slingerland was published in the August 2014 issue of Clinical Chemistry [1]. If you weren’t able to access the online report [2], it is now available in hard copy.

Sigma Calculations for second Lot of Reagents

In part 2 in this series, we used the Method Decision Chart to provide a graphical display of the CVs and biases observed for the 7 test methods based on the comparison data for the first reagent lot. Now we’ll show the results for the second reagent lot, along with the calculated sigma values.

First, I must admit to a mistake in the directions for setting up an electronic spreadsheet. In part 2, I said to calculate the systematic error, or bias, as X_c – Y_c. Actually, it is better to calculate Y_c – X_c because then the sign of the bias will then indicate whether the test method gives higher or lower results than the comparison method. That doesn’t change the graphical display with the Method Decision Chart because an absolute value of %Bias is used as the y-coordinate for the operating points. Nonetheless, it is better to calculate Y_c – X_c than the reverse.

The calculations for the second reagent lot are shown in the Table I, where the last column provides the calculated sigma-metric using the CAP and NRSP TE_a of 6.0%. Note that the sigma values range from 0 (actually -0.33) to 4.4. The negative sigma value indicates that the observed bias is greater than the defined TE_a. We consider that sigma is zero for this condition.

Graphical results are shown in Figure 1. This display makes it clear that 3 methods perform at less than 3-sigma quality, 2 methods border on 3-sigma quality, and 2 methods are better than 3-sigma quality, one averaging about 4-sigma. You will have to be the judge whether the tabular or graphical results are more useful for understanding and interpreting the experimental data.

What SQC is needed?

Elsewhere on this website we have recently described “Westgard Sigma RulesTM” as a new and simple approach for customizing SQC for the sigma quality observed in the laboratory. This approach was recently published in our book on Basic Quality Management Systems [3]. We can use this new tool to assess the SQC needed for POC HbA1c testing.

Figure 2 shows that a method achieving 4-sigma quality should be controlled using a multirule 1_3s/2_2s/R_4s/4_1s procedure with 4 control measurements. Note the sigma scale at the bottom of the figure. For a 4-sigma quality test, read up from that scale to identify the control rules above and to the left. Read the number of control measurements next to the up arrow, in this case N=4 R=1 (4 control measurements in 1 run) or N=2 R=2 (2 control measurements in each of 2 runs).

Figure 3 shows that a method achieving 3-sigma quality will need to add an 8x rule and 4 control measurements in each of 2 runs, or 2 control measurements in each of 4 runs. For methods with less than 3-sigma quality, it won’t be possible to do enough QC to ensure the test achieves the quality required for the intended clinical use.

Sigma Calculations for SI Units

In the last lesson, I suggested that it would be useful to also perform the calculations using SI Units, which are commonly used in Europe and elsewhere around the globe. Up to now, the data analysis here has focused on DCCT units of %Hb because their use is common practice in the US. In other countries, however, test results are often presented in SI Units because the IFCC reference method provides results in those units. For comparison, the diagnostic criterion of 6.5 %Hb is equivalent to 48 mmol/mol. Published studies show a linear relationship between the IFCC reference method and the DDCT methods, but there is a large intercept of approximately 2 %Hb. For that reason, when the SDs in mmol/mol are converted to CV in %, the CVs are considerably higher; likewise, figures for %Bias are also higher. If TE_a is defined as 6.0%, that means the calculated sigma-metrics will be lower, as shown in Table II.

The results in SI Units are also displayed in Figure 4, which had to be re-scaled to accommodate the operating points. Sigma quality is clearly worse when results are provided in SI Units!

A Teaching Moment – 2014

Again, you may want to review the previous series of lessons to guide you through the method evaluation process. Those earlier lessons provide a more detailed description of the process and the analysis of the experimental data. What has changed since is (a) the quality requirement has gotten tighter, TE_a of 6.%, and (b) performance of some of the newer POC devices has improved. However, the more demanding quality requirement more than offsets improvements in performance, as evidenced by the observed sigma quality of these HbA1c methods. That means laboratories must still be critical in their selection of HbA1c POC devices to ensure attainment of the quality required for the intended clinical use.

References

1. Lenters-Westra E, Slingerland RJ. Three of 7 hemoglobin A1c point-of-care instruments do not meet generally accepted analytical performance criteria. Clin Chem 2014;60:1062-72.
2. Lenters-Westra E, Slingerland RJ. Three of 7 hemoglobin A1c point-of-care instruments do not meet generally accepted analytical performance criteria. http://hwaint.clinchem.org/cgi/doi/10.1373/clinchem.2014.224311
3. Westgard JO, Westgard SA. Basic Quality Management Systems. Chapter 12. Designing SQC procedures. Madison WI:Westgard QC, Inc., 2014.
4. Westgard JO. 2011 Update on HgA1c quality goals and performance requirements. See Part XI for links to these lessons. www.westgard.com/hba1c-methods-part11.htm