Sigma Metric Analysis

We don't often analyze hematology instruments. A study of a new miniature automated hematology analyzer system was published in the International Journal of Laboratory Hematology in 2015. So what are the goals for quality for hematology?

Sigma-metrics of a Mindray BC-5000 hematology analyzer

• The Precision and Comparison data
• Determine quality requirements
• Calculate Sigma metrics
• Summary of Performance by Sigma-metrics chart and OPSpecs Chart
• Conclusion

January 2016
Sten Westgard, MS

[Note: This QC application is an extension of the lesson From Method Validation to Six Sigma: Translating Method Performance Claims into Sigma Metrics. This article assumes that you have read that lesson first, and that you are also familiar with the concepts of QC Design, Method Validation, and Six Sigma. If you aren't, follow the link provided.]

Most of the Sigma-metric studies we've done on Westgard Web have been on chemistry methods and instruments. But that's more a matter of coincidence than choice - there are simply more studies on chemistry analytes available than many other types of testings. But Sigma-metrics can be applied to other quantitative testing, including hematology. :

Evaluation of Mindray BC-5000 hematology analyzer: a new miniature 5-part WBC differential instrument Xiang D, Yue J, Lan Y, Sha C, Ren S, Li Y Li M, Wang C, Int Jnl Lab Hem 2015, 37, 597-605.

The Imprecision and Bias Data

Precision was evaluated according to EP5-A2, with 10 consecutive measurements of low, normal, and high samples. Between-day imprecision was calculated by running controls twice a day over 20 days.

Next, we need the bias data. The study states that "Three hundred and ten samples were performed twice on BC-5000 and XE-2100 analyzer....[for] WBC, PLT...RBC, Hb....256 samples were used to evaluate the leukocyte differential count parameters between the XE-2100 and BC-5000."

The study determined the regression slope and y-intercept as well as the correlation coefficient. The regression equation can be used to determine the bias between the BC-5000 and the XE-2100. We will not duplicate all of those regression calculations, but simply show the end result of the bias calculations.

We're close to number saturation with this set of data now. It's hard to take all the details and create an assessment of whether or not this performance is good, bad, or indifferent.

This is where it's helpful to establish the quality requirements (analytical performance specifications) so the assay performance can be placed into the proper context.

Determine Quality Requirements at the decision levels

The CLIA goals for proficiency testing don't cover all the differential parameters, so we need to consult more than one resource to build a complete set of performance specifications. In this case, we're going to use the "Ricos goals" for desirable performance.

 The differential performance specifications may seem large, however performance of methods is much less precise for those methods. So we can't assume that the goal for Basophils, for instance, is too large or too easy.

Calculate Sigma metrics

Remember the equation for Sigma metric is (TE_a - bias) / CV.  (everything is a percentage here)

Example calculation: for WBC at the low level and a 15% quality requirement, with a 4.4% CV and a 5.4% bias:

(15 - 5.4) / 4.4 = -9.6 / 4.4 = 2.18

At the low level, this assay is not performing above Three Sigma.

Here's the full table with all the metrics:

There's some good news and some bad news. Several methods are consistently Six Sigma or better across all levels. However, that is not true for all methods. In particular, the Basophil method has a bias that exceeds that allowable total error, which means this BC-5000 method is getting a clinically significantly different result than what the XE-2100 method would report.

Summary of Performance by Sigma-metrics Method Decision Chart and OPSpecs chart

We can make visual assessments of each instrument's performance using a Normalized Sigma-metric Method Decision Chart:

Neutrophils, Eosinophils, Hemoglobin, Neutrophils and WBC are mostly in the bull's-eye. RBC and Hematocrit have imprecision challenges. There's a significant bias for monocytes.

Now what about QC? How do we monitor and control these methods? For that, we need a Normalized OPSpecs chart:

Many of the methods can be easily controlled with only two controls and wide control limits. 6 of the 10 analytes probably don't need to implement the "Westgard Rules."  However, for RBC, Hematocrit, Monocytes,  the full "Westgard Rules" are needed, possibly more frequent QC as well. For Basophils, we may need all that and more, if we believe that the bias between the BC 5000 and XE 2100 is significant and cannot be calibrated out.

Conclusion

The authors' conclusion relies on the old stand-by, the correlation coefficient: "The Mindray BC-5000 auto hematology analyzer performed well for the parameters evaluated, and the results for background, precision, carryover, linerarity were acceptable. Further, there was a good corrleation between the evaluated instruments and the reference analyzer the Sysmex XE-2100. Notably, for all of the analyzed parameters of CBC, the correlation of each parameter was acceptable."

The correlation coefficient is the wrong statistic to use to judge the acceptability of methods. This has been known for decades. The Sigma-metric analysis is more useful in its ability to determine the acceptability of the methods. For the BC-5000, many methods are quite good, even world class. But it would be a mistake to assume that the correlation coefficients from this study indicate that all the methods are acceptable.