Sigma Metric Analysis

Today's emerging EFLM standards for desirable imprecision and bias are demanding. Can today's analyzers meet these desires? We check using the package inserts of the Mindray BS-200 analyzer.

Sigma-metric Analysis of Mindray BS-200

Sten Westgard, MS
February 2019

• The Precision, Comparison and Sigma-metric data
• Summary of Performance according to CLIA and Ricos Goals by Sigma-metrics Normalized Method Decision charts
• QC Implications of Mindray BS200 Performance by Normalized OPSpecs chart
• Conclusion

[Note: This QC application is an extension of the lesson From Method Validation to Six Sigma: Translating Method Performance Claims into Sigma Metrics. This article assumes that you have read that lesson first, and that you are also familiar with the concepts of QC Design, Method Validation, and Six Sigma metrics. If you aren't, follow the link provided.]

The EuBIVAS papers that have been published in the last few years, representing an update to the "Ricos Goals" of 2014, paint a picture of tightening analytical performance specifications. As these studies are converted into data determining desirable specifications for analytical imprecision, inaccuracy, and allowable total error.

The Imprecision, Bias and Sigma-metric Data

Package inserts for the reagents for the Pointe Scientific Pointe c2000 and Mindray BS-200 analyzers were accessed online.

"Precision studies were performed using the Pointe c2000 / Mindray BS-200 analyzer following a modification of the guidelines which are contained in NCCLS [now known as CLSI] document EP5-T2 ..."

Each package insert has a slightly different comparison detail, but the most of them are similar to the following "A study was performed between the Pointe c2000 / Mindray BS-200 and a similar analyzer using this method, resulting in a correlation coefficient of .... with a regression equation of ..."  The correlation coefficient, slope, y-intercept, and number of patient samples monitored.

For most of the analytes two levels of control were run, while for CK, three levels of control were run. Based on these package inserts, CV and Bias is listed below:

 Yes, that is a whole lot of numbers!

Nevertheless, what do all these numbers mean? In the absence of context, it's hard to know.

So let's calculate the Sigma-metrics.

Sigma-metric calculations for the BS-200

Remember the equation for Sigma metric is (TEa - bias) / CV:

For a 10% quality requirement, for Albumin on the low level of BS200, the equation is (10 - 6.8) / 1.9 = 1.68

For a 10% quality requirement, for Albumin on the high level of BS200, the equation is (10 - 6.9) / 2.3 = 1.3

The metrics are displayed along the right column.

Yes, there are a lot of low Sigma-metrics. Negative Sigma is unusual, reflecting a situation when the bias completely exceeds the allowable total error. There are a few high Sigma-metrics, but the majority of the numbers are below 3 Sigma, typically considered the minimum acceptable performance.

 Even if we don't use Sigma-metrics and total analytical total allowable errors to judge these methods, we can use the Ricos 2014 desirable specifications for imprecision and bias, as well as the updated EuBIVAS studies (which started in 2017 through 2018) desirable specifications for imprecision and bias. We'll highlight in red the performance where the instrument failed to achieve those goals, and highlight in green the areas where the instrument did achieve those goals.

 As you can see, most of these assays meet the new desirable specifications, but 30% of the data here does NOT meet the desirable specifications set forth by EuBIVAS.

Considering this is the performance data that the company itself generated, that's a lot of failure right from the starting pistol. If performance in the real world for laboratories is worse than the manufacturer specifications, that means even more than a third of the assay points are below acceptability.

Summary of Performance by Sigma-metrics Method Decision Chart using primarily CLIA Goals

We can make visual assessments of this performance using a Normalized Sigma-metric Method Decision Chart:

A few have both more than the allowable bias and more than the allowable imprecision.

Summary of QC Design by Normalized OPSpecs chart - using primarily CLIA Goals

The benefit of the Sigma-metric approach is that labs can do more than assess their quality, they can act on it. Using OPSpecs charts, they can actually optimize their QC procedures for each test. In this case, they can use the data to try and mitigate the risk of poor performance.

For a dozen of these points, we will need the maximum "Westgard Rules" with 8 control measurements per run. Put bluntly, there is no practical statistical quality control procedure that can help. There aren't enough "Westgard rules" to keep some of these methods in control. 

Assessing Performance and QC Design by Normalized Method Decision and OPSpecs Chart - using EuBIVAS goals

Now, what if we used analytical performance specifications for allowable total error derived from these new EuBIVAS studies? These will create quality requirements much more stringent than current CLIA goals. How would we assess this instrument if we used EuBIVAS TEa goals?

First we should look at the new Sigma-metrics calculated using Desirable TEa goals from EuBIVAS studies:

You can see the metrics got a lot worse once the EuBIVAS goals are applied. If analytical allowable total erros are calculated from EuBIVAS specifications, the acceptability of this instrument is dramatically reduced.  A majority of the performance here is now below Three Sigma. And about 70% of the performance is below Four Sigma. One method is ZERO Sigma, and another one is negative Sigma, which simply means the bias is larger than the allowable total error.

The Normailzed Method Decision chart makes it more obvious we have a problem if EuBIVAS goals are applied.

You can see the number of unacceptable data points has exploded. The number of Six Sigma assays has shrunk.

Again, a majority of these assay control levels will need maximum "Westgard Rules" and a quadrupling of control measurements. Which is really another way of saying - we can't control those methods, there are not enough rules or enough controls we can run. If we looked at how often we should be running, it would be even more impractical.

In other words, the BS200 does not meet the desirable analytical performance specifications of EuBIVAS.

Conclusion

Package inserts from manufacturers are often thought of as overly optimistic, idealized, perhaps too good to be true. Typically, real world laboratories experience more bias and more imprecision than what is shown on the insert. Yet for the BS-200, just under a third of the performance is unacceptable by CLIA goals, and more than 50% of the performance is unacceptable by new EuBIVAS standards.

What are we to conclude? Is the BS-200 unacceptable for today's laboratories? tomorrow's laboratories? Or are we holding assays to standards that no laboratory, no instrument can meet? Perhaps the goals are unacceptable, but the instrument is fine.

Of course, these possibilities are not mutually exclusive.  It is entirely possible that some EuBIVAS goals are unattainable AND some instruments are unacceptable by nearly any quality standard.