Sigma Metric Analysis

We analyze some data from the field and look at performance in the Roche Modular instrument.

• The CVs and Sigma metrics
• Estimates of Tolerable bias
• A combination of data for surrogate bias estimates
• Conclusions

September 2006

[Note: This QC application is an extension of the lesson From Method Validation to Six Sigma: Translating Method Performance Claims into Sigma Metrics. This article assumes that you have read that lesson first, and that you are also familiar with the concepts of QC Design, Method Validation, and Six Sigma. If you aren't, follow the links provided.]

Recently, a user submitted us with data from one of the hospitals where she consults. This laboratory has been working with the Roche Modular and decided to benchmark the performance of their methods on the Sigma scale.

The CVs and Sigma metrics

The TEa column lists the Total Allowable Error, as specified by the CLIA proficiency testing criteria. For Reticulocyte, which has no CLIA mandated quality requirement, data from the biological variation database was used to calculate an Biologic Total Error (TE_ba).

Obviously, the numbers here are not only good, they're great. Of course, we have to keep in mind that these Sigma metrics are calculated without any bias data. Therefore, they are artificially high. Nevertheless, only 2 of the 21 methods are unacceptable at this point.

Estimates of Tolerable Bias

Now that we have the optimistic numbers, let's try to get more realistic. As noted before, we do not have bias figures for this instrument. However, we can make estimates of the bias that could be tolerated if defined Sigma metrics are assumed. From a QC Design perspective, there is not difference between a Sigma metric of 6.0 and a higher Sigma metric of 22.0. We can illustrate this by showing some of these metrics by OPSpecs charts [Note: the source of these graphs and recommendations is the EZ Rules(r) 3 QC Design program]:

Here's a 6.6 Sigma method:

Now here's a 9.6 Sigma method:

And now here's a 22.1 Sigma method:

And what is the result of all these metrics? a recommendation for limits set at 3.5s with two controls per run, rejecting the run anytime a value is outside those limits. Beyond 6.0 Sigma, there is no practical difference in the control rule recommendation.

However, there is a practical effect of these metrics if they are only taking CV into account. A method with a Sigma metric of 22.1 based on CV alone will have a higher tolerable bias than a method with a Sigma metric of 6.0. For example, The CPK method above could tolerate up to 23.2% bias. Clearly, the higher Sigma metrics give you more safety margin while still achieving world class quality.

How do we calculate this tolerable bias? We manipulate the Sigma metric equation, plugging in 6.0 for the metric, and solve for bias:

Sigma = (TE_a – bias)/CV.

6.0* CV = TE_a -bias

bias + (6.0 *CV) = TE_a

bias = TE_a - (6.0*CV)

If we perform these calculations on all the methods, here are the results, not only for 6 Sigma but 5 Sigma as well:

For those metrics that achieve 6 Sigma, the QC Design recommendation is a 1_3.5s control rule with 2 controls.

For those metrics that achieve 5 Sigma, the QC Design recommendation is a 1_2.5s control rule with 2 controls, or you could use a mini-"Westgard Rules" combination of 1_3s/2_2s with 2 controls.

What message can you take away from this data? That the performance of most of these methods is so good that there are rather large margins of safety for bias, despite not knowing what the bias is. This lab can still achieve world class quality with method bias issues.

A combination of data from other sources

Still the question remains, just what is the bias? Are the real world figures within the calculated tolerable bias figures?

Between 29 and 39 Roche Modular instruments participated in a recent (May 2006) testing event, which is close the same time this data was gathered, and we calculated the difference of the Roche Modular group mean from the entire testing group mean. That difference (expressed as a % of the group mean) is going to be our surrogate for bias.

Let us stress: this is not a good practice for a individual laboratory to do. We are mixing data here that really should not be mixed if we were going to make serious decisions about an individual laboratory. But this approach is as close as we can get to bias data for this study.

Below follows the calculations for bias (n/c stands for a method where no PT data was available):

Given the surrogate bias, we find that most of the analytes can be expected to continue to perform at the Six Sigma level. Where the bias number is italicized, that indicates a bias higher than the calculated tolerances, which would indicate a lower than 5 Sigma method, requiring more rules, possibly even more controls. In the eleven instances where methods achieved Six Sigma, there were only three cases where the surrogate bias knocked them down below Six Sigma.

Conclusions

This is an instance where there is a lot of good news. For this instrument, even in the absence of reliable bias data, it can be estimated that most of the analytes can use wide limits like 3.5s with only two controls and still tolerate a method bias.

For a laboratory that had this instrument, it would be advisable to minimize the QC efforts on most of the analytes, concentrating on just the two or three methods at the top of the table that have proven to be troublesome. Indeed, the performance of those methods were found to be outside the manufacturer's specifications (i.e. somehow broken) and this instrument was scheduled for replacement.