Sigma Metric Analysis

Our first look at Abbott's newest, biggest chemistry instrument showed interesting Sigma performance. So we decided to take a second look, using 2007 AACC conference poster data. Does an on-site evaluation corroborate the manufacturer's evaluation?

• The Precision and Comparison data
• Calculate bias at the critical decision level
• Determine quality requirements at the critical decision level
• Calculate Sigma metrics
• Summary of Performance by Normalized OPSpecs Chart
• Comparison of Manufacturer vs Site Metrics
• Conclusions

October 2007

[Note: This QC application is an extension of the lesson From Method Validation to Six Sigma: Translating Method Performance Claims into Sigma Metrics. This article assumes that you have read that lesson first, and that you are also familiar with the concepts of QC Design, Method Validation, and Six Sigma. If you aren't, follow the links provided.]

This application comes from another set of posters from the 2007 AACC/ASCLS annual meeting. This conference has become about the only source of method evaluation data on instrument performance.

In this example, we're going to take a look at a method validation study performed for the Abbott Architect c16000 Clinical Chemistry System (E-45: S. A. Gough, L. French-English, Medical Laboratory Staff, Site Evaluation of the Abbott Architect c16000 Clinical Chemistry).

The Precision and Comparison data

According to the abstract, this evaluation study adhered closely to CLSI (formerly NCCLS) protocols, calculating total precision using EP10, and method comparison using EP9 with the Roche Modular system as the reference method. All that we need to do is select which estimates to use, supply the quality requirements and calculate the Sigma metrics.

Imprecision Estimates:

Comparison of Methods Data:

At this point, remember the following: the correlation coefficient is not the key statistic here. The impressive values of the correlation coefficient merely tell us that linear regression is sufficient for these analytes (for those r values below 0.95, other forms of regression like Deming or Passing-Bablock are preferable, but in this case, are not available).

Calculate bias at the critical decision level

Now we take the comparison of methods data and set those equations at one of the levels covered in the imprecision studies. Solving those equations will give us bias estimates. For this illustration, we're going to select the levels where imprecision was highest, that is, a worst-case scenario approach.

Using glucose as an example, let's see how to calculating bias:

((slope*level) + Y_Intercept) - level) / level = % bias

((1.02 * 91.0) - 0.8) - 91) / 91 = ((92.82 - 0.8) - 91 ) / 91

(92.02-91) / 91 = 1.02 / 91 = 0.0112 * 100 = 1.1%

Determine the quality requirements at the critical decision level

Now that we have both bias and CV estimates, we are almost ready to calculate the Sigma metrics for these analytes. The last thing we need is the quality requirement for each method. CLIA provides most of the quality requirements we need, but in several cases, we need to transform those requirements into useful percentages.

Note that some of these requirements are extremely tight. Sodium and Potassium have a fixed target values, regardless of the level of the test, which creates very small windows of opportunity. Chloride's requirement is also quite small.

Calculate Sigma metrics

Now we have all the pieces in place.

Remember the equation for Sigma metric is (TE_a - bias) / CV:

For Glucose, (10.0 - 1.1) / 1.3 = 6.85

Not much needs to be said here. 7 out the 9 methods are world class. Even the lowest Sigma metrics (for Creatinine and Urea) are nearly at Six Sigma performance.

Summary of Performance by Normalized OPSpecs chart

Here's a graphic depiction of these analytes, normalized so they can be presented together on a Normalized OPSpecs chart:

As you can see, most of the operating points are on "solid ground." For many of these analytes, there is a great deal of wiggle room (allowable variation) available to them without jeopardizing world class performance.

Note also that the Normalized OPSpecs chart displays some unusual rules (see the key at right). For seven of the analytes on the Architect c16000, a single control with control limits set at three or even four times the standard deviation would provide more than sufficient error detection of medically important errors. This is one of those (somewhat rare) areas where the CLIA minimums are over-controlling those methods. Since CLIA requires at least 2 controls per run, the other solutions displayed here show control limits set at 3.5 and 4 times the standard deviation. Even the two "problem" methods, Urea and Creatinine, can still be adequately controlled with 2 controls and limits set at 3s.

All of these QC procedures would essentially eliminate false rejection problems. If you set your limits that wide, you will only get a flag when there is a real problem. But remember, once you get that flag, you must do something about it (trouble-shoot the method), not just repeat the control.

Comparison of Sigma Metrics from Manufacturer's Evaluation and Site Evaluation

We are lucky enough to have two evaluations of this instrument. Two sets of data were presented as posters at the AACC/ASCLS conference. The former set was detailed in an earlier QC Application While not every analyte was shared by the two evaluations, there were five tests that were analyzed in common by the two studies. Given the chance to compare Sigma metrics, it is interesting to see if there are any similarities or differences:

There is a definite consistency of metrics. In theory, this is exactly what should occur; the instruments should behave the same, regardless of setting, particularly given the level of automation and sophistication. The only real difference here is between the Abbott measurement and the Site measurement of Urea performance. What's more, the difference is good news - the laboratory performed better than the manufacturer. Indeed, the manufacturer metrics are all lower than the site evaluation, suggesting a conservative approach adopted by their testing.

Finally, the biggest difference, of 23.5 versus 14.8 Sigma for Total Protein, is actually an irrelevant difference. Once you've exceeded Six Sigma, there is no practical significance to a metric of 10, 14, 23 or higher (unless the quality requirement is tightened - then the recalculated metric will be significantly different).

Conclusions

As we have said before, it is rare to find so much good news in a method evaluation study. If this were your laboratory and your data, it would be time to celebrate.