QC APPLICATION: A MULTITEST CHEMISTRY ANALYZER Sigma metrics and QC Procedures for Individual Tests An Update, 16 years later The original papers Some more recent data: A fruitbowl approach The CVs Resulting Sigma metrics What do these differences means? Conclusions References

June 2006
Sten Westgard, MS

The Original papers.

Back in 1990, Dr. Westgard and colleagues produced an analysis of the Hitachi Multichemistry AutoAnalyzer:

• Koch DD, Oryall JJ, Quam EF, Feldbruegge DH, Dowd DE, Barry PL, Westgard JO. Selection of medically useful quality control procedures for individual tests done in a multitest analytical system, Clin Chem 1990;36:230-233).
• Westgard JO, Oryall JJ, Koch DD. Predicting effects of QC practices on the cost- effective operation of a multitest analytical system. Clin Chem 1990;36:1760-64.
• Westgard JO, Barry PL. Cost-Effective Quality Control: Managing the Quality and Productivity of Analytical Processes. Washington DC, AACC Press, 1986, p 2.

Given the current insistence that analytical quality is no longer a problem, we thought it would be useful to make a comparison of recent data with this past data.

Some more recent data: a fruitbowl approach

It's very difficult to get an apples-to-apples comparison of performance data. We can't recreate the conditions of 16 years ago. The instruments are several generations more advanced: higher volume, more sophistication, with different capabilities and requirements. The staff that maintains and operates these instruments has changed, too. There are more educational requirements for lab workers these days, even as other factors such as cost pressures and long hours work against the acquisition of the best candidates.

So we've got a fruitbowl approach: there are sets of data here from different laboratories in different countries on different instruments. The amount of data for each study is different; sometimes the data is from hundreds of points, other times only from an initial method validation study. A perfect comparison isn't really possible. Nevertheless, the comparison of then and now should be useful.

We've selected here a number of method validation studies available to us. Several of them have been highlighted on this website before:

2002: A Roche Integra from the Laboratory at St. Joseph Hospital in Houston, Texas, as highlighted previously on this website. The data is listed here as Texas.

2005 AACC poster: Dade RxL instruments from "A Multisite Validation that selected 'Westgard Rules' is efficient and cost effective." David Plaut, Plano, TX; Beth O'Neill, Monongahela Hospital, Morgantown, WV; Shane Holonitch, Quest Laboratories, Denver, Colorado; Sherry Tighe, University Hospital, Cleveland, Ohio; Annette Yamaguchi, Margaret Mary Medical Center, Batesville, Indiana; Linda Griffiths and Wendy Palmeter, Memorial Hospital, Craig, Colorado. The Data is listed here as Ohio and Indiana.

2005 AACC poster: "Assessment of Total Clinical Laboratory Performance: Normalized OPSpecs Charts, Sigma-Values and Patient Test Results", Diler Aslan, Selahattin Sert, Hulya Aybek, Gamze Can Yilmazturk, Pamukkale University, School of Medicine, Biochemistry Department, Denizli Turkey. The data is listed here as Turkey.

2005 AACC poster: "Six Sigma in Clinical Laboratory: comparison of two automated chemistry systems" F.A. Berlitz, M.L.Haussen. Weinmanns Laboratory, Porto Alegre, Brazil. The data is listed here as Brazil.

The CVs

Note first that we are only considering the imprecision figures (smeas or CV, expressed as a percentage). Bias is not measured or compared here. In cases where the studies provided more than one imprecision figure (for example, imprecision figures listed at 2 levels), we report below the lower imprecision figure. Thus, this is the most optomistic comparison possible. To make the results easier to understand, numbers highlighted in red indicate imprecision that is worse than the original paper, and nubmers highlighted in blue indicate impercision that is better than the original paper.

The quick summary is that there are 30 cases where imprecision is worse, 38 cases where imprecision is better, and 1 case where imprecision is the same. A few analytes jump out, like Total protein and Albumin, where all the new studies show improvements over the original paper.

Resulting Sigma metrics

Using the Six Sigma calculations, we can convert these imprecision figures into Sigma metrics. Again, note that we are not taking any bias into account.

Since these metrics do not take bias into account, they are optomistic (and higher) than other metrics reported on this website or in other literature. Nevertheless, it's worth noting that even back in 1990, the original paper reported that 11 of the 18 analytes performed at Six Sigma or higher. Only 7 analytes were less than Six Sigma, and of those, only 2 of those analytes were below Five Sigma.

Since the metrics are based solely on the imprecision, the distribution of the metrics are the same as earlier table: there are 30 cases where the Sigma metrics are worse, 38 cases where the Sigma metrics are better, and 1 case where the Sigma metric is the same. What's more interesting to note is that there are 50 cases where performance is higher than Six Sigma, 8 cases where the metric is between 5 and 6 sigma, 5 cases where the metric is between 4 and 5, 1 case where the metric is between 3 and 4, and finally 5 cases where the metrics are below 3 sigma.

What do these differences mean?

Beyond the numbers, there is a practical significance to the metrics. The QC procedures necessary for the the analytes depend on the Sigma metrics. Thus, the cases where Sigma is above 6, QC is very easy - go to the mininum number of controls (2, as CLIA specifies) and wide limits such as 3s or 3.5s.

Here we see that the QC procedures necessary are mostly the same. 40 cases where the QC procedure didn't change, 12 cases where less QC is needed than the original paper (notably concentrated in two tests, Albumin and Creatinine). Most troubling, though, are the 17 cases where more QC is needed than the original paper.

Conclusions

The data here does not support the conclusion that instrument performance have gotten so good that QC is no longer needed or can be reduced. Indeed, the cases where performance is worse outnumber the cases where performance is better.

Remember, these are the optomistic figures. If we took bias into account, the metrics would be lower, and the QC needs would be greater.

References

1. Koch DD, Oryall JJ, Quam EF, Feldbruegge DH, Dowd DE, Barry PL, Westgard JO. Selection of medically useful quality control procedures for individual tests done in a multitest analytical system, Clin Chem 1990;36:230-233).
2. Westgard JO, Oryall JJ, Koch DD. Predicting effects of QC practices on the cost- effective operation of a multitest analytical system. Clin Chem 1990;36:1760-64.
3. Westgard JO, Barry PL. Cost-Effective Quality Control: Managing the Quality and Productivity of Analytical Processes. Washington DC, AACC Press, 1986, p 2.