Sigma Metric Analysis
Indian Pilot Study in Sigma-metrics
In a 2011 study, an India hospital laboratory undertook a quality assessment of their clinical chemistry methods using Sigma-metrics. Is laboratory method quality the same the world over?
- The Precision and Comparison data
- Determine quality requirements at the measured decision levels
- Calculate Sigma metrics
- Summary of Performance by Sigma-metrics chart and OPSpecs Chart
- Conclusion
FEBRUARY 2012
Sten Westgard, MS
 |
[Note: This QC application is an extension of the lesson From Method Validation to Six Sigma: Translating Method Performance Claims into Sigma Metrics. This article assumes that you have read that lesson first, and that you are also familiar with the concepts of QC Design, Method Validation, and Six Sigma. If you aren't, follow the link provided.] |  |
This application looks at a paper from a 2011 issue of The Indian Journal of Clinical Biochemistry which examined the performance of a clinical chemistry instrument:
Application of Sigma Metrics for the Assessment of Quality Assurance in Clinical Biochemistry Laboratory in India: A Pilot Study, Bhawna Singh, Binita Goswami, Vinod Kumar Gupta, Ranjna Chawla, Venkatesan Mallika, Ind J Clin Biochem (Apr-June 2011) 26(2):131-135.
This study was conducted at GB Pant hospital, a 600-bed tertiary hospital, over a period of 6 months. The control materials used were from Randox.
The Imprecision and Bias Data
The imprecision data used in the study represents not a short-term method validation study, but routine performance data collected over a longer period of time:
"Internal statitistical QC data was extracted from the... biochemistry analyzer... data for the period of 6 months from July 2009 to December 2009. Control materials were obtained from Randox.... Both normal (L2) and pathological (L3) levels of QC materials were assayed before commencing reporting of patient samples every day. Next QC scheduled event was undertaken after running 50 patient samples (bracketed QC)."
Thus, we have two estimates of imprecision at different levels for each of the assays.
An average bias was also calculated using 6 months of data from an External Quality Assurance Scheme (EQAS) - specifically, RIQAS. One estimate of bias was made, an average of the difference between the laboratory's mean and the mean of all the laboratories using the same instrument and method.
Thus, we have good estimates of bias and imprecision
| Method | CV% | Avg. Bias% |
| Urea, mg/dL | 3.08% | 4.2% |
| 3.6% | ||
| Creatinine, mg/dL | 0.87% | 6.3% |
| 1.89% | ||
| Cholesterol, mg/dL | 2.22% | 2.6% |
| 2.65% | ||
| Triglycerides, mg/dL | 2.55% | 2.9% |
| 3.25% | ||
| HDL, mg/dL | 3.4% | 8.3% |
| 7.45% | ||
| AST, IU/L | 2.6% | 4.5% |
| 2.2% | ||
| ALT, IU/L | 4.2% | 3.4% |
| 3.12% | ||
| ALP, IU/L | 7% | 7.7% |
| 6.5% | ||
| Total bilirubin, mg/dL | 3.4% | 7.9% |
| 3.5% | ||
| Total protein, g/dL | 1.89% | 2.6% |
| 2.06% | ||
| Creatinine kinase, IU/L | 3.02% | 4.75% |
| 2.8% | ||
| Amylase, IU/L | 2.2% | 5.4% |
| 2.09% | ||
| Sodium, mEq/L | 1.23% | 1.91% |
| 1.3% | ||
| Potassium, mEq/L | 1.46% | 1.82% |
| 1.6% |
A lot of numbers, right? We have two levels of control materials, so we have imprecision estimates for both of those levels. Using the regression equation, we can estimate the bias at each of those levels.
Looking at the raw numbers, you may find it difficult to judge the method performance. From experience, you might be able to tell when a particular method CV is high or low. But the numbers themselves don't tell the story.
If we want an objective assessment, we can set analytical goals - specify quality requirements - and use those to calculate the Sigma-metric.
Determine Quality Requirements
Now that we have our imprecision and bias data, we're almost ready to calculate our Sigma-metrics. But we're missing one key thing: the analytical quality requirement.
In this study, the laboratory specifically chose CLIA quality requirements.
| Method | CV% | Avg. Bias% | Quality Requirement |
| Urea, mg/dL | 3.08% | 4.2% | 9% |
| 3.6% | |||
| Creatinine, mg/dL | 0.87% | 6.3% | 15% |
| 1.89% | |||
| Cholesterol, mg/dL | 2.22% | 2.6% |
10% |
| 2.65% | |||
| Triglycerides, mg/dL | 2.55% | 2.9% | 25% |
| 3.25% | |||
| HDL, mg/dL | 3.4% | 8.3% | 30% |
| 7.45% | |||
| AST, IU/L | 2.6% | 4.5% | 20% |
| 2.2% | |||
| ALT, IU/L | 4.2% | 3.4% | 20% |
| 3.12% | |||
| ALP, IU/L | 7% | 7.7% | 30% |
| 6.5% | |||
| Total bilirubin, mg/dL | 3.4% | 7.9% | 20% |
| 3.5% | |||
| Total protein, g/dL | 1.89% | 2.6% | 10% |
| 2.06% | |||
| Creatinine kinase, IU/L | 3.02% | 4.75% | 30% |
| 2.8% | |||
| Amylase, IU/L | 2.2% | 5.4% | 30% |
| 2.09% | |||
| Sodium, mEq/L | 1.23% | 1.91% | 5% |
| 1.3% | |||
| Potassium, mEq/L | 1.46% | 1.82% | 6% |
| 1.6% |
Calculate Sigma metrics
Now the pieces are in place. Remember the equation for Sigma metric is (TEa - bias) / CV.
Example calculation: for Urea for the CLIA 9% quality requirement, at the level of 5.3 mg/dL, 3.08% imprecision, 4.2% bias:
(9 - 4.2) / 3.08 = 4.8 / 3.08 = 1.5
Now, at the level of 12.5 mg/dL, again using 9% as the CLIA quality requirement, with 4.2% bias, 3.6% imprecision:
(9 - 4.2) / 3.6 = 4.8 / 3.6 = 1.3
Recall that in industries outside healthcare, 3.0 Sigma is the minimum performance for routine use. 6.0 Sigma and higher is considered world class performance. In the table below, we'll highlight the <3.0 Sigma in red, and highlight the >6.0 Sigma in green.
| Method | CV% | Avg. Bias% | Quality Requirement |
Sigma metric |
| Urea, mg/dL | 3.08% | 4.2% | 9% | 1.5 |
| 3.6% | 1.3 | |||
| Creatinine, mg/dL | 0.87% | 6.3% | 15% | 10.0 |
| 1.89% | 4.6 | |||
| Cholesterol, mg/dL | 2.22% | 2.6% | 10% | 3.3 |
| 2.65% | 2.79 | |||
| Triglycerides, mg/dL | 2.55% | 2.9% | 25% | 8.6 |
| 3.25% | 6.8 | |||
| HDL, mg/dL | 3.4% | 8.3% | 30% | 6.3 |
| 7.45% | 2.9 | |||
| AST, IU/L | 2.6% | 4.5% | 20% | 5.9 |
| 2.2% | 7.0 | |||
| ALT, IU/L | 4.2% | 3.4% | 20% | 3.9 |
| 3.12% | 5.3 | |||
| ALP, IU/L | 7% | 7.7% | 30% | 3.2 |
| 6.5% | 3.4 | |||
| Total bilirubin, mg/dL | 3.4% | 7.9% | 20% | 3.5 |
| 3.5% | 3.4 | |||
| Total protein, g/dL | 1.89% | 2.6% | 10% | 3.9 |
| 2.06% | 3.5 | |||
| Creatinine kinase, IU/L | 3.02% | 4.75% | 30% | 8.3 |
| 2.8% | 9.0 | |||
| Amylase, IU/L | 2.2% | 5.4% | 30% | 11.2 |
| 2.09% | 11.7 | |||
| Sodium, mEq/L | 1.23% | 1.91% | 5% | 2.5 |
| 1.3% | 2.3 | |||
| Potassium, mEq/L | 1.46% | 1.82% | 6% | 2.8 |
| 1.6% | 2.6 |
Again, the numbers can be overwhelming.
There is a mixed batch of verdicts, with some world class methods and some methods that need serious improvement.
Summary of Performance by Sigma-metrics Method Decision Chart and QC Design by OPSpecs chart
If the numbers are too much to digest, we can put this into a graphic format with a Six Sigma Method Decision Chart. Here's the chart for CLIA specifications for allowable total error
Now the question becomes, what would the laboratory do if this instrument was in routine operation? What QC would be necessary to assure the level of quality required for use of the tests? In this case, we use the same data, but plot the methods on an OPSpecs (Operating Specifications) chart.
For analytes like AST, CK, Triglycerides, and Amylase, we can use wider limits than we're probably used to (3s, 3.5s, even 4s). For analytes like Total protein and Total Bilirubm, ALP, ALT, and HDL, the news is not so good. We probably need to implement robust QC procedures like like an extensive set of "Westgard Rules." For Sodium, Potassium, Cholesterol, and Urea, the news is more concerning. Below 3 Sigma, there isn't enough QC to bring the assays back into acceptable performance. Even if the laboratory decides to run the full "Westgard Rules", that won't be sufficient. The method has to be improved in order to raise the performance to an acceptable level.
As the study notes: "[For the] worst performers in our laboratory, diverting special attention to them is mandatory for revamping performance. It is of utmost importance to explore urea method performance and practice stringent maintenance of ISE unit to alleviate inaccuracies resulting in poor performance of ISE module."
Conclusion
With today's modern analyzers, you can't take for granted that performance is always great. Even if many of the methods have great performance, that doesn't guarantee that all methods are acceptable. In the drive toward large test menus, inevitably compromises on performance are made. So bad methods get on the same box with great methods. For laboratories, it's important to identify which methods you don't have to worry about, and which methods require extra attention.
It's also worth noting that CLIA requirements are often considered too lenient. This case shows that for some methods, that's true. For ISEs, however, even the CLIA limits are pretty tight and demanding. If we had applied the desirable specifications for total allowable error (sometimes called the Ricos Goals), the results here may have been even more concerning.