Sigma Metric Analysis
AU5800 in an Indian laboratory
A recent study of the AU 5800 in Hyderabad, India raises the question about the quality of the major diagnostic instruments when they are in a more challenging environment.
Sigma-metric Analysis of AU 5800 in India
Sten Westgard, MS
May 2019
- The Precision, Comparison and Sigma-metric data
- Summary of Performance according to CLIA Goals by Sigma-metrics Normalized Method Decision charts
- QC Implications of AU5800 Performance by Normalized OPSpecs chart
- Conclusion
[Note: This QC application is an extension of the lesson From Method Validation to Six Sigma: Translating Method Performance Claims into Sigma Metrics. This article assumes that you have read that lesson first, and that you are also familiar with the concepts of QC Design, Method Validation, and Six Sigma metrics. If you aren't, follow the link provided.]
In March of 2019, a new study was published evaluating the performance of a Beckman Coulter AU 5800:
Evaluation of Quality Assurance in a New Clinical Chemistry Laboratory by Six Sigma Metrics, Suresh Babu Ganji, Suneetha Revupalli, Journal of Clinical and Diagnostic Research, 2019 March, Vol 13(3): BC04-BC07.
The Imprecision, Bias and Sigma-metric Data
"The present retrospective study was conducted at the Clinical Biochemistry Laboratory of a Government Hospital: the Department of Biochemistry, Niloufer Hospital for Women and Children, Hyderabad, Telangana, India. IQC and EQAS data of six months (February 2018 to July 2018) were analysed retrospectively for the 16 most common parameters.... Authors ran both levels of controls i.e. L1 normal range and L2 abnormal range controls on Beckman Coulter AU5800 fully auto analyser, using Bio-Rad controls."
"Bias was computed from the External Quality Assurance records (a monthly EQAS program run by Bio-Rad) of the peer group data.... CV was determined from the calculated laboratory mean and calculated SD procured from the IQC data over the last six months."
The TEa goals used include current CLIA goals, as well as some Ricos goals, and some EuBIVAS goals that were more recently determined.
TEST | TEa Source | TEa | % Bias | CV |
Albumin | CLIA | 10 | 1.1 | 5.2 |
Albumin | 10% | 10 | 1.1 | 5.5 |
Alk Phos | CLIA | 30 | 4.0 | 7.7 |
Alk Phos | 30% | 30 | 4.0 | 5.9 |
ALT | EuBivas | 14.4 | 2.7 | 6.9 |
ALT | 14.4% | 14.4 | 2.7 | 4.2 |
Amylase | EuBivas | 13.40 | 3.3 | 5.6 |
Amylase | 13.4% | 13.40 | 3.3 | 5.1 |
AST | EuBivas | 13.40 | 3.7 | 5.8 |
AST | 13.4% | 13.40 | 3.7 | 5.0 |
Bilirubin, Direct | Ricos | 44.50 | 1.4 | 4.1 |
Bilirubin, Direct | 44.50% | 44.50 | 1.4 | 5.4 |
Bilirubin, Total | CLIA 20% or | 30.00 | 2.3 | 3.7 |
Bilirubin, Total | 0.4 mg/dL | 20.00 | 2.3 | 4.2 |
Calcium | CLIA | 11 | 2.8 | 3.8 |
Calcium | 1 mg/dL | 11 | 2.8 | 4.0 |
Cholesterol | CLIA | 10 | 3.0 | 4.4 |
Cholesterol | 10% | 10 | 3.0 | 4.2 |
Creatinine | CLIA 15% or | 15.00 | 2.3 | 4.0 |
Creatinine | 0.3 mg/dL | 15.00 | 2.3 | 6.9 |
Glucose | CLIA 10% or | 10.00 | 3.4 | 4.2 |
Glucose | 6 mg/dL | 10.00 | 3.4 | 5.2 |
HDL | CLIA | 30.00 | 2.5 | 4.3 |
HDL | 30% | 30.00 | 2.5 | 4.5 |
Phosphorous | CAP 10.7% or | 10.70 | 3.5 | 4.7 |
Phosphorous | 0.3 mg/dL | 10.70 | 3.5 | 4.6 |
Protein, Total | CAP | 10.00 | 5.8 | 12.0 |
Protein, Total | 10% | 10.00 | 5.8 | 10.6 |
Urea Nitrogen | CLIA 9% or | 9.00 | 3.5 | 5.9 |
Urea Nitrogen | 2.0 mg/dL | 9.00 | 3.5 | 7.0 |
Uric Acid | CLIA | 17.00 | 2.0 | 3.1 |
Uric Acid | 17% | 17.00 | 2.0 | 4.2 |
Yes, that is a whole lot of numbers!
Nevertheless, what do all these numbers mean? In the absence of context, it's hard to know.
So let's calculate the Sigma-metrics.
Sigma-metric calculations for the AU5800
Remember the equation for Sigma metric is (TEa - bias) / CV:
For a 10% quality requirement, for Albumin on the low level of AU5800, the equation is (10 - 1.1) / 5.2 = 1.7
For a 10% quality requirement, for Albumin on the high level of AU5800, the equation is (10 - 1.1) / 5.5 = 1.6
The metrics are displayed along the right column.
TEST | TEa Source | TEa | % Bias | CV | Sigma |
Albumin | CLIA | 10 | 1.1 | 5.2 | 1.7 |
Albumin | 10% | 10 | 1.1 | 5.5 | 1.6 |
Alk Phos | CLIA | 30 | 4.0 | 7.7 | 3.4 |
Alk Phos | 30% | 30 | 4.0 | 5.9 | 4.4 |
ALT | EuBivas | 14.4 | 2.7 | 6.9 | 1.7 |
ALT | 14.4% | 14.4 | 2.7 | 4.2 | 2.8 |
Amylase | EuBivas | 13.40 | 3.3 | 5.6 | 1.8 |
Amylase | 13.4% | 13.40 | 3.3 | 5.1 | 1.97 |
AST | EuBivas | 13.40 | 3.7 | 5.8 | 1.7 |
AST | 13.4% | 13.40 | 3.7 | 5.0 | 1.9 |
Bilirubin, Direct | Ricos | 44.50 | 1.4 | 4.1 | 10.5 |
Bilirubin, Direct | 44.50% | 44.50 | 1.4 | 5.4 | 7.9 |
Bilirubin, Total | CLIA 20% or | 30.00 | 2.3 | 3.7 | 7.4 |
Bilirubin, Total | 0.4 mg/dL | 20.00 | 2.3 | 4.2 | 4.2 |
Calcium | CLIA | 11 | 2.8 | 3.8 | 2.1 |
Calcium | 1 mg/dL | 11 | 2.8 | 4.0 | 2.1 |
Cholesterol | CLIA | 10 | 3.0 | 4.4 | 1.6 |
Cholesterol | 10% | 10 | 3.0 | 4.2 | 1.7 |
Creatinine | CLIA 15% or | 15.00 | 2.3 | 4.0 | 3.2 |
Creatinine | 0.3 mg/dL | 15.00 | 2.3 | 6.9 | 1.9 |
Glucose | CLIA 10% or | 10.00 | 3.4 | 4.2 | 1.6 |
Glucose | 6 mg/dL | 10.00 | 3.4 | 5.2 | 1.3 |
HDL | CLIA | 30.00 | 2.5 | 4.3 | 6.3 |
HDL | 30% | 30.00 | 2.5 | 4.5 | 6.1 |
Phosphorous | CAP 10.7% or | 10.70 | 3.5 | 4.7 | 1.5 |
Phosphorous | 0.3 mg/dL | 10.70 | 3.5 | 4.6 | 1.6 |
Protein, Total | CAP | 10.00 | 5.8 | 12.0 | 0.4 |
Protein, Total | 10% | 10.00 | 5.8 | 10.6 | 0.4 |
Urea Nitrogen | CLIA 9% or | 9.00 | 3.5 | 5.9 | 0.9 |
Urea Nitrogen | 2.0 mg/dL | 9.00 | 3.5 | 7.0 | 0.8 |
Uric Acid | CLIA | 17.00 | 2.0 | 3.1 | 4.8 |
Uric Acid | 17% | 17.00 | 2.0 | 4.2 | 3.6 |
Yes, there are a lot of low Sigma-metrics. There are a few high Sigma-metrics, but the majority of the numbers are below 3 Sigma, typically considered the minimum acceptable performance.
Summary of Performance by Sigma-metrics Method Decision Chart
We can make visual assessments of this performance using a Normalized Sigma-metric Method Decision Chart:
Overall, it seems like imprecision here is high and a majority of these assays are worse than 3 Sigma.
Summary of QC Design by Normalized OPSpecs chart - using primarily CLIA Goals
The benefit of the Sigma-metric approach is that labs can do more than assess their quality, they can act on it. Using OPSpecs charts, they can actually optimize their QC procedures for each test. In this case, they can use the data to try and mitigate the risk of poor performance.
For most of these points, we will need the maximum "Westgard Rules" with 8 control measurements per run. Put bluntly, there is no practical statistical quality control procedure that can help. There aren't enough "Westgard rules" to keep some of these methods in control. If we would go further to determine QC frequency using Sigma-metrics, we would find that we can't afford to run QC often enough to properly monitor these methods.
Conclusion
The paper reviewed more than a half dozen recent studies of instrument performance in India. Of these publications, the AU5800 had the worst performance than all the other studied instruments. The AU5800 had the highest number of below 3 Sigma assays of any other paper. That's not a great distinction.