Tools, Technologies and Training for Healthcare Laboratories

AU5800 in an Indian laboratory

A recent study of the AU 5800 in Hyderabad, India raises the question about the quality of the major diagnostic instruments when they are in a more challenging environment.

Sigma-metric Analysis of AU 5800 in India

Sten Westgard, MS
May 2019

[Note: This QC application is an extension of the lesson From Method Validation to Six Sigma: Translating Method Performance Claims into Sigma Metrics. This article assumes that you have read that lesson first, and that you are also familiar with the concepts of QC Design, Method Validation, and Six Sigma metrics. If you aren't, follow the link provided.]

In March of 2019, a new study was published evaluating the performance of a Beckman Coulter AU 5800:

Evaluation of Quality Assurance in a New Clinical Chemistry Laboratory by Six Sigma Metrics, Suresh Babu Ganji, Suneetha Revupalli, Journal of Clinical and Diagnostic Research, 2019 March, Vol 13(3): BC04-BC07.

The Imprecision, Bias and Sigma-metric Data

"The present retrospective study was conducted at the Clinical Biochemistry Laboratory of a Government Hospital: the Department of Biochemistry, Niloufer Hospital for Women and Children, Hyderabad, Telangana, India. IQC and EQAS data of six months (February 2018 to July 2018) were analysed retrospectively for the 16 most common parameters.... Authors ran both levels of controls i.e. L1 normal range and L2 abnormal range controls on Beckman Coulter AU5800 fully auto analyser, using Bio-Rad controls."

"Bias was computed from the External Quality Assurance records (a monthly EQAS program run by Bio-Rad) of the peer group data.... CV was determined from the calculated laboratory mean and calculated SD procured from the IQC data over the last six months."

The TEa goals used include current CLIA goals, as well as some Ricos goals, and some EuBIVAS goals that were more recently determined.

TEST TEa Source TEa % Bias CV
Albumin CLIA 10 1.1 5.2
Albumin 10% 10 1.1 5.5
Alk Phos CLIA 30 4.0 7.7
Alk Phos 30% 30 4.0 5.9
ALT EuBivas 14.4 2.7 6.9
ALT 14.4% 14.4 2.7 4.2
Amylase EuBivas 13.40 3.3 5.6
Amylase 13.4% 13.40 3.3 5.1
AST EuBivas 13.40 3.7 5.8
AST 13.4% 13.40 3.7 5.0
Bilirubin, Direct Ricos 44.50 1.4 4.1
Bilirubin, Direct 44.50% 44.50 1.4 5.4
Bilirubin, Total CLIA 20% or 30.00 2.3 3.7
Bilirubin, Total 0.4 mg/dL 20.00 2.3 4.2
Calcium CLIA 11 2.8 3.8
Calcium 1 mg/dL 11 2.8 4.0
Cholesterol CLIA 10 3.0 4.4
Cholesterol 10% 10 3.0 4.2
Creatinine CLIA 15% or 15.00 2.3 4.0
Creatinine 0.3 mg/dL 15.00 2.3 6.9
Glucose CLIA 10% or 10.00 3.4 4.2
Glucose 6 mg/dL 10.00 3.4 5.2
HDL CLIA 30.00 2.5 4.3
HDL 30% 30.00 2.5 4.5
Phosphorous CAP 10.7% or 10.70 3.5 4.7
Phosphorous 0.3 mg/dL 10.70 3.5 4.6
Protein, Total CAP 10.00 5.8 12.0
Protein, Total 10% 10.00 5.8 10.6
Urea Nitrogen CLIA 9% or 9.00 3.5 5.9
Urea Nitrogen 2.0 mg/dL 9.00 3.5 7.0
Uric Acid CLIA 17.00 2.0 3.1
Uric Acid 17% 17.00 2.0 4.2

Yes, that is a whole lot of numbers!

Nevertheless, what do all these numbers mean? In the absence of context, it's hard to know.

So let's calculate the Sigma-metrics.

Sigma-metric calculations for the AU5800

Remember the equation for Sigma metric is (TEa - bias) / CV:

For a 10% quality requirement, for Albumin on the low level of AU5800, the equation is (10 - 1.1) / 5.2 = 1.7

For a 10% quality requirement, for Albumin on the high level of AU5800, the equation is (10 - 1.1) / 5.5 = 1.6

The metrics are displayed along the right column.

TEST TEa Source TEa % Bias CV Sigma
Albumin CLIA 10 1.1 5.2 1.7
Albumin 10% 10 1.1 5.5 1.6
Alk Phos CLIA 30 4.0 7.7 3.4
Alk Phos 30% 30 4.0 5.9 4.4
ALT EuBivas 14.4 2.7 6.9 1.7
ALT 14.4% 14.4 2.7 4.2 2.8
Amylase EuBivas 13.40 3.3 5.6 1.8
Amylase 13.4% 13.40 3.3 5.1 1.97
AST EuBivas 13.40 3.7 5.8 1.7
AST 13.4% 13.40 3.7 5.0 1.9
Bilirubin, Direct Ricos 44.50 1.4 4.1 10.5
Bilirubin, Direct 44.50% 44.50 1.4 5.4 7.9
Bilirubin, Total CLIA 20% or 30.00 2.3 3.7 7.4
Bilirubin, Total 0.4 mg/dL 20.00 2.3 4.2 4.2
Calcium CLIA 11 2.8 3.8 2.1
Calcium 1 mg/dL 11 2.8 4.0 2.1
Cholesterol CLIA 10 3.0 4.4 1.6
Cholesterol 10% 10 3.0 4.2 1.7
Creatinine CLIA 15% or 15.00 2.3 4.0 3.2
Creatinine 0.3 mg/dL 15.00 2.3 6.9 1.9
Glucose CLIA 10% or 10.00 3.4 4.2 1.6
Glucose 6 mg/dL 10.00 3.4 5.2 1.3
HDL CLIA 30.00 2.5 4.3 6.3
HDL 30% 30.00 2.5 4.5 6.1
Phosphorous CAP 10.7% or 10.70 3.5 4.7 1.5
Phosphorous 0.3 mg/dL 10.70 3.5 4.6 1.6
Protein, Total CAP 10.00 5.8 12.0 0.4
Protein, Total 10% 10.00 5.8 10.6 0.4
Urea Nitrogen CLIA 9% or 9.00 3.5 5.9 0.9
Urea Nitrogen 2.0 mg/dL 9.00 3.5 7.0 0.8
Uric Acid CLIA 17.00 2.0 3.1 4.8
Uric Acid 17% 17.00 2.0 4.2 3.6

Yes, there are a lot of low Sigma-metrics. There are a few high Sigma-metrics, but the majority of the numbers are below 3 Sigma, typically considered the minimum acceptable performance.

Summary of Performance by Sigma-metrics Method Decision Chart

We can make visual assessments of this performance using a Normalized Sigma-metric Method Decision Chart:

2019 india au5800 nmedx

Overall, it seems like imprecision here is high and a majority of these assays are worse than 3 Sigma.

Summary of QC Design by Normalized OPSpecs chart - using primarily CLIA Goals

The benefit of the Sigma-metric approach is that labs can do more than assess their quality, they can act on it. Using OPSpecs charts, they can actually optimize their QC procedures for each test. In this case, they can use the data to try and mitigate the risk of poor performance.

2019 india au5800 nopspecs

For most of these points, we will need the maximum "Westgard Rules" with 8 control measurements per run. Put bluntly, there is no practical statistical quality control procedure that can help. There aren't enough "Westgard rules" to keep some of these methods in control. If we would go further to determine QC frequency using Sigma-metrics, we would find that we can't afford to run QC often enough to properly monitor these methods.

 

Conclusion

The paper reviewed more than a half dozen recent studies of instrument performance in India. Of these publications, the AU5800 had the worst performance than all the other studied instruments. The AU5800 had the highest number of below 3 Sigma assays of any other paper. That's not a great distinction.