Tools, Technologies and Training for Healthcare Laboratories

Sigma-metrics of an AU 680

In 2016, Clinical Biochemistry published a study evaluating the Sigma-metrics of the Beckman Coulter Olympus AU 680 using biological variation goals.

Sigma-metrics comparison of an AU 680

February 2017
Sten Westgard, MS

[Note: This QC application is an extension of the lesson From Method Validation to Six Sigma: Translating Method Performance Claims into Sigma Metrics. This article assumes that you have read that lesson first, and that you are also familiar with the concepts of QC Design, Method Validation, and Six Sigma. If you aren't, follow the link provided.]

Clinical Biochemistry published a study conducted by the Clinical Biochemistry Laboratory at the National Hospital of Pediatrics, in Hanoi, Vietnam, that evaluated the Beckman Coulter (formerly Olympus) AU 680.

Practical application of biological variation and Sigma metrics quality models to evaluate 20 chemistry analytes on the Beckman Coulter AU680 Tran MTC, Hoang KT, Greaves RF, Clinical Biochemistry 2016 Nov 49(16-17): 1259-1266.

In this study, the instrument performance was compared against the biologic variation criteria developed primarily by the Spanish EQA groups, popularly known as the "Ricos goals," after Dr. Carmen Ricos, one of the strongest proponents of their use. This study did not only an evaluation of the imprecision and bias, but also total error and Sigma-metrics.

What we'll add to this study is a graphic analysis of the same instrument data, this time using CLIA goals, as well as a focused graph that just looks at one critical level of performance for each test.

The Imprecision and Bias Data

"Commericial lyophilized control samples of human origin were used for evaluation. There were purchased from Randox Laboratories... and included Randox human assay control 2 lot 887 UN and Randox human assay control 3 lot 649UN...Within-run and day-to-day imprecision was used to determine the extent of random error. Within-run imprecision was determined on 20 consecutive measurements of different analyte concentrations in the respective control sera. Day-to-day imprecision (CVa) was determined by measuring the concentration of analytes in the control sera of different concentration ranges over a 20 day period."

Bias calculations from external quality assurance data. "Commercial pool serum samples from Randox International Quality Assurance Scheme (RIQAS) Monthly Chemistry Program were used for assessment. Bias of measurement of all analytes... was determined by running the serum samples from the RIQAS Monthly Chemistry Program (six EQA samples) over a six month period, among these samples four samples were analysed twice independently (so that we got 10 results)." 

Then Passing-Bablock regression was used to determine the systematic error between the test method and the instrument group mean. .

TEST Group or PeerMean slope y-int Bias% CV%
Albumin 39.80 1.03 -1.59 0.69 2.1
Albumin 27.80 1.03 -1.59 2.42 1.4
Alkaline Phosphatase 195.00 0.99 1.50 0.5 3.1
Alkaline Phosphatase 418.00 0.99 1.50 0.9 2.1
ALT 36.00 1.01 -1.11 2.3 3.8
ALT 134.00 1.01 -1.11 0.0 1.8
Amylase 83.00 1.06 -11.10 7.2 2.2
Amylase 291.00 1.06 -11.10 2.4 2.9
AST 36.00 1.04 -1.03 0.8 3.4
AST 152.00 1.04 -1.03 3.0 2.2
Bilirubin, Total 32.20 0.98 -0.64 4.2 1.8
Bilirubin, Total 93.60 0.98 -0.64 2.9 2.3
Calcium 2.26 1.08 -0.20 0.94 1.3
Calcium 3.11 1.08 -0.20 1.50 1.2
Chloride 101.00 1.00 -0.07 0.4 1.0
Chloride 114.00 1.00 -0.07 0.4 0.7
Creatinine Kinase (CK) 230.00 1.10 -33.30 4.7 2.5
Creatinine Kinase (CK) 622.00 1.10 -33.30 4.4 1.1
Iron 18.20 0.99 0.45 1.4 2.1
Iron 38.60 0.99 0.45 0.1 3.3
Creatinine 130.00 0.99 0.60 0.9 3.3
Creatinine 368.00 0.99 0.60 1.2 1.7
GGT 53.00 1.01 -0.52 0.2 1.8
GGT 172.00 1.01 -0.52 0.5 1.7
Glucose 6.28 1.02 -0.15 0.5 1.5
Glucose 15.80 1.02 -0.15 0.9 1.2
LDH 232.00 1.05 -11.20 0.27 3.1
LDH 742.00 1.05 -11.20 3.59 2.2
Potassium 3.99 1.00 0.03 0.65 1.0
Potassium 5.88 1.00 0.03 0.41 1.2
Protein, Total 45.70 0.99 -0.56 2.03 1.3
Protein, Total 60.30 0.99 -0.56 1.73 1.4
Sodium 138.00 1.04 -5.43 0.0 1.3
Sodium 155.00 1.04 -5.43 0.4 0.7
Urea Nitrogen 7.46 1.03 -0.16 0.64 2.4
Urea Nitrogen 20.00 1.03 -0.16 2.00 2.3
Uric Acid 360.00 1.00 0.00 0.00 1.1
Uric Acid 560.00 1.00 0.00 0.00 0.8

 Yes, that is a whole lot of numbers! What do they all mean? In the absence of context, it's often hard to know.

So let's apply Sigma-metrics and plot the performance visually.

Summary of Performance by Sigma-metrics Method Decision Chart and OPSpecs chart - using Ricos Goals

We can make visual assessments of this performance using a Normalized Sigma-metric Method Decision Chart:

2017 AU 680 NMEDX 

 What would this mean for QC? That's when we look at the OPSpecs chart:

2017 AU 680 Normalized OPSpecs chart

This is looking at two levels of each of 19 assays, with a large percentage of these falling below 2 Sigma. If we use Ricos goals, in other words, this AU 680 looks like it has a number of assays that are not fit for purpose, and would require a massive investment in quality control materials (up to 8 control measurements per run) in order to keep these methods adequately monitored.

But what if we apply the CLIA goals?

Summary of Performance by Sigma-metrics Method Decision Chart and OPSpecs chart - using CLIA Goals

2017 AU 680 CLIA Normalized MEDX chart

It's quite a different, isn't it? Suddenly, if we apply the US standards, this instrument is a lot more compliant. Many more of the points fall into the bulls-eye, or at least hit the target.

2017 AU 680 CLIA Normalized OPspecs chart 

Using the CLIA goals, it's also much easier to maintain proper, but practical, quality control of the instrument. A majority of the methods can be easily controlled with simply 3s control limits and 2 controls. Not even necessary to use "Westgard Rules"!

 But what about assays where the low control is a low Sigma-metric, while the high control is a high Sigma-metric? What does that imply?

Summary of Performance by Sigma-metrics Method Decision Chart and OPSpecs chart - using CLIA Goals and a Critical Decision Level

When we advise laboratories to design their QC using Sigma-metrics, we typically have them designate one critical decision level for each assay, and then use the Sigma-metric of the control  closest to that critical decision level. In this evaluation, we're using the same decision levels that we use to judge all of our Sigma VP laboratories. In this case, it helps to focus on which assays are doing well and which are problematic.

2017 AU 680 CLIA critical decision level Normalized  MEDx chart

11 of 19 assays are in the bulls-eye, with 3 more at 5 Sigma. Two assays are three and two Sigma, which is where most of the laboratory effort will need to focus.

2017 AU 680 CLIA critical decisio nlevel Normalized OPspecs chart

Maybe as many as 13 assays won't need the "Westgard Rules", while two assays will need the "Westgard Rules" but can get by with two controls. The 2 and 3 Sigma assays (yellow and orange dots) will need the full "Westgard Rules" and probably double, maybe triple the QC. The lab may even want to consider adding additional control mechanisms like Average of Normals, repeated patient samples, and maybe even averaging more than one sample to get a result.

Conclusion

The authors stated "The data demonstrates that the SM [Sigma-metric] calculations match very well with the FFP [Fitness For Purpose] calculations used to interpret imprecision and bias. Hence both approaches are complementary when using TEa based on BV. The individual interpretation of bias and imprecision using FFP criteria allowed us to clearly determine the major source of error. Whereas the SM level provided a summative evaluation of method performance for each analyte. But the selection of TEa is fundamental to this interpretation and harmonisation of the TEa applied for the SM calculation is needed." 

Based on Sigma-metric analysis using different sources of goals, we can see the need for harmonisation. The Ricos goals appear to be unattainable for most assays by this instrument. Yet by the standards of CLIA, this is a pretty good instrument. So an instrument may appear to be unacceptable in Europe, but perfectly compliant and good in the USA. As the debate continues over what goals should be applied with Sigma-metrics. But we don't have to wait for someone else to choose the right TEa sources. We can evaluate them ourselves and choose the ones that are right for our laboratory. It's becoming clear that many of the "Ricos goals," as evidence-based as they are, are not reality-based in their implementation. Our future instruments should strive to reach those goals, but for today, we should concentrate on the practical goals we can achieve. For now, some of the CLIA goals may be more appropriate than the Ricos goals to use for Sigma-metrics.