Sigma Metric Analysis
Sigma-metrics of an AU 680
In 2016, Clinical Biochemistry published a study evaluating the Sigma-metrics of the Beckman Coulter Olympus AU 680 using biological variation goals.
Sigma-metrics comparison of an AU 680
- The Precision and Comparison data
- Summary of Performance according to Ricos Goals by Sigma-metrics chart
- Summary of Performance according to CLIA Goals by Sigma-metrics chart
- Summary of Performance at critical decision levels according to CLIA goals by Sigma-metrics chart
- Conclusion
February 2017
Sten Westgard, MS
[Note: This QC application is an extension of the lesson From Method Validation to Six Sigma: Translating Method Performance Claims into Sigma Metrics. This article assumes that you have read that lesson first, and that you are also familiar with the concepts of QC Design, Method Validation, and Six Sigma. If you aren't, follow the link provided.]
Clinical Biochemistry published a study conducted by the Clinical Biochemistry Laboratory at the National Hospital of Pediatrics, in Hanoi, Vietnam, that evaluated the Beckman Coulter (formerly Olympus) AU 680.
Practical application of biological variation and Sigma metrics quality models to evaluate 20 chemistry analytes on the Beckman Coulter AU680 Tran MTC, Hoang KT, Greaves RF, Clinical Biochemistry 2016 Nov 49(16-17): 1259-1266.
In this study, the instrument performance was compared against the biologic variation criteria developed primarily by the Spanish EQA groups, popularly known as the "Ricos goals," after Dr. Carmen Ricos, one of the strongest proponents of their use. This study did not only an evaluation of the imprecision and bias, but also total error and Sigma-metrics.
What we'll add to this study is a graphic analysis of the same instrument data, this time using CLIA goals, as well as a focused graph that just looks at one critical level of performance for each test.
The Imprecision and Bias Data
"Commericial lyophilized control samples of human origin were used for evaluation. There were purchased from Randox Laboratories... and included Randox human assay control 2 lot 887 UN and Randox human assay control 3 lot 649UN...Within-run and day-to-day imprecision was used to determine the extent of random error. Within-run imprecision was determined on 20 consecutive measurements of different analyte concentrations in the respective control sera. Day-to-day imprecision (CVa) was determined by measuring the concentration of analytes in the control sera of different concentration ranges over a 20 day period."
Bias calculations from external quality assurance data. "Commercial pool serum samples from Randox International Quality Assurance Scheme (RIQAS) Monthly Chemistry Program were used for assessment. Bias of measurement of all analytes... was determined by running the serum samples from the RIQAS Monthly Chemistry Program (six EQA samples) over a six month period, among these samples four samples were analysed twice independently (so that we got 10 results)."
Then Passing-Bablock regression was used to determine the systematic error between the test method and the instrument group mean. .
TEST | Group or PeerMean | slope | y-int | Bias% | CV% |
Albumin | 39.80 | 1.03 | -1.59 | 0.69 | 2.1 |
Albumin | 27.80 | 1.03 | -1.59 | 2.42 | 1.4 |
Alkaline Phosphatase | 195.00 | 0.99 | 1.50 | 0.5 | 3.1 |
Alkaline Phosphatase | 418.00 | 0.99 | 1.50 | 0.9 | 2.1 |
ALT | 36.00 | 1.01 | -1.11 | 2.3 | 3.8 |
ALT | 134.00 | 1.01 | -1.11 | 0.0 | 1.8 |
Amylase | 83.00 | 1.06 | -11.10 | 7.2 | 2.2 |
Amylase | 291.00 | 1.06 | -11.10 | 2.4 | 2.9 |
AST | 36.00 | 1.04 | -1.03 | 0.8 | 3.4 |
AST | 152.00 | 1.04 | -1.03 | 3.0 | 2.2 |
Bilirubin, Total | 32.20 | 0.98 | -0.64 | 4.2 | 1.8 |
Bilirubin, Total | 93.60 | 0.98 | -0.64 | 2.9 | 2.3 |
Calcium | 2.26 | 1.08 | -0.20 | 0.94 | 1.3 |
Calcium | 3.11 | 1.08 | -0.20 | 1.50 | 1.2 |
Chloride | 101.00 | 1.00 | -0.07 | 0.4 | 1.0 |
Chloride | 114.00 | 1.00 | -0.07 | 0.4 | 0.7 |
Creatinine Kinase (CK) | 230.00 | 1.10 | -33.30 | 4.7 | 2.5 |
Creatinine Kinase (CK) | 622.00 | 1.10 | -33.30 | 4.4 | 1.1 |
Iron | 18.20 | 0.99 | 0.45 | 1.4 | 2.1 |
Iron | 38.60 | 0.99 | 0.45 | 0.1 | 3.3 |
Creatinine | 130.00 | 0.99 | 0.60 | 0.9 | 3.3 |
Creatinine | 368.00 | 0.99 | 0.60 | 1.2 | 1.7 |
GGT | 53.00 | 1.01 | -0.52 | 0.2 | 1.8 |
GGT | 172.00 | 1.01 | -0.52 | 0.5 | 1.7 |
Glucose | 6.28 | 1.02 | -0.15 | 0.5 | 1.5 |
Glucose | 15.80 | 1.02 | -0.15 | 0.9 | 1.2 |
LDH | 232.00 | 1.05 | -11.20 | 0.27 | 3.1 |
LDH | 742.00 | 1.05 | -11.20 | 3.59 | 2.2 |
Potassium | 3.99 | 1.00 | 0.03 | 0.65 | 1.0 |
Potassium | 5.88 | 1.00 | 0.03 | 0.41 | 1.2 |
Protein, Total | 45.70 | 0.99 | -0.56 | 2.03 | 1.3 |
Protein, Total | 60.30 | 0.99 | -0.56 | 1.73 | 1.4 |
Sodium | 138.00 | 1.04 | -5.43 | 0.0 | 1.3 |
Sodium | 155.00 | 1.04 | -5.43 | 0.4 | 0.7 |
Urea Nitrogen | 7.46 | 1.03 | -0.16 | 0.64 | 2.4 |
Urea Nitrogen | 20.00 | 1.03 | -0.16 | 2.00 | 2.3 |
Uric Acid | 360.00 | 1.00 | 0.00 | 0.00 | 1.1 |
Uric Acid | 560.00 | 1.00 | 0.00 | 0.00 | 0.8 |
Yes, that is a whole lot of numbers! What do they all mean? In the absence of context, it's often hard to know.
So let's apply Sigma-metrics and plot the performance visually.
Summary of Performance by Sigma-metrics Method Decision Chart and OPSpecs chart - using Ricos Goals
We can make visual assessments of this performance using a Normalized Sigma-metric Method Decision Chart:
What would this mean for QC? That's when we look at the OPSpecs chart:
This is looking at two levels of each of 19 assays, with a large percentage of these falling below 2 Sigma. If we use Ricos goals, in other words, this AU 680 looks like it has a number of assays that are not fit for purpose, and would require a massive investment in quality control materials (up to 8 control measurements per run) in order to keep these methods adequately monitored.
But what if we apply the CLIA goals?
Summary of Performance by Sigma-metrics Method Decision Chart and OPSpecs chart - using CLIA Goals
It's quite a different, isn't it? Suddenly, if we apply the US standards, this instrument is a lot more compliant. Many more of the points fall into the bulls-eye, or at least hit the target.
Using the CLIA goals, it's also much easier to maintain proper, but practical, quality control of the instrument. A majority of the methods can be easily controlled with simply 3s control limits and 2 controls. Not even necessary to use "Westgard Rules"!
But what about assays where the low control is a low Sigma-metric, while the high control is a high Sigma-metric? What does that imply?
Summary of Performance by Sigma-metrics Method Decision Chart and OPSpecs chart - using CLIA Goals and a Critical Decision Level
When we advise laboratories to design their QC using Sigma-metrics, we typically have them designate one critical decision level for each assay, and then use the Sigma-metric of the control closest to that critical decision level. In this evaluation, we're using the same decision levels that we use to judge all of our Sigma VP laboratories. In this case, it helps to focus on which assays are doing well and which are problematic.
11 of 19 assays are in the bulls-eye, with 3 more at 5 Sigma. Two assays are three and two Sigma, which is where most of the laboratory effort will need to focus.
Maybe as many as 13 assays won't need the "Westgard Rules", while two assays will need the "Westgard Rules" but can get by with two controls. The 2 and 3 Sigma assays (yellow and orange dots) will need the full "Westgard Rules" and probably double, maybe triple the QC. The lab may even want to consider adding additional control mechanisms like Average of Normals, repeated patient samples, and maybe even averaging more than one sample to get a result.
Conclusion
The authors stated "The data demonstrates that the SM [Sigma-metric] calculations match very well with the FFP [Fitness For Purpose] calculations used to interpret imprecision and bias. Hence both approaches are complementary when using TEa based on BV. The individual interpretation of bias and imprecision using FFP criteria allowed us to clearly determine the major source of error. Whereas the SM level provided a summative evaluation of method performance for each analyte. But the selection of TEa is fundamental to this interpretation and harmonisation of the TEa applied for the SM calculation is needed."
Based on Sigma-metric analysis using different sources of goals, we can see the need for harmonisation. The Ricos goals appear to be unattainable for most assays by this instrument. Yet by the standards of CLIA, this is a pretty good instrument. So an instrument may appear to be unacceptable in Europe, but perfectly compliant and good in the USA. As the debate continues over what goals should be applied with Sigma-metrics. But we don't have to wait for someone else to choose the right TEa sources. We can evaluate them ourselves and choose the ones that are right for our laboratory. It's becoming clear that many of the "Ricos goals," as evidence-based as they are, are not reality-based in their implementation. Our future instruments should strive to reach those goals, but for today, we should concentrate on the practical goals we can achieve. For now, some of the CLIA goals may be more appropriate than the Ricos goals to use for Sigma-metrics.