Tools, Technologies and Training for Healthcare Laboratories

Sigma-metrics of a cobas 8000 c701 chemistry analyzer

A 2015 study in Revista de Caldida Asistencial analyzed a new core laboratory chemistry analyzer and the importance of implementing an analytical quality control system. 

Sigma-metrics of a core laboratory chemistry analyzer

January 2016
Sten Westgard, MS

Six Sigma QC Design, 2nd Edition [Note: This QC application is an extension of the lesson From Method Validation to Six Sigma: Translating Method Performance Claims into Sigma Metrics. This article assumes that you have read that lesson first, and that you are also familiar with the concepts of QC Design, Method Validation, and Six Sigma. If you aren't, follow the link provided.] Basic Method Validation, 3rd Edition

This analysis looks at a chemistry analyzer and the importance of implementing an analytical quality control system:

Importance of implementing an analytical quality control system in a core laboratory. Marques-Garcia F, Garcia-Codesal F, del Rosario Caro-Narros M, Contreras-SanFeliciano T,  Revista de Caldidad Asistencial 2015 Nov-Dec;30(6):302-9.

The University Hospital of Salamanca, Spain, performed an analysis of their analytical performance of their Cobas 8000 c701 and 28 biochemistry tests.

The Imprecision and Bias Data

The laboratory assessed imprecision and bias using three months of data of the Assayed Control Multiqual controls from Bio-Rad, as calculated by the Unity Real Time software. The study selected the critical decision level, so there is only one set of data per test.

Performance of Roche cobas 8000 c701
Assay Level
CV% Bias%
Albumin 2.57 4.8% 0.4%
Alk Phos 32.65 3.3%  1.3%
ALT 23.68 3.8% 8.7%
Amylase 42.0 2.5% 2.6%
Bilirubin, Direct 0.384 5.8% 5.8%
Bilirubin, Total 0.577 4.0% 2.8%
Calcium 6.15 2.5% 0.2%
Chloride 72.68 1.6% 2.0%
Cholesterol 109.6 2.7% 3.9%
Creatinine Kinase 79.17 3.2% 1.2%
Creatinine 0.56 9.6% 0.5%
GGT 25.08 3.4% 3.4%
Glucose 58.18 2.1% 1.3%
HDL-cholesterol 26.05 3.0% 0.0%
Iron 74.94 2.7% 1.8%
Lipase 18.35 2.5% 4.5%
LDL-cholesterol 68.08 2.5% 3.8%
LDH 121.1 2.6% 2.5%
Magnesium 1.05 3.8% 2.8%
Phosphate 2.0 2.8% 1.5%
Potassium 2.66 1.9% 3.5%
Protein, Total 4.0 2.7% 0.7%
Sodium  115.5 1.3% 0.6%
Transferrin  154.0 1.5% 0.5%
Triglycerides  88.18 2.1% 5.9%
Urea Nitrogen  30.05 2.7% 1.3%
Uric Acid  3.59 3.0% 1.1%

 

Determine Quality Requirements at the decision levels

Before we calculate the Sigma-metrics, we can judge the acceptability of the methods with separate specifications for imprecision and bias. If we consult the Ricos database for desirable specifications for imprecision and bias, we can compare those numbers to the errors we've observed in the instruments and methods:

Performance of Roche cobas 8000 c701
Assay Level
CV% Bias% TEa Source
Albumin 2.57 4.8% 0.4% 10% CLIA
Alk Phos 32.65 3.3%  1.3% 30% CLIA
ALT 23.68 3.8% 8.7% 20% CLIA
Amylase 42.0 2.5% 2.6% 30% CLIA
Bilirubin, Direct 0.384 5.8% 5.8% 44.5% Ricos
Bilirubin, Total 0.577 4.0% 2.8% 118% CLIA
Calcium 6.15 2.5% 0.2% 4.06% CLIA
Chloride 72.68 1.6% 2.0% 5.0% CLIA
Cholesterol 109.6 2.7% 3.9% 10.0% CLIA
Creatinine Kinase 79.17 3.2% 1.2% 30.0% CLIA
Creatinine 0.56 9.6% 0.5% 15.0% CLIA
GGT 25.08 3.4% 3.4% 22.1% Ricos
Glucose 58.18 2.1% 1.3% 10% CLIA
HDL-cholesterol 26.05 3.0% 0.0% 30% CLIA
Iron 74.94 2.7% 1.8% 20% CLIA
Lipase 18.35 2.5% 4.5% 29.1% Ricos
LDL-cholesterol 68.08 2.5% 3.8% 20% CAP PT
LDH 121.1 2.6% 2.5% 20% CAP PT
Magnesium 1.05 3.8% 2.8% 25% CAP PT
Phosphate 2.0 2.8% 1.5% 10.7% CAP
Potassium 2.66 1.9% 3.5% 18.8% CLIA
Protein, Total 4.0 2.7% 0.7% 10% CLIA
Sodium  115.5 1.3% 0.6% 3.46% CLIA
Transferrin  154.0 1.5% 0.5% 3.8% Ricos
Triglycerides  88.18 2.1% 5.9% 25.0% CLIA
Urea Nitrogen  30.05 2.7% 1.3% 9% CLIA
Uric Acid  3.59 3.0%

1.1%

17% CLIA

For many of these analytes, CLIA sets a unit-based goal, which means there is a variable allowable total error across the range of the assay. We convert those unit goals into percentage goals at the level where the CV is measured and the bias is estimated.

Note that the study itself used a different set of goals, one that was "based on the criteria for internationally accepted consensus conference in Stockholm...recommendations from experts or scientific societies, biological variation (best, minimal or desirable) or based on the Spanish Conensus Document on minimum analytical quality specifications based on the programs resuults of external quality assurance, established as 'state of the art.'" However, a list of these goals is not fully provided. So we don't know what all those performance specifications are.

Nevertheless, we've got all the data we need now to calculate Sigma-metrics. With Sigma-metrics we can start to make some sense of the performance.

Calculate Sigma metrics

Remember the equation for Sigma metric is (TEa - bias%) / CV. All terms are expressed as percentages

Example calculation: for Albumin for the cobas c701, with a 10% quality requirement, at the level of 2.57 g/dL, given 4.8% imprecision and 0.4% bias:

(10 - 0.4) / 4.8 = 9.6 / 4.8 = 2.0 Sigma

So here's the full table with all the metrics, where possible:

Performance of Roche cobas 8000 c701
Assay Level
CV% Bias% TEa Source Sigma-metric Study Sigma-metric
Albumin 2.57 4.8% 0.4% 10% CLIA 2.0 2.81
Alk Phos 32.65 3.3%  1.3% 30% CLIA >6 3.31
ALT 23.68 3.8% 8.7% 20% CLIA 3.0 4.99
Amylase 42.0 2.5% 2.6% 30% CLIA >6 4.85
Bilirubin, Direct 0.384 5.8% 5.8% 44.5% Ricos >6 >6
Bilirubin, Total 0.577 4.0% 2.8% 118% CLIA >6 >6
Calcium 6.15 2.5% 0.2% 4.06% CLIA 1.6 4.32
Chloride 72.68 1.6% 2.0% 5.0% CLIA 1.8 4.29
Cholesterol 109.6 2.7% 3.9% 10.0% CLIA 2.3 1.91
Creatinine Kinase 79.17 3.2% 1.2% 30.0% CLIA >6 >6
Creatinine 0.56 9.6% 0.5% 15.0% CLIA 1.5 2.03
GGT 25.08 3.4% 3.4% 22.1% Ricos 5.6 5.6
Glucose 58.18 2.1% 1.3% 10% CLIA 4.1 2.67
HDL-cholesterol 26.05 3.0% 0.0% 30% CLIA >6 3.89
Iron 74.94 2.7% 1.8% 20% CLIA >6 >6
Lipase 18.35 2.5% 4.5% 29.1% Ricos >6 >6
LDL-cholesterol 68.08 2.5% 3.8% 20% CAP PT >6 3.21
LDH 121.1 2.6% 2.5% 20% CAP PT >6 3.48
Magnesium 1.05 3.8% 2.8% 25% CAP PT 5.8 1.16
Phosphate 2.0 2.8% 1.5% 10.7% CAP 3.3 3.13
Potassium 2.66 1.9% 3.5% 18.8% CLIA >6 3.41
Protein, Total 4.0 2.7% 0.7% 10% CLIA 3.4 4.16
Sodium  115.5 1.3% 0.6% 3.46% CLIA 2.2 3.43
Transferrin  154.0 1.5% 0.5% 3.8% Ricos 2.3 2.27
Triglycerides  88.18 2.1% 5.9% 25.0% CLIA >6 >6
Urea Nitrogen  30.05 2.7% 1.3% 9% CLIA 2.8 5.26
Uric Acid  3.59 3.0%

1.1%

17% CLIA 5.3 3.65

Note there are two Sigma-metric columns. The first column of Sigma-metrics are from the calculations we made here, mostly with the CLIA goals, which are supposed to be more lenient than the Ricos goals. Nevertheless, the Sigma-metrics from CLIA are none too kind. The second column of Sigma-metrics are those calculated from the study itself. You can see that the authors' verdict is far harsher.

For this core laboratory, 7 out of the 28 analytes are below 3 Sigma, 25%, according to CLIA goals.  According to the authors' metrics, 6 out of the 28 analytes are below 3 Sigma, or 21%. On the other end of the scale, our assessment puts 12 analytes at better than Six Sigma performance, or 42.9%.  The authors' metrics only place 6 analytes at better than Six Sigma, again just 21%.

Summary of Performance by Sigma-metrics Method Decision Chart and OPSpecs chart

Let's focus on our CLIA-based performance specifications now. We can make a visual assessment of instrument performance using a Normalized Sigma-metric Method Decision Chart:

 2016 cobas c701 NMEDX

You can see that a quite a few methods hit the bull's eye, while many methods seem to be missing the target. One or two of these dots are actually "off the map" - so far off the chart that they are floating off to the right of the graph.

Now what about QC? How do we monitor and control these methods? For that, we need a Normalized OPSpecs chart:

2016 cobas c701 Normalized OPSpecs Chart

For Sodium, Calcium, Creatinine, Albumin, Chloride, Transferrin, and Urea, all the "Westgard Rules" are needed. Even then, the laboratory can't be certain they will catch medically important errors in the first run that they occur. The laboratory may need to consider running more controls and increasing the QC frequency. However, for those assays of 5 Sigma and higher, no "Westgard Rules" are necessary. So there are opportunities to make some savings in time and effort on some assays, and shore up the performance of the challenging assays on the other hand.

Conclusion

This analysis shows that in any published study, the reader needs to be aware that the choice of performance specification impacts the resulting Sigma-metrics significantly. While the authors presented a harsh review of this instrument, applying the CLIA quality requirements provides a better picture of performance. Clearly, this highlights the need for harmonization of performance specifications and quality requirements.