Sigma Metric Analysis
Multimode analysis of Cobas 6000 IA assays in Turkey
Multimode analysis isn't just for chemistry assays. It's time to look at immunoassay performance from the major diagnostic manufacturers. IA is an area where the goal differences get bigger. As EFLM gets wider and CLIA gets tighter, the impacts in IA are also important. This time we look at a Roche cobas 6000, with a evaluation of measurement uncertainty as the cherry on top.
Multimode analysis of Roche cobas 6000 immunoassays in Turkey
November 2022
Sten Westgard, MS
Keeping up with our series on major instrument analysis, it's time to shift our attention to immunoassays. Chemistry performance specifications can be demanding, but the engineering is also more mature, sometimes with 6 generations of instrumentation design. Roche cobas 6000, representing one of the market dominant platforms, is a useful device to examine. This analysis will use the latest desirable and minimum performance specifications from the EFLM, as well as CLIA 1992 and CLIA 2024 goals.
See the other analyses in this series:
- Beckman Coulter DxC 700
- Abbott Alinity
- Siemens Atellica
- Siemens Atellica in Romania
- Siemens Atellica in Spain
- Siemens ADVIA 2120i
- Roche c501 in Turkey
- Roche c501 in Saudi Arabia
- MicroLab RX-50 in India
- Roche cobas 6000 immunoassays in Turkey
- Sysmex XN 350 in India
- Mindray 7500 in China
- Mindray BS 2000M in China
This Roche cobas 6000 immunoassay data comes from Ankara Numune Training and Reseach Hospital, Ankara, Turkey:
Evaluation of Measurement Uncertainties of Immunoassays Analytes According to Biological Variation Databases. Muhammed F. Kilinckaya and Turan Turhan, Clin Lab 2022;68:1784-1791.
The study was conducted with IQC data from August 2018 to January 2018. Imprecision was measured at normal and pathologic levels. Bias was estimated as the average of the RIQAS survey from the same period. The paper looked at 14 common biochemistry tests.
TEST | % Bias | CV |
AFP | 6.11 | 5.48 |
AFP | 6.11 | 5.36 |
CA 125 II | 2.48 | 4.59 |
CA 125 II | 2.48 | 4.60 |
CA 15-3 | 3.75 | 5.81 |
CA 15-3 | 3.75 | 5.13 |
CA 19-9 xr | 3.28 | 4.70 |
CA 19-9 xr | 3.28 | 4.21 |
CEA | 3.13 | 4.12 |
CEA | 3.13 | 4.18 |
Cortisol | 3.1 | 4.20 |
Cortisol | 3.1 | 4.75 |
Estradiol | 2.2 | 5.7 |
Estradiol | 2.2 | 3.9 |
Ferritin | 2.59 | 3.83 |
Ferritin | 2.59 | 4.01 |
Folate | 3.15 | 8.34 |
Folate | 3.15 | 6.26 |
FSH | 2.8 | 3.36 |
FSH | 2.8 | 3.21 |
LH | 1.85 | 3.81 |
LH | 1.85 | 3.20 |
fPSA | 2.73 | 3.41 |
fPSA | 2.73 | 3.02 |
tPSA | 5.01 | 3.20 |
tPSA | 5.01 | 3.01 |
PTH | 11.27 | 4.85 |
PTH | 11.27 | 4.05 |
T3, Free | 1.98 | 3.16 |
T3, Free | 1.98 | 2.13 |
T4, Free | 3.47 | 2.06 |
T4, Free | 3.47 | 2.50 |
Testosterone | 3.74 | 4.04 |
Testosterone | 3.74 | 4.63 |
TSH | 3.55 | 3.72 |
TSH | 3.55 | 3.23 |
Please note the imprecision is determined on two different control levels, but the bias is one average bias, so the same bias will be used for all Sigma-metric calculations for each of the analytes. This will result in pairs of dots in the graphs that follow. Please also note that some of these analytes are not covered by EFLM, CLIA, etc.
The analysis will use the performance specifications from latest (2022) EFLM, both minimum and desirable as derived from the EuBIVAS biological variation database (accessed November 2022), as well as the desirable goals from the Ricos 2014 performance specifications; and finally the 1992 and newly announced 2024 CLIA goals.
Sigma-metrics according to EuBIVAS-derived DESIRABLE performance specifications
The EFLM desirable specifications used to be the global standard, but have fallen out of favor due to their toughness.
If those goals are used, there is almost nothing at the level of Six Sigma (<10%), and the vast majority of performance (62.5%) is poor or unacceptable. Notice the major challenge is imprecision.
So what happens if we lower the EFLM standards?.
Sigma-metrics according to EuBIVAS-derived MINIMUM performance specifications
There is a huge improvement, now over half of the performance is at 5 and 6 Sigma, and less than 10% is below 2 Sigma. It's almost a complete reversal of the desirable specifications analysis.
Next, let's look at the "original" version of the biological variation database. The last version of the database was updated in 2014 by Ricos et al, so we refer to these performance specifications as Ricos 2014 goals.
The original Ricos goals are somewhere in the middle of the EFLM desirable and minimum specifications. Instead of a plurality of world class or poor assays, the cetner of gravity is closer to 4 or 5 Sigma.
What happens if we move away from biologically-derived goals toward the CLIA goals, including those newly announced to take effect in 2024? Let's start with the original, now "classic" CLIA goals issued in 1992.
Sigma-metrics according to CLIA 1992 performance specifications
Here's the bald explanation for why CLIA goals needed to be updated. In 1992, CLIA essentially had NO performance specifications for immunoassays. So there is little insight CLIA 1992 goals can provide to the analysis of this device.
What happens when CLIA tightens its goals and expands coverage to include immunoassays?
Sigma-metrics according to CLIA 2024 performance specifications
CLIA 2024 ends up somewhere in between EFLM minimum and desirable specifications. There is not a large number of 6 Sigma assays, nor is there a proliferation of poor assays. Mostly the assays are 4 and 5 Sigma.
Conclusion
Having pivoted to IA, we can see that our assumptions about CLIA and EFLM goals are going to be up-ended. CLIA goals are no longer the most permissive - it's EFLM minimum that gets that distinction. The complaints that CLIA was too big and EFLM goals should be adopted instead are being turned on its head. What will EFLM do now that their goals are the most permissive?
Clearly, we will need to assess more IA devices to get a better picture of where the state of the art is for instruments, as well as the state of the performance specifications.
Bonus Analysis: How are these assays judged when MU, MAU, EAMMU is used?
As the title of the publcation indicates, the real goal of the authors was to assess measurement uncertainty. So let's take a good look at that.
Analyte | Expanded combined Uncertainty, U |
EFLM Desirable |
EFLM Minimum MAU |
Verdict |
AFP | 14.78 | 4.6 | 6.9 | Fails both MAU |
CA 125 | 7.94 | 8.7 | 13.0 | Passes both MAU |
CA 15.3 | 7.94 | No EFLM specification | ||
CA 19.9 | 8.54 | 4.3 | 6.4 | Fails both MAU |
CEA | 8.4 | 6.8 | 10.2 | Passes minimum MAU |
Cortisol | 8.67 | 16.3 | 24.5 | Passes both MAU |
Ferritin | 8.26 | 12.8 | 19.2 | Passes both MAU |
Folate | 10.11 | 11.8 | 17.7 | Passes both MAU |
FSH | 7.39 | 12.4 | 18.6 | Passes both MAU |
LH | 5.46 | 22.8 | 34.2 | Passes both MAU |
Estradiol | 6.93 | 15 | 22.5 | Passes both MAU |
fPSA | 6.95 | 8.8 | 13.2 | Passes both MAU |
tPSA | 11.98 | 6.98 | 10.2 | Fails both MAU |
PTH | 27.47 | 15.7 | 23.5 | Fails both MAU |
fT3 | 7.18 | 5 | 7.5 | Fails both MAU |
fT4 | 9.2 | 4.9 | 7.4 | Fails both MAU |
Testosterone | 9.67 | 12.5 | 18.8 | Passes both MAU |
TSH | 9.68 | 17.7 | 26.5 | Passes both MAU |
Curiously, this analysis has one of the highest pass rates: 10 pass, 6 fail, and 1 passes the minimum MAU. That's almost the easiest set of benchmarks we've looked at. Only the EFLM minimum TEa are more permissive. One of the reasons for this, of course, is that the impact of bias is either ignored or minimized.
It appears EFLM is trending much more permissive. This may make labs and particularly manufacturers happier, but the resulting quality delivered to patients may not improve at all.