Tools, Technologies and Training for Healthcare Laboratories

Multimode analysis of Cobas 6000 IA assays in Turkey

Multimode analysis isn't just for chemistry assays. It's time to look at immunoassay performance from the major diagnostic manufacturers. IA is an area where the goal differences get bigger. As EFLM gets wider and CLIA gets tighter, the impacts in IA are also important. This time we look at a Roche cobas 6000, with a evaluation of measurement uncertainty as the cherry on top.

Multimode analysis of Roche cobas 6000 immunoassays in Turkey

November 2022
Sten Westgard, MS

Keeping up with our series on major instrument analysis, it's time to shift our attention to immunoassays. Chemistry performance specifications can be demanding, but the engineering is also more mature, sometimes with 6 generations of instrumentation design. Roche cobas 6000, representing one of the market dominant platforms, is a useful device to examine. This analysis will use the latest desirable and minimum performance specifications from the EFLM, as well as CLIA 1992 and CLIA 2024 goals.

See the other analyses in this series:

This Roche cobas 6000 immunoassay data comes from Ankara Numune Training and Reseach Hospital, Ankara, Turkey:

Evaluation of Measurement Uncertainties of Immunoassays Analytes According to Biological Variation Databases. Muhammed F. Kilinckaya and Turan Turhan, Clin Lab 2022;68:1784-1791.

The study was conducted with IQC data from August 2018 to January 2018. Imprecision was measured at normal and pathologic levels.  Bias was estimated as the average of the RIQAS survey from the same period. The paper looked at 14 common biochemistry tests.

TEST % Bias CV
AFP 6.11 5.48
AFP 6.11 5.36
CA 125 II 2.48 4.59
CA 125 II 2.48 4.60
CA 15-3 3.75 5.81
CA 15-3 3.75 5.13
CA 19-9 xr 3.28 4.70
CA 19-9 xr 3.28 4.21
CEA 3.13 4.12
CEA 3.13 4.18
Cortisol 3.1 4.20
Cortisol 3.1 4.75
Estradiol 2.2 5.7
Estradiol 2.2 3.9
Ferritin 2.59 3.83
Ferritin 2.59 4.01
Folate 3.15 8.34
Folate 3.15 6.26
FSH 2.8 3.36
FSH 2.8 3.21
LH 1.85 3.81
LH 1.85 3.20
fPSA 2.73 3.41
fPSA 2.73 3.02
tPSA 5.01 3.20
tPSA 5.01 3.01
PTH 11.27 4.85
PTH 11.27 4.05
T3, Free 1.98 3.16
T3, Free 1.98 2.13
T4, Free 3.47 2.06
T4, Free 3.47 2.50
Testosterone 3.74 4.04
Testosterone 3.74 4.63
TSH 3.55 3.72
TSH 3.55 3.23

Please note the imprecision is determined on two different control levels, but the bias is one average bias, so the same bias will be used for all Sigma-metric calculations for each of the analytes. This will result in pairs of dots in the graphs that follow. Please also note that some of these analytes are not covered by EFLM, CLIA, etc.

The analysis will use the performance specifications from latest (2022) EFLM, both minimum and desirable as derived from the EuBIVAS biological variation database (accessed November 2022), as well as the desirable goals from the Ricos 2014 performance specifications; and finally the 1992 and newly announced 2024 CLIA goals.

Sigma-metrics according to EuBIVAS-derived DESIRABLE performance specifications

The EFLM desirable specifications used to be the global standard, but have fallen out of favor due to their toughness.

 2022 11 26 cobas 6000 IA EFLMdes NMEDX

If those goals are used, there is almost nothing at the level of Six Sigma (<10%), and the vast majority of performance (62.5%) is poor or unacceptable. Notice the major challenge is imprecision.

So what happens if we lower the EFLM standards?.

Sigma-metrics according to EuBIVAS-derived MINIMUM performance specifications

 2022 11 26 cobas 6000 IA EFLMmin NMEDX

There is a huge improvement, now over half of the performance is at 5 and 6 Sigma, and less than 10% is below 2 Sigma. It's almost a complete reversal of the desirable specifications analysis.

Next, let's look at the "original" version of the biological variation database. The last version of the database was updated in 2014 by Ricos et al, so we refer to these performance specifications as Ricos 2014 goals.

The original Ricos goals are somewhere in the middle of the EFLM desirable and minimum specifications. Instead of a plurality of world class or poor assays, the cetner of gravity is closer to 4 or 5 Sigma.

 What happens if we move away from biologically-derived goals toward the CLIA goals, including those newly announced to take effect in 2024? Let's start with the original, now "classic" CLIA goals issued in 1992.

Sigma-metrics according to CLIA 1992 performance specifications

 2022 11 26 cobas 6000 IA CLIA 1992 NMEDX

Here's the bald explanation for why CLIA goals needed to be updated. In 1992, CLIA essentially had NO performance specifications for immunoassays. So there is little insight CLIA 1992 goals can provide to the analysis of this device. 

What happens when CLIA tightens its goals and expands coverage to include immunoassays?

Sigma-metrics according to CLIA 2024 performance specifications

 2022 11 26 cobas 6000 IA CLIA24 NMEDX

CLIA 2024 ends up somewhere in between EFLM minimum and desirable specifications. There is not a large number of 6 Sigma assays, nor is there a proliferation of poor assays. Mostly the assays are 4 and 5 Sigma.

Conclusion

Having pivoted to IA, we can see that our assumptions about CLIA and EFLM goals are going to be up-ended. CLIA goals are no longer the most permissive - it's EFLM minimum that gets that distinction. The complaints that CLIA was too big and EFLM goals should be adopted instead are being turned on its head. What will EFLM do now that their goals are the most permissive?

Clearly, we will need to assess more IA devices to get a better picture of where the state of the art is for instruments, as well as the state of the performance specifications.

Bonus Analysis: How are these assays judged when MU, MAU, EAMMU is used?

As the title of the publcation indicates, the real goal of the authors was to assess measurement uncertainty. So let's take a good look at that.

Analyte Expanded combined
Uncertainty, U

EFLM Desirable
MAU

EFLM Minimum
MAU
Verdict
AFP 14.78 4.6 6.9 Fails both MAU
CA 125 7.94 8.7 13.0 Passes both MAU
CA 15.3 7.94     No EFLM specification
 CA 19.9 8.54  4.3 6.4 Fails both MAU
 CEA  8.4 6.8 10.2 Passes minimum MAU
 Cortisol  8.67 16.3 24.5 Passes both MAU
Ferritin 8.26 12.8 19.2 Passes  both MAU
 Folate 10.11 11.8 17.7 Passes both MAU
FSH 7.39 12.4 18.6 Passes both MAU
LH 5.46 22.8 34.2 Passes both MAU
 Estradiol 6.93 15 22.5 Passes both MAU
 fPSA 6.95 8.8 13.2 Passes both MAU
 tPSA 11.98 6.98 10.2 Fails both MAU
 PTH 27.47 15.7 23.5 Fails both MAU
 fT3 7.18 5  7.5 Fails both MAU
 fT4 9.2 4.9 7.4 Fails both MAU
Testosterone 9.67 12.5 18.8 Passes both MAU
TSH 9.68 17.7 26.5 Passes both MAU

 Curiously, this analysis has one of the highest pass rates: 10 pass, 6 fail, and 1 passes the minimum MAU. That's almost the easiest set of benchmarks we've looked at. Only the EFLM minimum TEa are more permissive. One of the reasons for this, of course, is that the impact of bias is either ignored or minimized.

It appears EFLM is trending much more permissive. This may make labs and particularly manufacturers happier, but the resulting quality delivered to patients may not improve at all.