Sigma Metric Analysis

It's time to take another look at a major instrument system, and apply Sigma metrics using goals from the most popular sources, CLIA, Ricos 2014, and EFLM 2022.

Multimode analysis of an Atellica CH 930 chemistry analyzer in Romania

March 2022, updated July 2022, updated April 2023
Sten Westgard, MS

A few years ago, we took a look at the Siemens Atellica. It's time to look again, using the latest performance specifications from the EFLM, Ricos 2014, and CLIA 1992. June 2022: updated to assess new EFLM biological minimum goals. April 2023: updated to assess new CLIA 2024 goals

See the other analyses in this series:

• Beckman Coulter DxC 700
• Abbott Alinity
• Siemens Atellica
• Siemens Atellica in Romania
• Siemens Atellica in Spain
• Siemens ADVIA 2120i
• Roche c501 in Turkey
• Roche c501 in Saudi Arabia
• MicroLab RX-50 in India
• Roche cobas 6000 immunoassays in Turkey
• Sysmex XN 350 in India
• Mindray 7500 in China
• Mindray BS 2000M in China

This latest set of data comes from a very thorough study from Revista Romana de Medicina de Laborator, Vol. 29 Nr. 4, October 2021:

Atellica CH 930 chemistry analyzer versus Cobas 6000 c501 and Architect ci4100 - a multi-analyte method comparison, Imola Gyorfi, Krisztina Pal, Ion Bogdan Manescu, Oana R. Porea, Minodora Dobreanu.

The study was conducted between April and June 2020 at the Central Laboratory of Targu Mures County Emergency Clinical Hospital in Romania. Daily QC was recorded for over 30 days, using 3 levels of Bio-Rad controls. They compared the new Atellica with the laboratory's then-current method, the Roche Cobas 6000 c501.  "Analytical difference... between test and comparative methods was calculated at medical decision levels (MDL) based on Passing-Bablock regression equations." The study actually calculated the biases between the Atellica and the Architect and the Atellica and the Cobas; for our purposes we are going to use the bias between Cobas and Atellica, since Roche instruments are more widespread.

The raw data of imprecision and bias

There are 22 test methods that they analyzed, in some cases they only focused on 2 levels, in other cases, 3 levels. Now, the next thing to do is take a look at the latest (2022) EFLM performance specifications, as derived from the EuBIVAS biological variation database; Ricos 2014 performance specifications; and finally the 1992 CLIA goals.

Please note that the number of assays will vary from table to table, since not all assays are covered by the latest EuBIVAS database, Ricos goals, or the CLIA goals. That makes the comparison slightly uneven - but it's a comparison of slightly different sizes of apples, not entirely different fruits.

Sigma-metrics according to EuBIVAS-derived desirable performance specifications (as of January 2022)

If you're looking for the toughest set of specifications, EFLM gives them. More than half of the assay performance is located at 3 Sigma and below, while just over a third of performance is at 6 Sigma. That's the most worst and the least best in our comparison.

The news here is grim. We have a lot of very low Sigma metrics, 64% of the values are below 3, and in many cases the bias is larger than the goal (resulting in what we can "negative" Sigma). Only a little over 11% of the values are bove 6 Sigma.

Next, we can look at the "original" version of the biological variation database. The last version of the database was updated in 2014 by Ricos et al, so we refer to these performance specifications as Ricos 2014 goals.

Sigma-metrics according to Ricos 2014 desirable performance specifications

The original Ricos goals are not quite as hard on this Atellica. Over 53% of the values are still below 3 Sigma, and now 16% of the values are above 6 Sigma. But that is still not a good look.

Sigma-metrics according to CLIA 1992 performance specifications

If you judge the Atellica with the oldest specifications, also the most popular ones in the USA. Now only a third of the assays are below 3 Sigma, and a third of the assays are above 6 Sigma.

Note that the CLIA goals have unit-based goals for some analytes, like sodium, so in the table we converted them a %-based goal.

Sigma-metrics according to EuBIVAS-derived MINIMUM performance specifications (as of July 2022)

Interesting to see the differences. Nearly 40% of the performance is considered below 3 Sigma, even by the most relaxed EFLM minimum performance specifications. Yet nearly every level - More than 93%-  of this performance "passes" the MAU / EAMMU specification. The relaxed measurement uncertainty goals are even looser than EFLM's minimum specifications, which are the least demanding TEa goals seen here.

If MAU is simply a rubber stamp, giving approval to all methods, it does not lead to any progress. It merely enables a status quo.

Sigma-metrics according to CLIA 2024 performance specifications

The new CLIA 2024 goals are no pushover. Over 40% of the levels here are below 3 Sigma, and 14% are 3 Sigma, so the majority of performance here is 3 Sigma or lower. Only 20% of the performance hits the bull's-eye. If this lab was in the US, proficiency testing would be a scary event.

Conclusion

This data set doesn't really look good from any perspective, but the desirable EFLM goals are the most difficult to achieve, followed by the CLIA 2024 goals. EFLM minimum goals are easier to achieve than CLIA's new standards, which represents a major shift in the hierarchy of goals. Even applying goals from CLIA that are 3 decades old is not successful. This provides another example of the danger of adopting performance specifications out of popular enthusiasm, rather than practical capability. We need a bigger conversation about what goals are appropriate to apply to today's instruments, not just applying goals because we can calculate them.