Sigma Metric Analysis

New instruments running immunoassays - can they meet the goals of today and tomorrow? We evaluate real-world performance against goals set by Ricos, EuBIVAS, and the proposed goals of CLIA.

 Multimode Sigma Analysis of Alinity Immunoassays

January 2021
Sten Westgard, MS

See the other analyses in this series:

• Beckman Coulter DxC 700
• Abbott Alinity
• Siemens Atellica
• Siemens Atellica in Romania
• Siemens Atellica in Spain
• Siemens ADVIA 2120i
• Roche c501 in Turkey
• Roche c501 in Saudi Arabia
• MicroLab RX-50 in India
• Roche cobas 6000 immunoassays in Turkey
• Sysmex XN 350 in India
• Mindray 7500 in China
• Mindray BS 2000M in China

Continuing in our series of analyzing major diagnostic instrumentation using the latest sets of performance specifications

(EFLM/EuBIVAS 2021, Ricos 2014, and CLIA 2019), we are taking a look at a recent paper on the Abbott Alinity. But for once, the paper is covering immunoassays, not biochemistry assays.

[Reader note: I have written papers and posters on the Alinity performance. Here are disclosures about the relationship between Westgard QC, Inc. and Abbott Diagnostics. Westgard QC, Inc. has a contract with Abbott Diagnostics covering education, training, and consulting services.  Regarding this specific paper and analysis of this paper, however, no guidance or funds were involved.]

 The imprecision and bias data

Two types of controls were used, an independent control from Technopath, as well as in-kit (manufacturer provided) controls. The latter are considered less reliable, as they may have been optimized by the manufacturer to provide a rosy picture of performance. For immunoassays, these performance studies are fairly rare, so we'll have to accept this weakness and retain a grain of salt when reviewing the results.

Bias was determined by comparing the Alinity with its predecessor, an ARCHITECT i2000. While this is therefore a relative difference being calculated, it may cover one of the most common customer scenarios, where a laboratory is upgrading their instruments to the latest generation.

Most of the controls ran 3 levels, with the exception of CA 15-3, which only ran two levels.

What goals to use?

For many immunoassays, the current CLIA regulation does not provide PT goals. Often the only goals available for these tests have been from the biological variation databases. When CMS proposed updated goals for CLIA (hereafter referred to as CLIA 2019 goals), a lot more immunoassay analytes were brought under direct regulation. But those goals remain proposals only - they are not law.

Nevertheless, we will evaluate the ability of the Alinity to fulfill those goals, presuming that at some point in the future, they will become law.

A quick comparison of goals

For about 15 years, the "Ricos goals" were the best and largest source of information on goals for performance. While the EuBIVAS goals have supplanted them, there are still tests where EuBIVAS has not yet provided a specification (basically, no recent study has provided any data to make a performance specification), thus there are still reasons to consult the Ricos goals. 

The other interesting finding here: EuBIVAS goals for immunoassays seem to be growing larger than the Ricos goals. This is the opposite of what has been seen on the biochemistry side, where EuBIVAS goals are often so tight they are impossible to achieve (thus, making the Ricos goals attractive again).

 The proposed CLIA goals have a 2-part specification, a unit-based goal near the low end of the range, and a percentage-based goal at the upper end of the range. For comparative purposes, we are just looking at the percentage goal here.

Sigma-metric performance: Ricos 2014 Desirable Specifications

Out of all the levels measured, 63.2% are 6 Sigma or better, 2.94% are 5 Sigma, and 11.76% are 4 Sigma.

So if we take 4 Sigma and higher as our goal for good performance, we would reach about 77.94%. On the chemistry side, this is what we aim for, hoping to reach around 80%. So it's impressive to see the performance of immunoassays about as good as the chemistry side.

The other side of this, the number of performance levels below 3 Sigma, is about 19.12%. Some of the performance at low levels of the tests is challenging - but if we were analyzing with more detail, we might decide those low level areas are not as clinically relevant for some tests.

Sigma-metric Performance: EuBIVAS 2021 specifications

As we said earlier, there are 4 fewer assays covered by the EuBIVAS goals. But of what is covered, there were some goals bigger, other goals smaller, than the Ricos goals.

Here only 56% of the assays can reach 6 Sigma, 3.51% can reach both 5 and 4 Sigma. 29.82% is less than 3 Sigma.

So a larger number of levels cannot meet the EuBIVAS goals, almost a third of the performance. Troubling if we think this instrument represents the highest achievement of instrumentation the market can provide. So again this provides more evidence that the EuBIVAS goals may be more evidence-based, but they may also be less realistic or practical for today's laboratories.

Sigma-metric Performance: Proposed CLIA 2019 goals

 So even if there are updates to CLIA goals, they are the easier to achieve than the EuBIVAS goals. Less than 10% of the performance levels are below 3 Sigma, while 73.73% are 4, 5, and 6 Sigma. Interesting that CLIA goals are actually harder to achieve than the Ricos 2014 goals.