Sigma Metric Analysis

Continuing in our series of Sigma analysis of the leading diagnostic platforms, we turn our attention to the Abbott Alinity, and assess what goals can be achieved by this chemistry instrument.

Sigma metric analysis of an Abbott Alinity - benchmarking with multiple goals

February 2020
Sten Westgard, MS

See the other analyses in this series:

• Beckman Coulter DxC 700
• Abbott Alinity
• Siemens Atellica
• Siemens Atellica in Romania
• Siemens Atellica in Spain
• Siemens ADVIA 2120i
• Roche c501 in Turkey
• Roche c501 in Saudi Arabia
• MicroLab RX-50 in India
• Roche cobas 6000 immunoassays in Turkey
• Sysmex XN 350 in India
• Mindray 7500 in China
• Mindray BS 2000M in China

Continuing in our series of analyzing major diagnostic instrumentation using the latest sets of performance specifications

(EFLM/EuBIVAS 2020, Ricos 2014, CLIA 2019, and CLIA 1992), we are taking a look at one of the first papers on the Abbott Alinity .

One thing the reader may quickly note, I am one of the primary authors of this paper. Therefore, I should note some disclosures necessary to make when discussing this paper. Of specific relevance here: Westgard QC, Inc. has a contract with Abbott Diagnostics covering education, training, and consulting services.  For the writing and editing of this specific paper, however, no funds were involved.

The raw data of imprecision and bias

We will display the set of chemistry assay imprecision and bias published in this study. The paper noted performance at a single critical decision level, so there is only one Sigma metric per assay, rather than Sigma metrics measured at all control levels. This makes the comparison with the other platforms a little less like an "apples-to-apples" comparison.

"The imprecision for each assay was estimated via a precision study conducted at Abbott using the Alinity ci system following the guidelines of CLSI EP05-A2 [14].... Each control and panel was tested in replicates of 2–3 during 2 runs each day for at least 20 days, and an analysis of variance (ANOVA) of the data was performed to estimate the variance components [standard deviation (SD) and percent coefficient of variation (%CV)] for repeatability, between-run and a within-laboratory precision of the assay. For each assay, the precision study was conducted using a minimum of one Alinity system and one reagent lot. The within-laboratory %CV at or near the medical decision level was used in the Sigma-metric calculation...."

"Bias was estimated, following the principles of the CLSI EP09-A3 guideline, by testing between 100 and 200 samples, over at least three days, with testing in duplicate at Abbott on the Alinity c and ARCHITECT c8000 systems for clinical chemistry and
ICT assays and on the Alinity i and ARCHITECT i2000SR systems for the immunoassays. The ARCHITECT c8000 and i2000SR systems were chosen as the comparator methods because they were the measuring systems upon which the assays were originally developed. Each assay was tested using at least one Alinity and one ARCHITECT system. Using Passing-Bablok or weighted Deming regression methods, the first replicate of the Alinity concentration was compared against the mean of
the ARCHITECT results; these more sophisticated regression techniques help to identify the real bias versus the comparator to identify and minimize the effect of imprecision present during the comparison study. Using these regression models, the %bias was estimated near a medical decision point...."

Now, the next thing to do is take a look at the latest EFLM performance specifications, as derived from the EuBIVAS biological variation database, which is being updated frequently, so we have 2020 TEa goals for this analysis. We also used the Ricos 2014 performance specifications; proposed CLIA 2019 goals; and finally the current 1992 CLIA goals.

Please note that the number of assays will vary from table to table, since not all assays are covered by the latest EuBIVAS database, nor by the CLIA goals. That makes the comparison slightly uneven - but it's a comparison of slightly different sizes of apples, not entirely different fruits.

Sigma-metrics according to EuBIVAS-derived desirable performance specifications (as of January 2020)

If you're looking for the toughest set of specifications, this is it. More than half of the assay performance is below 3 Sigma, while just under a quarter of performance is at 6 Sigma. That's the most worst and the least best in our comparison.

We simplify the assessment here to state "above 6 Sigma" and "below 3 Sigma" because in the extreme cases, there is no practical difference to how a laboratory runs QC for a 6 Sigma assay vs. a 12 Sigma assay.

Perhaps uniquely among the instruments on the market, the Alinity has a majority of assays that can achieve 6 Sigma even when the most stringent performance specifications are applied. There are still a few assays that have unacceptable performance when judged by EuBIVAS standards, but these are assays (Calcium, Chloride, Sodium, Creatinine) where the goals being suggested are next to impossible for today's technology. For those assays, with those goals, there are currently no instruments on the market that are really capable of consistently achieving high sigma performance

Next, we can look at the "original" version of the biological variation database. The last version of the database was last updated in 2014 by Ricos et al, so we refer to these performance specifications as "Ricos 2014 goals."

Sigma-metrics according to Ricos 2014 desirable performance specifications

The original Ricos goals (desirable performance specifications) are still challenging, but nearly 60% of the assays on the Alinity are still at 6 Sigma. 6 of the assays are below 3 Sigma. This particular data set is the largest that can be benchmarked, since the 2014 Ricos database is still the largest set of performance specifications available.

Next we can look at performance specifications which are set by regulatory authorities, not biological variation.

Sigma-metrics according to Proposed CLIA goals of 2019 performance specifications

The 2019 proposals for updated CLIA goals expanded the number of analytes covered and tightened goals over the original CLIA specifications. With these goals, Alinity reaches a point where no assays are considered unacceptable.

For Alinity, if CLIA does in fact tighten the goals set for proficiency testing, there is no reason to worry that the instrument performance would be in jeopardy.

Sigma-metrics according to current CLIA performance specifications (established 1992)

Finally, the oldest set of performance specifications grades all Alinity assays at above 4 Sigma, with no assays at 3 Sigma or lower.

Conclusion

Even though this instrument fares better than the others in our series, the latest desirable performance specifications derived from EuBIVAS are quite a challenge to achieve.