Sigma Metric Analysis

Recent posters offer a glimpse of the new Atellica performance. Can this new instrument meet the latest EFLM goals for performance?  

Sigma-metric Analysis of an Atellica analyzer in Brazil

Sten Westgard, MS
October 2019, Corrected November 2019

• The Precision, Comparison and Sigma-metric data
• Summary of Performance according to EFLM, CLIA and Ricos Goals by Sigma-metrics Normalized Method Decision charts
• Conclusion

[Note: This QC application is an extension of the lesson From Method Validation to Six Sigma: Translating Method Performance Claims into Sigma Metrics. This article assumes that you have read that lesson first, and that you are also familiar with the concepts of QC Design, Method Validation, and Six Sigma metrics. If you aren't, follow the link provided.]

[Important correction: a previous version of this article contained technical errors. These have been corrected and we regret the mistakes. Other requests for clarifications about the scope and intention of the article were made - we have attempted to address those questions and concerns.]

See other analyses in this series:

• Multimode Analysis of a Cobas 8000
• Multimode Analysis of a DxC 700
• Multimode Analysis of an Abbott Alinity

At the AACC conference there were two posters from a Brazilian laboratory about Atellica instruments.

Performance Evaluation of Electrolytes and Renal Function Biomarkers Assays on the Atellica CH930, Deise Waltrick, Livia Avallone, Lucilene Rodrigues et al. DASA

Performance Evaluation of Hepatic Function Biomarkers Assays on the Atellica CH930 and Atellica IM 1600 Analyzers, Deise Waltrick, Livia Avallone, Lucilene Rodrigues et al. DASA

Since these represent the most advanced instrumentation on the marketplace, it's a good opportunity to check how the latest performance matches up with the latest goals. The EFLM (European Federation of Laboratory Medicine) has a new website, which posts the latest information on biological variation for various analytes. The approach is also known as EuBIVAS. There is some confusion over what goals can be derived from this database, as there are individual published articles, meta-studies, and the actual biological database itself. We will note which reference is used in the table. For simplicity's sake in the table, we will refer to all of them as "EFLM database goals".

As an approach, however, we are attempting to perform an "acid test" - take the most demanding desirable goals that are coming from EFLM, EuBIVAS, and recently published papers.  The point here is not to flog the manufacturer - but to identify which goals may be most appropriate, realistic, and practical for the latest instrumentation in the field.

The key question: can the latest instrument from the diagnostic industry meet the latest performance specifications from the scientific community?

The Imprecision, Bias and Sigma-metric Data

"The precision study was performed according to EP15-A3, the method comparison according to EP09-0A3....For the precision study, two concentration ranges were used: each level of QC material was tested in one run per day with five replicates per run for five days, resulting in a total of 25 replicates per sample for each run. Method comparison studies were performed using at least 40 serum samples that covered the linearity of the method."

Sigma-metric calculations for the Atellica

Remember the equation for Sigma metric is (TEa - bias) / CV:

For a 3.5% quality requirement, the latest EFLM specification, for Albumin on the normal level of the Atellica, the equation is (3.5 - 0.5) / 1.7 = 1.8

In contrast, for a 10% quality requirement, the CLIA goal, for Albumin on the normal level of the Atellica, the equation is (10 - 0.5) / 1.7 = 5.6

 Just with this one example, we can see the major difference in judging Atelllica performance by EFLM or CLIA standards.

 First, thought, let's calculate the performance by the 2019 EFLM desirable specifications for performance based on biological variation:

Negative Sigma-metrics at certain levels. Very low Sigma-metrics at others. This is not a good sign for a "next generation" analyzer, unless the goals being used to judge the instrument are not appropriate.

* for Uric Acid, the database doesn't have an entry (11/21/2019) but a paper from the group does: EuBIVAS: Within- and Between-Subject Biological Variation Data for Electrolytes, Lipids, Urea, Uric Acid, Total Protein, Total Bilirubin, Direct Bilirubin, and Glucose, Aarsand AK, Diaz-Garzon J, Fernandez-CAlle P et al, Clin Chem 64:9 2017

Then if we believe the EFLM goals are too challenging, we can instead use a set of goals we have adopted for global performance standards at the Westgard Sigma Verification of Performance Program (WSVP), which contain a few EFLM goals from earlier years, as well as some CLIA goals. CLIA goals, according to common wisdom, are more permissive, and some even say, too loose.

Note: The above TEa goals correspond with the Sigma Verification of Performance (Sigma VP) Program goals of 2019.  So they are a bit tighter than CLIA in some places but they are not nearly as challenging as EFLM goals in others. The result? more assays at 6 Sigma and higher.

Now, for some observers, the Westgard Sigma Verification goals may not satisfy. They might prefer to look at "pure" sets of goals such as, All CLIA 1992 goals, All CLIA proposed 2019 goals, and/or all Ricos 2014 goals. Don't worry, we have them all worked out here.

 There are even more Six Sigma assays when Atellica is judged by CLIA 1992 TEa goals. This instrument is well positioned to meet the current compliance requirements of US laboratories. Of course, there aren't as many TEa goals set by CLIA from 1992, GGT and CEA are not directly regulated in those guidelines.

What about the proposed update to goals for CLIA?

The proposed goals for updating CLIA are a bit more challenging. The new goals include specifications for CEA and GGT, so we can see the improvements. But this Atellica looks poised to be able to achieve compliance with updates to CLIA if the specifications are in fact made law.

Finally, here is the assessment if the desirable specifications are created from the Ricos 2014 biological variation database:

Summary of Performance by Sigma-metrics Method Decision Chart using WSVP, CLIA and EFLM Goals

We can make visual assessments of this performance using a Normalized Sigma-metric Method Decision Chart:

This graph shows that about a third of the levels measured, using EFLM 2019 goals, is 3 Sigma or lower.

 Performance by Westgard Sigma VP standards is much better, we would expect that 8 to 9 of the 11 assays would meet the verification criteria.

 CLIA 1992 PT goals are incrementally more forgiving to the Atellica performance.

If the proposed tightening of CLIA goals carries forward, a few more levels will be considered less ideal in performance.

The original "Ricos goals" are about as demanding as the EFLM 2019 goals, with 7 of the 22 measured levels at 3 Sigma or lower.

Conclusion

The authors of the posters conclude that the Atellica performance is uniformly acceptable, but even in the case of the more lenient (or realistic) goals from CLIA, there are assays and levels that do have unacceptable performance. It's possible that laboratories might decide those specific levels are not relevant to clinical care, and that the Sigma-metrics at other levels are more germane. With the Sigma VP goals, performance is definitely more acceptable than what we see using the EFLM goals.

Would the Atellica performance be considered the best choice for a new chemistry analyzer? Individual laboratories will have to make that decision, depending on which goals are being adopted as benchmarks.

The greater issue is to raise concerns about the viability of new performance specifications derived from "EFLM goals". The desirable performance specifications that can be calculated from the database, as stated in earlier articles, are so tight that even the latest instrument is challenged to meet them. Does this invalidate the goals, the instrument, or both? Perhaps a new version of performance specifications are necessary, ones that are no longer "optimal" "desirable" or "minimum" as the Ricos goals from 1999-2014 formulated (according to the path laid out by Callum Fraser). Perhaps there is a need for "achievable" goals.