Sigma Metric Analysis

A recent study assessed the performance of a new compact chemistry analyzer, the Mindray BS480.  

Sigma-metric Analysis of Mindray BS480

Sten Westgard, MS
September 2018

• The Precision, Comparison and Sigma-metric data
• Summary of Performance according to CLIA and Ricos Goals by Sigma-metrics Normalized Method Decision charts
• QC Implications of Mindray BS480 Performance by Normalized OPSpecs chart
• Conclusion

[Note: This QC application is an extension of the lesson From Method Validation to Six Sigma: Translating Method Performance Claims into Sigma Metrics. This article assumes that you have read that lesson first, and that you are also familiar with the concepts of QC Design, Method Validation, and Six Sigma metrics. If you aren't, follow the link provided.]

Performance and Suitability of Mindray BS480: A Fully Open Clinical Chemistry Analyzer Anne-Marie Dupuy, Maelle Plawecki, Anne-Sophie Bargnoux, Stephanie Badiou, Martine Delage, Manuela Lotierzo, Jean-Paul Cristol.

The Imprecision, Bias and Sigma-metric Data

"Imprecision is carried out according to CLSI EP15-A3 guidelines from a pool of plasma/serum or urine patient samples with different levels. An Aliquot of each level thawed just before analysis was assayed in duplicate, twice a day, on the basis of a single calibration during 20 consecutive days. Final results are expressed as the coefficient of variation (CV%)....

"Analytical methods are compared using plasma or serum and uring patient samples under simulated routine conditions. Mindray BS480 was compared with Cobas 8000 for different routine parameters such as glucose, ALT, AST, creatinine, CRP, ionogram (Sodium, potassium, chloride and calcium) in serum and plasma, both turbidimetric and colorimetric albumin in serum and urine. Comparison between Mindray BS480 and ABX Pentra 400 was performed with PYR and AA samples thawed from -80C."

Three levels of control performance were recorded. CV and Bias was listed in Table 2 of the paper:

 Yes, that is a whole lot of numbers!

Nevertheless, what do all these numbers mean? In the absence of context, it's hard to know.

So let's calculate the Sigma-metrics.

Sigma-metric calculations for the BS480

Remember the equation for Sigma metric is (TEa - bias) / CV:

For a 10% quality requirement, for Albumin colorimetry on the low level of BS480, the equation is (10 - 6.7) / 2.9 = 1.1

For a 10% quality requirement, for Albumin colorimetry on the high level of BS480, the equation is (10 - 12.2) / 7.9 = negative

The metrics are displayed along the right column.

 Yes, there are a lot of low Sigma-metrics. Negative Sigma is unusual, reflecting a situation when the bias completely exceeds the allowable total error. There are a few high Sigma-metrics, but the majority of the numbers are below 3 Sigma, typically considered the minimum acceptable performance.

 Even if we don't use Sigma-metrics and total analytical total allowable errors to judge these methods, we can use the Ricos 2014 desirable specifications for imprecision and bias, as well as the updated 2017-2018 EFLM desirable specifications for imprecision and bias. We'll highlight in red the performance where the instrument failed to achieve those goals, and highlight in green the areas where the instrument did achieve those goals.

As you can see, most of these assays are NOT meeting the desirable specifications set forth by Ricos and EFLM.

Summary of Performance by Sigma-metrics Method Decision Chart using CLIA and Ricos Goals

We can make visual assessments of this performance using a Normalized Sigma-metric Method Decision Chart:

Overall, you can see that many of the x-coordinates (or imprecision) are on the right side of the graph, in fact a few of them are "off the chart" to the right if we plotted them where they should be, which is a very bad sign. Any points that you see tucked at the top edge of the graph are "off the charts" in the vertical axis, because of the significant bias which exceed the allowable total error. A few have both more than the allowable bias and more than the allowable imprecision.

Summary of QC Design by Normalized OPSpecs chart - using CLIA and Ricos Goals

The benefit of the Sigma-metric approach is that labs can do more than assess their quality, they can act on it. Using OPSpecs charts, they can actually optimize their QC procedures for each test. In this case, they can use the data to try and mitigate the risk of poor performance.

For many assays, there is no statistical quality control procedure that can help. There aren't enough "Westgard rules" to keep some of these methods in control. 

Conclusion

The authors stated "In conclusion, Mindray BS480 provides precise measurements for a variety of analytes, including routine chemistry, ISE and specific proteins. It is a reliable analyzer with the need for a low sample volume, which is particularly suitable for pediatric samples. The possibility to adapt some methodologies is a great convenience, leading to transfer from manual methodology to automation. Besides its great versatility, Mindray BS480 is very user friendly and allows great flexibility in programming possibilities, thus opening the field to other parameter adaptations. The system is a multi-parametric analyzer that is able to perform almost all parameters based on liquid chemistries on a single workstation."

Given the evidence of Sigma-metric analysis as well as the comparison of these parameters to desirable specifications for imprecision and bias based on Ricos and EFLM, there is very little reason to endorse this instrument. High imprecision, High bias contradict the claims of reliability. Convenience is not a good substitute for accuracy and precision.

It is becoming all too clear that many papers written about new instruments suffer from "publication bias", that is, the conclusions are directly impacted by the authors' and sponsor desires to produce a favorable outcome and judgment.