Tools, Technologies and Training for Healthcare Laboratories

WESTGARD WEB

BS-480 Sigma-metric Analysis

A recent study assessed the performance of a new compact chemistry analyzer, the Mindray BS480.  

Sigma-metric Analysis of Mindray BS480

Sten Westgard, MS
September 2018

[Note: This QC application is an extension of the lesson From Method Validation to Six Sigma: Translating Method Performance Claims into Sigma Metrics. This article assumes that you have read that lesson first, and that you are also familiar with the concepts of QC Design, Method Validation, and Six Sigma metrics. If you aren't, follow the link provided.]

Performance and Suitability of Mindray BS480: A Fully Open Clinical Chemistry Analyzer Anne-Marie Dupuy, Maelle Plawecki, Anne-Sophie Bargnoux, Stephanie Badiou, Martine Delage, Manuela Lotierzo, Jean-Paul Cristol.

The Imprecision, Bias and Sigma-metric Data

"Imprecision is carried out according to CLSI EP15-A3 guidelines from a pool of plasma/serum or urine patient samples with different levels. An Aliquot of each level thawed just before analysis was assayed in duplicate, twice a day, on the basis of a single calibration during 20 consecutive days. Final results are expressed as the coefficient of variation (CV%)....

"Analytical methods are compared using plasma or serum and uring patient samples under simulated routine conditions. Mindray BS480 was compared with Cobas 8000 for different routine parameters such as glucose, ALT, AST, creatinine, CRP, ionogram (Sodium, potassium, chloride and calcium) in serum and plasma, both turbidimetric and colorimetric albumin in serum and urine. Comparison between Mindray BS480 and ABX Pentra 400 was performed with PYR and AA samples thawed from -80C."

Three levels of control performance were recorded. CV and Bias was listed in Table 2 of the paper:

Mindray BS480   
TEST TEa Source TEa Level Slope Y-intercept % Bias %CV
Albumin colorimetry CLIA 10.0 35 1.07 -4.8 6.7% 2.9%
CLIA 10.0 25 1.07 -4.8 12.2% 7.9%
Albumin turbidimetry CLIA 10.0 35 0.74 8.5 1.7% 2.8%
CLIA 10.0 25 0.74 8.5 11.0% 4.1%
ALT EFLM 2018 14.4 204 1.07 1.33 7.7% 14.7%
ALT EFLM 2018 14.4 18 1.07 1.33 14.4% 15.3%
AST EFLM 2018 13.4 387 1.07 0.89 7.2% 1.2%
AST EFLM 2018 13.4 27 1.07 0.89 10.3% 2.4%
CRP Ricos 2014 56.6 8.6 1.22 -0.36 17.8% 1.7%
CRP Ricos 2014 56.6 2.8 1.22 -0.36 9.1% 2.0%
Calcium CLIA 7.23% 3.45 0.94 0.05 4.6% 1.7%
Calcium CLIA 11.6% 2.16 0.94 0.05 3.7% 1.4%
Chloride CLIA 5.0 105 1.05 -2.9 2.2% 1.0%
Chloride CLIA 5.0 81 1.05 -2.9 1.4% 1.9%
Creatinine CLIA 15% or 0.3 mg/dL7.00%7 15.0 257  1.01 -4.9 0.9% 0.4%
Creatinine 35.84 74  1.01  -4.9  5.6% 3.1%
Creatinine 57.65 46  1.01  -4.9  9.7% 3.3%
Glucose CLIA 10.0 8 1.03 -0.54 3.8% 3.5%
Gluocse CLIA 10.0 5 1.03  -0.54 7.8% 3.8%
Potassium CLIA 7.14 7 1.06 -0.36 0.9% 2.0%
Potassium +/- 0.5 12.1 4.13 1.06 -0.36 2.7% 1.5%
Sodium CLIA 2.82 142 1.04 -2.37 2.3% 1.1%
Sodium +/- 4 mmol 3.28 122 1.04 -2.37 2.1% 0.8%

 Yes, that is a whole lot of numbers!

Nevertheless, what do all these numbers mean? In the absence of context, it's hard to know.

So let's calculate the Sigma-metrics.

Sigma-metric calculations for the BS480

Remember the equation for Sigma metric is (TEa - bias) / CV:

For a 10% quality requirement, for Albumin colorimetry on the low level of BS480, the equation is (10 - 6.7) / 2.9 = 1.1

For a 10% quality requirement, for Albumin colorimetry on the high level of BS480, the equation is (10 - 12.2) / 7.9 = negative

The metrics are displayed along the right column.

Mindray BS480   
TEST TEa Source TEa Level Slope Y-intercept % Bias %CV Sigma metric
Albumin colorimetry CLIA 10.0 35 1.07 -4.8 6.7% 2.9% 1.1
CLIA 10.0 25 1.07 -4.8 12.2% 7.9% negative
Albumin turbidimetry CLIA 10.0 35 0.74 8.5 1.7% 2.8% 2.96
CLIA 10.0 25 0.74 8.5 11.0% 4.1% negative
ALT EFLM 2018 14.4 204 1.07 1.33 7.7% 14.7% 0.5
ALT EFLM 2018 14.4 18 1.07 1.33 14.4% 15.3% 0.0
AST EFLM 2018 13.4 387 1.07 0.89 7.2% 1.2% 5.0
AST EFLM 2018 13.4 27 1.07 0.89 10.3% 2.4% 1.3
CRP Ricos 2014 56.6 8.6 1.22 -0.36 17.8% 1.7% 22.2
CRP Ricos 2014 56.6 2.8 1.22 -0.36 9.1% 2.0% 24.2
Calcium CLIA 7.23% 3.45 0.94 0.05 4.6% 1.7% 1.5
Calcium CLIA 11.6% 2.16 0.94 0.05 3.7% 1.4% 5.7
Chloride CLIA 5.0 105 1.05 -2.9 2.2% 1.0% 2.7
Chloride CLIA 5.0 81 1.05 -2.9 1.4% 1.9% 1.9
Creatinine CLIA 15% or 0.3 mg/dL7.00%7 15.0 257  1.01 -4.9 0.9% 0.4% 40.2
Creatinine 35.84 74  1.01  -4.9  5.6% 3.1% 9.8
Creatinine 57.65 46  1.01  -4.9  9.7% 3.3% 14.7
Glucose CLIA 10.0 8 1.03 -0.54 3.8% 3.5% 1.8
Glucose CLIA 10.0 5 1.03  -0.54 7.8% 3.8% 0.58
Potassium CLIA 7.14 7 1.06 -0.36 0.9% 2.0% 3.1
Potassium +/- 0.5 12.1 4.13 1.06 -0.36 2.7% 1.5% 6.5
Sodium CLIA 2.82 142 1.04 -2.37 2.3% 1.1% 0.4
Sodium +/- 4 mmol 3.28 122 1.04 -2.37 2.1% 0.8% 1.5

 Yes, there are a lot of low Sigma-metrics. Negative Sigma is unusual, reflecting a situation when the bias completely exceeds the allowable total error. There are a few high Sigma-metrics, but the majority of the numbers are below 3 Sigma, typically considered the minimum acceptable performance.

 Even if we don't use Sigma-metrics and total analytical total allowable errors to judge these methods, we can use the Ricos 2014 desirable specifications for imprecision and bias, as well as the updated 2017-2018 EFLM desirable specifications for imprecision and bias. We'll highlight in red the performance where the instrument failed to achieve those goals, and highlight in green the areas where the instrument did achieve those goals.

Mindray BS480  compared to Ricos and EFLM 2017-2018 desirable specifications for CV and bias
TEST TEa Source TEa Level Slope Y-intercept Des Bias % Bias Des CV %CV
Albumin colorimetry CLIA 10.0 35 1.07 -4.8 1.43 6.7% 1.6 2.9%
CLIA 10.0 25 1.07 -4.8 1.43 12.2% 1.6 7.9%
Albumin turbidimetry CLIA 10.0 35 0.74 8.5 1.43 1.7% 1.6 2.8%
CLIA 10.0 25 0.74 8.5 1.43 11.0% 1.6 4.1%
ALT EFLM 2018 14.4 204 1.07 1.33 6.72 7.7% 4.65 14.7%
ALT EFLM 2018 14.4 18 1.07 1.33 6.72 14.4% 4.65 15.3%
AST EFLM 2018 13.4 387 1.07 0.89 5.6 7.2% 4.75 1.2%
AST EFLM 2018 13.4 27 1.07 0.89 5.6 10.3% 4.75 2.4%
CRP Ricos 2014 56.6 8.6 1.22 -0.36  21.1 17.8% 21.8 1.7%
CRP Ricos 2014 56.6 2.8 1.22 -0.36  21.1 9.1% 21.8 2.0%
Calcium CLIA 7.23% 3.45 0.94 0.05  0.91 4.6% 0.79 1.7%
Calcium CLIA 11.6% 2.16 0.94 0.05  0.91 3.7% 0.79 1.4%
Chloride CLIA 5.0 105 1.05 -2.9  0.49 2.2% 0.42 1.0%
Chloride CLIA 5.0 81 1.05 -2.9  0.49 1.4%  0.42 1.9%
Creatinine CLIA 15% or 0.3 mg/dL7.00%7 15.0 257  1.01 -4.9  2.2 0.9%  2.8 0.4%
Creatinine 35.84 74  1.01  -4.9   2.2 5.6%  2.8 3.1%
Creatinine 57.65 46  1.01  -4.9   2.2 9.7%  2.8 3.3%
Glucose CLIA 10.0 8 1.03 -0.54  2.35 3.8%  2.13 3.5%
Glucose CLIA 10.0 5 1.03  -0.54  2.35 7.8%  2.13 3.8%
Potassium CLIA 7.14 7 1.06 -0.36  1.96 0.9%  1.41 2.0%
Potassium +/- 0.5 12.1 4.13 1.06 -0.36  1.96 2.7%  1.41 1.5%
Sodium CLIA 2.82 142 1.04 -2.37 0.27 2.3% 0.27  1.1%
Sodium +/- 4 mmol 3.28 122 1.04 -2.37 0.27  2.1% 0.27  0.8%

As you can see, most of these assays are NOT meeting the desirable specifications set forth by Ricos and EFLM.

Summary of Performance by Sigma-metrics Method Decision Chart using CLIA and Ricos Goals

We can make visual assessments of this performance using a Normalized Sigma-metric Method Decision Chart:

2018 BS480 NMEDX

Overall, you can see that many of the x-coordinates (or imprecision) are on the right side of the graph, in fact a few of them are "off the chart" to the right if we plotted them where they should be, which is a very bad sign. Any points that you see tucked at the top edge of the graph are "off the charts" in the vertical axis, because of the significant bias which exceed the allowable total error. A few have both more than the allowable bias and more than the allowable imprecision.

Summary of QC Design by Normalized OPSpecs chart - using CLIA and Ricos Goals

The benefit of the Sigma-metric approach is that labs can do more than assess their quality, they can act on it. Using OPSpecs charts, they can actually optimize their QC procedures for each test. In this case, they can use the data to try and mitigate the risk of poor performance.

2018 BS480 NOPSPECS

For many assays, there is no statistical quality control procedure that can help. There aren't enough "Westgard rules" to keep some of these methods in control. 

 

Conclusion

The authors stated "In conclusion, Mindray BS480 provides precise measurements for a variety of analytes, including routine chemistry, ISE and specific proteins. It is a reliable analyzer with the need for a low sample volume, which is particularly suitable for pediatric samples. The possibility to adapt some methodologies is a great convenience, leading to transfer from manual methodology to automation. Besides its great versatility, Mindray BS480 is very user friendly and allows great flexibility in programming possibilities, thus opening the field to other parameter adaptations. The system is a multi-parametric analyzer that is able to perform almost all parameters based on liquid chemistries on a single workstation."

Given the evidence of Sigma-metric analysis as well as the comparison of these parameters to desirable specifications for imprecision and bias based on Ricos and EFLM, there is very little reason to endorse this instrument. High imprecision, High bias contradict the claims of reliability. Convenience is not a good substitute for accuracy and precision.

It is becoming all too clear that many papers written about new instruments suffer from "publication bias", that is, the conclusions are directly impacted by the authors' and sponsor desires to produce a favorable outcome and judgment.

Joomla SEF URLs by Artio

About this website

WESTGARD QC promotes the latest news, education, and tools in the quality control field. Our goal is to bring tools, technology and training into today's healthcare industry — by featuring QC lessons, QC case studies and frequent essays from leaders in the quality control area. This is also a reference source for quality requirements, including CLIA requirements for analytical quality. This website features the best explanation of the Multirule ("Westgard Rules") and how to use them. For laboratory and healthcare professionals looking for educational and reference material in the quality control field.

THIS IS THE WEBSITE FOR YOU!