Tools, Technologies and Training for Healthcare Laboratories

Mindray BS-2000 and Beckman AU 5800: Sigma metrics of Reagents and Reagent swaps

What happens when you swap reagents between two analyzers? Do the instruments become "strangers on a train"? Or is Reagent "Freaky Friday" what results?

Mindray BS-2000 and Beckman AU 5800: Sigma Metric Analysis of reagents and reagent swapping

Sten Westgard, MS
March 2021

An intriguing paper was published recently in Journal of Clinical Laboratory Analysis. Not only did it evaluate the quality of two instruments when using their validated (manufacturer-designed) reagents, it also studied what happened when the reagents were swapped. That is, what happens on Instrument X when you the the reagent from Manufacturer Y? And vice-versa.

In other words, it's Reagent Freaky Friday.

Sigma metrics application for validated and non-validated detecting systems performance assessment. Yong Xia, Mingyang Li, Bowen Li, Hao Xue, Yu Lin, Jie Li, Ling Ji. J Clin Lab Ana 2021 Mar;35(3):e23676. DOI: 10.1002/jcla.23676

Of course, the authors use a much more dignified terminology: they use the term "Validated" to indicate an instrument that's using the original manufacturer's reagent. When it's something other than that, they use the term "non-validated."

So this study is really not just a reagent swap, it's another look at the performance of third-party reagents. When you use someone else's reagent, do you get the same performance?

The goals for performance

The study was conducted in China, at Peking University Shenzhen Hospital. So it's no surprise that they used WS/T403-2012 performance specifications, which are the rule of the land in China. For our purposes, we will apply additional performance specifications that are more relevant to countries outside China, such as the original CLIA requirements, the update to CLIA requirements that was proposed in 2019, the Ricos goals of 2014, and the latest (2021) goals of the EFLM EuBIVAS database.

It's worth stacking these goals up and taking a moment to discuss them.

Assay CLIA goal CLIA 2019 goal Ricos 2014 goal EuBIVAS 2021 goal
ALT 20 15 27.5 19.51
LDH 20 15 11.4 7.7
Glucose 10 8 7 8.2
Total Protein 10 8 3.6 4.35
Total Cholesterol 10 10 9 10.46
Triglycerides 25 15 26 33.84

Within this small set of tests, some of our common wisdom about the goals gets upended. We are used to thinking that the Ricos and EuBIVAS goals are the tightest, but actually on ALT and Triglycerides, the biological goals are wider than the regulatory goals. We also see that for total cholesterol and glucose we can see an equilibrium seems to be close - the biological and regulatory goals are very close to one another.  Total protein lives up to the traditional reputation of the biological goals: too tight to be achieved by any method currently on the market. Finally, it seems that the years are starting to weigh on the EuBIVAS goals - once they were very tight, now as more studies and data are collected, they are "widening out". In this scenario, more of the Ricos 2014 goals are the tightest goals than the EuBIVAS goals.

The performance data

This study did one thing really, really well: selection of the materials to analyze. They chose trueness verification materials, which are essentially reference materials, from the NCCL, the National Center for Clinical Laboratories of China. So this gives us human serum materials that are given expected values - our bias estimate is therefore much better than a simple field-method-to-field-method comparison. No worries here about using manufacturer controls, independent 3rd party controls

What's not quite as good is the length of the study. They followed the CLSI EP15-A2 short term precision verification study, which measures three replicates in five runs. That's not a great number of data points, and while it can be sufficient to verify a method, it's a bit under the usual standards for establishing a CV. It would have been better to test more samples for more runs, as the CLSI EP15-A3 now recommends, or to go even longer and follow CLSI EP5's protocol for imprecision. The shortcoming of this approach is that the imprecision estimate (CV) will be less reliable, and it might be optimistic (a bit smaller than a longer study - and routine operation - would find).

Now bear in mind, there are 4 different sets of data available for this study,

  • Beckman reagent on the Mindray instrument (thus, non-verified),
  • Mindray reagent on the Beckman instrument (also thus, non-verified), and then
  • Mindray reagent on the Mindray instrument (verified), and
  • Beckman reagent on the Beckman instrument (verified)

Below is the performance observed for the latter. This is Beckman reagent on the Beckman instrument:

Assay %Bias %CV
ALT lvl 1 0.48 1.10
ALT lvl 2 2.22 1.30
Cholesterol lvl 1 2.25 0.77
Cholesterol lvl 2 2.11 0.99
Glucose lvl 1 0.83 1.84
Glucose lvl 2 0.87 0.98
LDH lvl 1 1.41 0.80
LDH lvl 2 0.26 1.30
Total Protein lvl 1 0.74 1.90
Total Protein lvl 2 2.41 1.90
Triglycerides lvl 1 0.84 1.22
Triglycerides lvl 2 0.24 0.93

Even though we're only dealing with 6 assays, that's still a lot of numbers. And then to think, we're calculating not just one Sigma-metric for each set of observed performance, but 4 different Sigma-metrics (CLIA, CLIA 2019, Ricos 2014, and EuBIVAS 2021). We are potentially generating 16 different tables and could construct 16 different Method Decision (MEDx) charts.

As much as we love data and tables and graphs at Westgard QC, we recognize when data overwhelms the eye.

So we are just going to show a select few of the graphs.

How do these instruments look when we use the most up-to-date goals (EuBIVAS 2021) ?

Here's Beckman Reagent and Instrument performance:

 2021 Beckman on Beckman EuBIVAS nmedx

Those two red points, that's the total protein assay. (Remember, that's the hardest goal to hit in this scenario).

Now here is the Mindray reagent on the Beckman instrument:

2021 Mindray on Beckman EFLM2021 nmedx

Quite interesting to see that Mindray's reagent performs pretty well, and in some ways, even better than the Beckman reagent (not even the total protein assay is in the "red zone" below 2 Sigma).

Now here is Mindray reagent on the Mindray instrument:

2021 Mindray on Mindray EuBIVAS 2021 nmedx 

Mindray is doing very well, very comparable to the Beckman on Beckman, and again no "red zone" performance.

Finally, what does the Beckman reagent do when it's on the Mindray instrument:

2021 Beckman on Mindray EuBIVAS2021 nmedx

Of the four combinations, this is the worst. If you're trying to hit EuBIVAS 2021 goals, don't put Beckman reagent on a Mindray instrument.

How do these instruments and reagents look when we use the most lenient goals (CLIA) ?

 Let's return to the Beckman reagent on the Beckman instrument:

2021 Beckman on Beckman CLIA nmedx

Almost everything is in the bull's-eye. Nothing is below 3 Sigma.

What about the Mindray-Mindray combination?

2021 Mindray on Mindray CLIA nmedx

Actually, Mindray is even better. No assays at 3 Sigma or 4 Sigma. Everything is 5 Sigma and higher.

Now, Mindray reagent on the Beckman instrument?

2021 Mindray on Beckman Classic CLIA nmedx

This is not only very good, it's actually better than the performance of Beckman reagent on Beckman instrumentation!

Finally, what happens when Beckman's reagent is used on the Mindray instrument?

2021 Beckman on Mindray CLIA nmedx

While this qualifies as the worst peformance of the set, even this is pretty good, with no unacceptable assays, and a majority of assay performance in the Six Sigma zone.

Conclusion: By any standard, Mindray is a real competitor to Beckman. Reagent swapping is (almost) never better than the original. And CLIA is a standard by which Mindray may even be better than Beckman on Beckman

Let's summarize the findings of the metrics and the goals.

Ranking EuBIVAS 2021 goals CLIA goals
Best Beckman on Beckman Mindray on Mindray
Better Mindray on Beckman Mindray on Beckman
Worse Mindray on Mindray Beckman on Beckman
Worst Beckman on Mindray Beckman on Mindray

For a long time, it's been common to dismiss Chinese instruments as poor quality. But as they have proven in so many other fields, they learn fast, and it appears that they are now reaching parity with the "traditional" diagnostic manufacturers. When the standard CLIA goals are used to judge performance, Mindray can even win the day.

Now the size of this study (few days, few runs, few analytes) may mean that some of the results are irreproducible. The finding that Mindray reagent on the Beckman instrumentation can even out perform the original manufacturer reagent - we would be really curious to see if that can be reproduced anywhere else.

But it is clear that (1) reagent swapping almost never improves the situation. Further evidence that 3rd party reagents are not a quality choice. (2) Mindray is a real competitor in the chemistry space.

We look forward to seeing future studies that clarify just how good a competitor Mindray is.