Tools, Technologies and Training for Healthcare Laboratories

Evaluation of Mindray BC 6800 Hematology analyzer

A 2014 letter to the editor of the International Journal of Laboratory Hematology evaluated the performance of the Mindray automated hematology analyzer BC 6800. The letter authors concluded this instrument is "good and precise." Does Sigma-metric analysis agree?

Sigma-metrics of Mindray BC 6800 automated hematology analyzer

January 2015
Sten Westgard, MS

[Note: This QC application is an extension of the lesson From Method Validation to Six Sigma: Translating Method Performance Claims into Sigma Metrics. This article assumes that you have read that lesson first, and that you are also familiar with the concepts of QC Design, Method Validation, and Six Sigma. If you aren't, follow the link provided.]

[July 2016 Editor's note: After protests from Mindray, we have clarified the terminology of the conclusion to so that it is more metrologically correct.]

This analysis looks at an analyzer less well-known to the US and Europe: a Mindray BC 6800 automated hematology analyzer:

Evaluation of the fully automated hematological analyzer Mindray BC 6800: comparison with Horiba ABX Pentra DX120. Grillone R, Grimaldi E, Scopacasa F, Dente B. Int Jnl Lab Hem2014;36:e55-e58.

The Imprecision and Bias Data

For imprecision, "20 samples were analyzed on the BC-6800 in triplicate and in two batches on the same day. The instrument was switched off and recalibrated between the two batches to better mimic different batches and to emphasize potential imprecision." Only one level of imprecision was reported.

 For bias, the study authors "selected 1025 blood samples on the basis of EP9 protocol range of values and analyzed them side-by-side with the BC-6800 and Pentra 120 instruments once." 

Assay Level
CV% slope y-intercept Bias%
WBC 9.31 2.1% 0.8950 -0.0893  
RBC 4.7 2.35% 0.9720 -0.0008  
HGB 14.10 0.76% 0.9280 0.7889  
HCT 29.4 4.0%      
MCV 262.8 2.54% 1.0810 -.5135  
PLT 0.39 4.84% 0.8745 -3.4899  
Neutrophils 0.63 4.2% 0.8716 0.0317  
Lymphocytes 6.46 4.38% 0.8099 0.0607  
Monocytes 110 2.17% 1.0208 -0.0258  
Eosinophils 98.9 1.03% 0.7286 0.0480  
Basophils 1.19 3.97% 2.0 0.0090  

Notice, we haven't calculated the bias yet. Since we only have one level of imprecision, that's the level where we'll calculate bias.

The study shows the correlation coefficient, slope and y-intercept. The regression equation can be used to determine the difference between the the Pentra and the Mindray.

Newlevel = (slope * Oldlevel ) + Y-intercept

As an example, let's take WBC, where the study determined a slope of 0.895 and y-intercept of -0.0893.

Newlevel1 = (0.895 * 9.31 ) - 0.0893

Newlevel1 = (8.33245) -0.0893

Newlevel1 = 8.24315

The bias between the old and new level is the absolute value of the difference between 9.31 - 8.24315 = 1.067

This is an 11.46% bias at the level of 9.31.

 

Now we'll just fill in all the biases...

Assay Level
CV% slope y-intercept Bias%
WBC 9.31 2.1% 0.8950 -0.0893  11.46%
RBC 4.7 2.35% 0.9720 -0.0008  2.82%
HGB 14.10 0.76% 0.9280 0.7889  1.60%
HCT 42.8 4.7%      
MCV 91.0 0.76% 1.0810 -.5135  6.44%
PLT 270 1.9% 0.8745 -3.4899  13.84%
Neutrophils 19.69 2.0% 0.8716 0.0317  12.68%
Lymphocytes 1.8 2.2% 0.8099 0.0607  15.64%
Monocytes 0.71 4.2% 1.0208 -0.0258  1.55%
Eosinophils 0.06 16.7% 0.7286 0.0480  52.86%
Basophils 0.04 25.0% 2.0 0.0090  122.5%

Note that there was no comparison data included for HCT, so we will only have the imprecision data to work with.

Now you may be wondering about these bias numbers, because some of them are quite large. For example, the Eosinophils bias is over 50%, while the Basophils bias is more than 120%. What's happening here? The Slope is 2.0 and the y-intercept is 0.0090. But if you have a slope of >2, that means the new method is increasing twice as fast along the range as the comparative method. Even if you've got a great correlation coefficient, that's a whole lot of proportional error. Now keep in mind this Mindray is being compared to the Pentra. What the bias numbers are telling us is the two instruments are not getting the same answer.

Determine Quality Requirements at the decision levels

Now that we have our imprecision and bias data, we're almost ready to calculate our Sigma-metrics. We're just missing one key thing: the analytical quality requirement. In this example, we're going to use several different sets of quality requirements to judge the method. We'll start by using the "Ricos Goals" for imprecision and bias separately - we'll not even take into account the Total Error of the methods. Second, we'll look at the Ricos Desirable Total Allowable Errror and use that to make Sigma-metric calculations. Finally, we'll use the CLIA proficiency testing criteria, which sets specifications for some hematology parameters. Where CLIA doesn't regulate an analyte (for example Eosinophils), we'll use the "Ricos goals" again.

Here's a comparative table of the different quality goals available:

Assay Ricos CV%
Ricos Bias% Ricos TEa% CLIA TEa% Spanish
Minimum TEa%
Rilibak TEa%
WBC 5.73% 6.05%  15.44% 15% 9.0% 12.0%
RBC 1.6% 1.7% 4.4% 6.0% 4.0% 8.0%
HGB 1.43% 1.84% 4.19% 7.0% 5.0% 6.0%
HCT 1.35% 1.74% 3.97% 6.0% 8.0% 9.0%
MCV 0.7% 1.26% 2.42%   7.0%  
PLT 4.6% 5.9% 13.4% 25.0% 16.0%  
Neutrophils 8.55% 9.25% 23.35%      
Lymphocytes 5.1% 9.19% 17.6%      
Monocytes 8.9% 13.2% 27.9%      
Eosinophils 10.5% 19.8% 37.1%      
Basophils 14.0% 15.4% 38.5%      

Judging on imprecision and bias separately, using the Ricos desirable maximum allowable specifications, we'll highlight in green where performance is acceptable, and highlight in red where performance is unacceptable

Assay Ricos CV% Ricos Bias% Level
CV% slope y-intercept Bias%
WBC  5.73% 6.05% 9.31 2.1% 0.8950 -0.0893  11.46%
RBC  1.6% 1.7% 4.7 2.35% 0.9720 -0.0008  2.82%
HGB  1.43% 1.84% 14.10 0.76% 0.9280 0.7889  1.60%
HCT  1.35% 1.74% 42.8 4.71%  ?  ?  ?
MCV  0.7% 1.26% 91.0 0.8% 1.0810 -.5135  6.44%
PLT  4.6% 5.9% 270 1.9% 0.8745 -3.4899  13.84%
Neutrophils  8.55% 9.25% 19.69 2.0% 0.8716 0.0317 12.68%
Lymphocytes  5.1% 9.19% 1.8 2.22% 0.8099 0.0607  15.64%
Monocytes  8.9% 13.2% 0.71 4.2% 1.0208 -0.0258  1.55%
Eosinophils  10.5% 19.8% 0.06 16.7% 0.7286 0.0480  52.86%
Basophils  14.0% 15.4% 0.04 25.0% 2.0 0.0090  122.5%

It's starting to become clear there are some problems with this analyzer. If you look only at the imprecision, you find that 5 out of 11 methods are not acceptable in performance. If you were taking a measurement uncertainty approach, for instance, you might ignore bias and set the imprecision targets from Ricos as your target measurement uncertainty. In that case, you would find that only 7 of the 11 methods are acceptable in their uncertainty. But if you take bias into account, there's an even bigger problem.

Remember the bias values are determined from a comparison of patient values between the Mindray and the Pentra. So this is a comparison of two field methods, and we don't necessarily know which method is "right" or "wrong." It could be that the Mindray is a better instrument and the Pentra is a worse instrument - and all the biases are showing us how much better the Mindray is than the Pentra. However, regardless of which method is more right or wrong, these differences will manifest themselves whenever a patient is tested on both of these methods. Or, if the laboratory is changing over from Pentra to Mindray, all the patients will see these differences impact their test reports. So we can't ignore this bias. While there are some well-meaning laboratorians out there who would like to pretend bias doesn't exists, here's another example that this is dangerous assumption. Bias does exist here and it is considerable.

If we take bias and imprecision into account, then only two methods (Hemoglobin and Monocytes) are acceptable by the Ricos criteria. 9 out of 11 methods on this instrument are unacceptable.

That's the simple comparison of two separate components. What about using Sigma-metrics and allowable total error? Would that deliver the same verdict, or a different one?

Calculate Sigma metrics

Sigma-metrics takes both imprecision and bias into account in a single equation. We're going to calculate Sigma-metrics using both "Ricos goals" and the CLIA goals.

Remember the equation for Sigma metric is (TEa - bias%) / CV.

Example calculation: for WBC, with a 15.44% quality requirement, given 2.1% imprecision and 11.46% bias:

(15.44 - 11.46) / 2.1 = 3.98 / 2.1 = 1.9 Sigma

Similar to the judgement we made earlier with separate components, this Sigma-metric verdict on this WBC assay is harsh: we would consider it unacceptable.

So here's the table with all the Sigma-metrics using Ricos Goals:

Assay Ricos TEa% Level
CV% slope y-intercept Bias% Sigma-metric
WBC  15.49% 9.31 2.1% 0.8950 -0.0893  11.46% 1.9
RBC  1.6% 4.7 2.35% 0.9720 -0.0008  2.82% 0.7
HGB  1.43% 14.10 0.76% 0.9280 0.7889  1.60% 3.4
HCT  1.35% 42.8 4.71%  ?  ?  ? 0.8
MCV  0.7% 91.0 0.8% 1.0810 -.5135  6.44% negative
PLT  4.6% 270 1.9% 0.8745 -3.4899  13.84% negative
Neutrophils  8.55% 19.69 2.0% 0.8716 0.0317 12.68% 5.2
Lymphocytes  5.1% 1.8 2.22% 0.8099 0.0607  15.64% 0.9
Monocytes  8.9% 0.71 4.2% 1.0208 -0.0258  1.55% 6.2
Eosinophils  10.5% 0.06 16.7% 0.7286 0.0480  52.86% negative
Basophils  14.0% 0.04 25.0% 2.0 0.0090  122.5% negative

Now, let's be clear that there is no real "negative Sigma." Really, if you reach that point, where the bias exceeds the allowable total error, the assays are just not aiming at the same target. They are getting significantly different answers  (now the new method might still be giving consistent, precise answers, but nonetheless they are not the same answers as the comparative method). In cases where this happens, we prefer to put "n/a" for not applicable, rather than an artificial negative Sigma.

Recall that in industries outside healthcare, on the short-term scale, 3.0 Sigma is the minimum performance for routine use and 6.0 Sigma is considered world class quality. We're looking at the long-term scale for this Sigma-metric calculation, which is 1.5s higher (the short-term scale builds in a 1.5s shift, to allow for "normal process variation"). So we could go as low as 1.5 for the bare minimum acceptability. Still, what this is telling us is that this analyzer has a lot of problem assays, particularly if we compare the performance to some of the leading instruments. 

Using the Ricos allowable total error specifications, we come up with a slightly larger list of acceptable methods: Hemoglobin, Neutrophils, and Monocytes. That's similar to the first verdict, and the only reason Neutrophils is acceptable now is that its imprecision is very small relative to the allowable total error. Because its imprecision is so tight, there's some room for bias. (For those metrologists out to scour the world of bias, this is probably not an acceptable judgment.)

 Now, if we apply the CLIA criteria, we find that more methods are acceptable. The CLIA goals for RBC, Hemoglobin, Hematocrit, and Platelet count are larger than the Ricos goals. But for a majority of these hematology methods, CLIA makes no specification for acceptable performance, so we still have to use the Ricos goals. Since the CLIA PT criteria are not specified as separate components (no allowable max imprecision and no allowable max bias), we can only calculate the Sigma-metrics with CLIA TEa goals:

Assay CLIA/Ricos TEa% Level
CV% slope y-intercept Bias% Sigma-metric
WBC  15.0% 9.31 2.1% 0.8950 -0.0893  11.46% 1.7
RBC  6.0% 4.7 2.35% 0.9720 -0.0008  2.82% 1.4
HGB  7.0% 14.10 0.76% 0.9280 0.7889  1.60% 7.1
HCT  6.0% 42.8 4.71%  ?  ?  ? 1.3
MCV  0.7% 91.0 0.8% 1.0810 -.5135  6.44% negative
PLT  25% 270 1.9% 0.8745 -3.4899  13.84% 5.9
Neutrophils  23.35% 19.69 2.0% 0.8716 0.0317 12.68% 5.2
Lymphocytes  17.6% 1.8 2.22% 0.8099 0.0607  15.64% 0.9
Monocytes  27.9% 0.71 4.2% 1.0208 -0.0258  1.55% 6.2
Eosinophils  37.1% 0.06 16.7% 0.7286 0.0480  52.86% negative
Basophils  38.5% 0.04 25.0% 2.0 0.0090  122.5% negative

With the CLIA goals, which are dramatically larger for some parameters, we expand the list of acceptable tests again: Hemoglobin, Platelet count, Neutrophils, and Monocytes. Notice that these are also the same assays which we found acceptable previously. We now have 4 out of 11 tests that are acceptable, and those 4 are rated quite well by CLIA standards. However, the majority of these methods are still unacceptable by CLIA/Ricos standards. Since the CLIA goal for WBC is essentially the same as the Ricos goal, there is no large change in Sigma-metric. For hemoglobin and hematocrit, even the larger CLIA goals are not big enough to contain the variation and bias in performance.

The different metrics tell us a couple of things: that good assays often stand out whatever the analytical goals being applied, and that bad assays often are bad no matter what the goals applied. Finally, that the most stringent application of quality goals is to apply separate specifications of both imprecision and bias. In this case, applying maximum allowable imprecision and maximum allowable bias meant rejecting 9 out of 11 methods.

The question remains as to what goals are most appropriate. What is the most practical but appropriate allowable total error, imprecision, and/or bias for hemoglobin? It's clear that there are large differences between CLIA and the Ricos goals. Furthermore, the 2014 Milan conference suggested that the Ricos allowable total error specifications may in fact be too generous. An eventual correction might shrink those targets down even more.

 

Summary of Performance by Sigma-metrics Method Decision Chart and OPSpecs chart

We can make visual assessments of this performance using a Normalized Sigma-metric Method Decision Chart. First we'll look at the Ricos Goal MEDx chart:

Mindray BC 6800 Normalized Method Decision Chart (Ricos Goals)

Here we can see that the only method in the bull's eye is Monocyte, and a majority of the methods seem to be really missing the target. Several of those dots are actually "off the map" - so far off the chart that they are floating above your monitor. You can also that even when we don't know the bias (Hematocrit) the imprecision alone is beyond acceptability.

If we use the CLIA goals when they are available, the situation doesn't improve all that much:

Mindray BC 6800 hematology method decision chart (CLIA goals)

The Platelet count method performance has improved. Hemoglobin is now in the bull's eye. RGB improved performance still doesn't get it that much closer to the acceptability zones.

Now what about QC? How do we monitor and control these methods? For that, we need a Normalized OPSpecs chart:

Mindray BC 6800 NOPSpecs chart (Ricos goals)

Most of the methods are simply not controllable. That is, even with the full "Westgard Rules" we probably won't be catching errors when they first occur - it will take a while before we pick them up. We would also need to double or triple the number of controls in use for these troublesome methods, raising the expense of running this instrument. We may even need to increase the frequency of running controls because of the poor performance. Only monocytes can be controlled by single rules. Everything else needs "Westgard Rules."

Again, switching to CLIA goals for a few analytes doesn't improve the situation all that much:

Mindray BC 6800 NOPSpecs (CLIA goals)

Hemoglobin and Platelet count can be controlled with 3s limits and just two controls, but the rest of the methods still need full "Westgard Rules" and more.

Conclusion

The authors conclude "our evaluation data show that Mindray BC-6800 is a good and precise screening device and is comparable to ABX Pentra DX120."

Based on Sigma-metric analysis, we would disagree with every part of that statement. The results produced on the BC-6800 are significantly different from those produced on the Pentra. The methods have considerable imprecision. When a method is both imprecise and discrepant from other methods, we generally do not call that method good. Unless another study can show that performance is better than what was found here, this is probably not a good candidate instrument for a laboratory.