Tools, Technologies and Training for Healthcare Laboratories

Sigma metric Analysis of a Sysmex XN

A recent study compared two different generations of the same hematology analyzer, Sysmex XN and Sysmex XE.  We would expect there to be a good comparability of results between the analyzers. We would expect that...  [updated with CLIA 2024 and EFLM min goals]

Sigma-metric Analysis of Sysmex XN in comparison to a Sysmex XE

Sten Westgard, MS
February 2018
September 2023, updated with 2023 EFLM minimum and desirable goals, 2024 CLIA goals

[Note: This QC application is an extension of the lesson From Method Validation to Six Sigma: Translating Method Performance Claims into Sigma Metrics. This article assumes that you have read that lesson first, and that you are also familiar with the concepts of QC Design, Method Validation, and Six Sigma metrics. If you aren't, follow the link provided.]

We recently made a correction on a paper and analysis involving a Sysmex instrument. That made us curious to seek out other papers on new Sysmex instruments:

Multicenter verification of the Sysmex XN-Series. Schoorl M1, Schoorl M1, Chevallier M1, van der Ploeg T2, van Pelt J1. Int J Lab Hematol. 2017 Oct;39(5):489-496. doi: 10.1111/ijlh.12674. Epub 2017 May 18.

This study, submitted in 2016 and published in 2017, looked at 7 Sysmex XN modules across 3 laboratories, comparing the results to the Sysmex XE-2100. So this is the comparison of two instrument lines within the same company. We would expect a great deal of consistency, right?

The Imprecision, Bias and Sigma-metric Data

"Intra-assayreproducibility was determined for white blood cell count (WBC), red blood cell count (RBC), platelets (PLT), hemoglobin (Hb), hematocrit (Ht), mean cell volume (MCV), mean cell hemoglobin (MCH), mean cell hemoglobin concentration (MCHC), neutrophils (Neut), lymphocytes (Lymph),  monocytes (Mono), eosinophils (Eo), basophils (Baso), immature granulocytes (IG), red blood cell distribution width (RDW), and reticulocytes (Reti) on each Sysmex XN hematology module. Three samples for each low, normal, and high value were tested 10 times consecutively within one run....

For the comparative study, a set of 160 patient samples was used. Complete blood cell counts and leukocyte differential were determined on all seven XN hematology modules.....[A]ll measurements were repeated on the Sysmex XN-9000 module A3 (first module) to gain insight into the effect of multiple shakings, reduction of sample volume, and sample stability. Results of each XN analyzer were compared with the results of the routinely used Sysmex XE-2100 hematology analyzer, which was subjected to intensive internal and external quality controls for many years. The performance of the Sysmex XE-2100 analyzer is well known and completely in line with the consensus values from the Dutch Foundation for Quality Assessment in Medical Laboratories (SKML)."

Three levels of performance were recorded. Bias was calculated using the regression equations from the

TEST control level TEa Source TEa slope y-int % Bias CV
WBC 1.5 CLIA 15 1.035 0.95 66.83 2.80
  8.5 CLIA 15 1.035 0.95 14.68 1.30
  60 CLIA 15 1.035 0.95 5.08 0.80
RBC 1.6 CLIA 6 1.039 -0.9 52.35 0.80
  3.5 CLIA 6 1.039 -0.9 21.81 0.60
  5.5 CLIA 6 1.039 -0.9 12.46 0.50
Hgb 3.5 CLIA 7 0.983 0.135 2.16 0.60
  8.5 CLIA 7 0.983 0.135 0.11 0.60
  11 CLIA 7 0.983 0.135 0.47 0.50
Hematocrit 0.2 CLIA 6 1.04 -0.004 2.00 0.70
  0.4 CLIA 6 1.04 -0.004 3.00 0.60
  0.55 CLIA 6 1.04 -0.004 3.27 0.50
MCV 65 Ricos 2.4 0.983 3.266 3.32 0.70
  90 Ricos 2.4 0.983 3.266 1.93 0.60
  120 Ricos 2.4 0.983 3.266 1.02 0.50
MCH 1550 Ricos 2.5 0.984 5.376 1.25 0.80
  1950 Ricos 2.5 0.984 5.376 1.32 1.00
  2400 Ricos 2.5 0.984 5.376 1.38 0.80
MCHC 18 Ricos 2.2 0.835 2 5.39 1.10
  20 Ricos 2.2 0.835 2 6.50 1.00
  23 Ricos 2.2 0.835 2 7.80 0.80
PLT-I 100 Ricos 13.4 1.03 -2.893 0.11 3.90
  300 Ricos 13.4 1.03 -2.893 2.04 1.80
  1000 Ricos 13.4 1.03 -2.893 2.71 1.00
Neutrophils 0.5 Ricos 23.5 1.037 0.088 21.30 5.20
  5.5 Ricos 23.5 1.037 0.088 5.30 1.70
  25 Ricos 23.5 1.037 0.088 4.05 1.40
Lymphocytes 1 Ricos 17.6 1.03 -0.41 38.00 4.00
  3 Ricos 17.6 1.03 -0.41 10.67 3.30
  10 Ricos 17.6 1.03 -0.41 1.10 4.00
Monocytes 0.15 Ricos 27.9 0.99 0.051 33.00 9.00
  1.3 Ricos 27.9 0.99 0.051 2.92 5.20
  7 Ricos 27.9 0.99 0.051 0.27 4.30
Eosinophils 0.1 Ricos 37 1.028 0.003 5.80 17.70
  0.5 Ricos 37 1.028 0.003 3.40 5.90
  1.5 Ricos 37 1.028 0.003 3.00 5.50
Basophils 0.1 Ricos 39 1.261 0.015 11.10 12.20
  0.5 Ricos 39 1.261 0.015 23.10 7.60
  1.5 Ricos 39 1.261 0.015 25.10 0.50

 Yes, that is a whole lot of numbers!

Nevertheless, what do all these numbers mean? In the absence of context, it's hard to know.

So let's calculate the Sigma-metrics.

Sigma-metric calculations for the XNs

Remember the equation for Sigma metric is (TEa - bias) / CV:

For a 15% quality requirement, for WBC on the low level of the Sysmex XN, the equation is (15 - 66.83) / 2.8 = less than zero

For a 15% quality requirement, for WBC on the middle level of the Sysmex XN, the equation is (15 - 14.68) / 1.3 = 0.2

The metrics are displayed along the right columns.

TEST control level TEa Source TEa slope y-int % Bias CV Sigma
WBC 1.5 CLIA 15 1.035 0.95 66.83 2.80 negative
  8.5 CLIA 15 1.035 0.95 14.68 1.30 0.2
  60 CLIA 15 1.035 0.95 5.08 0.80 12.4
RBC 1.6 CLIA 6 1.039 -0.9 52.35 0.80 negative
  3.5 CLIA 6 1.039 -0.9 21.81 0.60 negative
  5.5 CLIA 6 1.039 -0.9 12.46 0.50 negative
Hgb 3.5 CLIA 7 0.983 0.135 2.16 0.60 8.1
  8.5 CLIA 7 0.983 0.135 0.11 0.60 11.5
  11 CLIA 7 0.983 0.135 0.47 0.50 13.1
Hematocrit 0.2 CLIA 6 1.04 -0.004 2.00 0.70 5.7
  0.4 CLIA 6 1.04 -0.004 3.00 0.60 5.0
  0.55 CLIA 6 1.04 -0.004 3.27 0.50 5.5
MCV 65 Ricos 2.4 0.983 3.266 3.32 0.70 negative
  90 Ricos 2.4 0.983 3.266 1.93 0.60 0.8
  120 Ricos 2.4 0.983 3.266 1.02 0.50 2.8
MCH 1550 Ricos 2.5 0.984 5.376 1.25 0.80 1.6
  1950 Ricos 2.5 0.984 5.376 1.32 1.00 1.2
  2400 Ricos 2.5 0.984 5.376 1.38 0.80 1.4
MCHC 18 Ricos 2.2 0.835 2 5.39 1.10 negative
  20 Ricos 2.2 0.835 2 6.50 1.00 negative
  23 Ricos 2.2 0.835 2 7.80 0.80 negative
PLT-I 100 Ricos 13.4 1.03 -2.893 0.11 3.90 3.4
  300 Ricos 13.4 1.03 -2.893 2.04 1.80 6.3
  1000 Ricos 13.4 1.03 -2.893 2.71 1.00 10.7
Neutrophils 0.5 Ricos 23.5 1.037 0.088 21.30 5.20 0.4
  5.5 Ricos 23.5 1.037 0.088 5.30 1.70 10.7
  25 Ricos 23.5 1.037 0.088 4.05 1.40 13.9
Lymphocytes 1 Ricos 17.6 1.03 -0.41 38.00 4.00 negative
  3 Ricos 17.6 1.03 -0.41 10.67 3.30 2.1
  10 Ricos 17.6 1.03 -0.41 1.10 4.00 4.1
Monocytes 0.15 Ricos 27.9 0.99 0.051 33.00 9.00 negative
  1.3 Ricos 27.9 0.99 0.051 2.92 5.20 4.8
  7 Ricos 27.9 0.99 0.051 0.27 4.30 6.4
Eosinophils 0.1 Ricos 37 1.028 0.003 5.80 17.70 1.8
  0.5 Ricos 37 1.028 0.003 3.40 5.90 5.7
  1.5 Ricos 37 1.028 0.003 3.00 5.50 6.2
Basophils 0.1 Ricos 39 1.261 0.015 11.10 12.20 2.2
  0.5 Ricos 39 1.261 0.015 23.10 7.60 2.0
  1.5 Ricos 39 1.261 0.015 25.10 0.50 26.8

 Yes, there are more than a few Negative Sigma-metrics. This is unusual, reflecting a situation when the bias completely exceeds the allowable total error. Considering the comparison, XN to XE, it's also surprising. The comparability of instruments within the same company is often assumed to be excellent. Here, a significant difference exists. Given that the authors state that the XE was rigorously tested by IQC and EQA, including the SKML EQA, this would point toward the XE being the "right" answer, while the new XN is generating something not quite correct.

The authors have a unique response to this issue:

"The method correlation was determined with a set of 160 human samples measured within 4 hours after collection on all seven XN modules and on a XE-2100 analyzer. Statistical analysis with ordinary linear regression showed excellent correlation coefficients, but the criteria for the slope and intercept, being that the 95% CI should include 1.0 and 0.0, respectively, were met by only two parameters [emphasis mine]. Obviously if the methods are precise and the 95% confidence intervals of the slope and intercept are very small, there is a reasonable chance that the equation y=x is not met, although the comparison is very good. Therefore, we used Bland-Altman
difference analysis as an alternative approach.

"Using Bland-Altman analysis with criteria for acceptance dependent on the reproducibility of a specific parameter (3× CV%) turned out to be a straightforward method for the verification of multiple Sysmex XN hematology modules. At least 90% of the measurements should fulfill the criterion for approval [please note: a 10% defect rate is the equivalent of a 2.7 or 2.8 Sigma-metric].... The measurements outside the limits of agreement are easily detected, while the linear regression lines...do not reveal this information so clearly."

Summary of Performance by Sigma-metrics Method Decision Chart using CLIA and Ricos Goals

We can make visual assessments of this performance using a Normalized Sigma-metric Method Decision Chart:

2018 Sysmex XN Series 2016study NMEDX

Overall, you can see that most of the x-coordinates (or imprecision) is on the left side of the graph, which is a good indicator. The problems are occurring at the bias level. Any points that you see tucked at the top edge of the graph are "off the charts" because of the significant bias which exceed the allowable total error.

2018 Sysmex XN Series 2017study Diffs NMEDX

 Again, a non-trivial number of data points are off the chart. The differentials are also scattered all over the place, with great precision at some levels, and then high imprecision at others.

Summary of QC Design by Normalized OPSpecs chart - using CLIA and Ricos Goals

The benefit of the Sigma-metric approach is that labs can do more than assess their quality, they can act on it. Using OPSpecs charts, they can actually optimize their QC procedures for each test.

2018 Sysmex XN Series 2016study main NOPSPECS

For the main hematology parameters, if bias is resolved, QC rules needed will be pretty simple. But if the bias is real, then either a recalibration needs to occur, or a re-education of the clinician to a new reference interval, or a kind of "fudge factor" will have to be applied when reporting out. None of these options is desirable.

2018 Sysmex XN Series 2016study diffs NOPSPECS

For the differentials, there is both better and worse news. More of the performance is "in the bull's eye" and can use very simple wide single control limits. But there are also more "off the chart" levels and levels of performance that will demand a maximum implementation of "Westgard Rules" with 3, 6 or more control measurements per run. 

 

EFLM biological database desirable performance specifications

2023 Sysmex XN recalc EFLM des NMEDX

EFLM biological database minimum performance specifications

2023 Sysmex XN recalc EFLM min NMEDX

CLIA 2024 performance specifications

2023 Sysmex XN recalc CLIA 24 NMEDX

Conclusion

The authors stated "In conclusion, seven Sysmex XN hematology modules, distributed over three laboratories, were part of a simultaneous verification procedure.  The results of the 14 different parameters by measuring 160 human samples were compared with the results of the Sysmex XE-2100 hematology analyzer whose performance has been extensively known from years of internal and external quality controls. Most parameters on all XN analyzers could be approved, but some XN modules needed readjustment for several parameters. The results of our multicenter evaluation are in line with those published earlier in stand-alone settings. We recommend the use of Bland-Altman difference analysis as the use of linear regression analyses was not successful in the case of precise measurements."

These conclusions demonstrate the practical challenges of managing instruments with less than world class quality. When the goal is a 10% error rate, even a below 3 Sigma level method can be considered acceptable. Given the volume of patient testing, we should be trying to get more than 9 out of 10 results correct.

Some of this difference may indeed be an improvement of XN over XE performance. In this case, though, even success is a challenge, because the XN results, while being more correct, will also be significantly different than results from the XE results. You would not want clinicians interpreting these changes in instruments as a change in patient health. Nor would you want to use both of XE and XN in the same healthcare system. By this study, it appears they are not comparable, nor are their results interchangeable.

[2023 update] When we benchmark this instrument against the most recent goals, the judgment is even harsher. The difference between the XE and the XN is quite large. If most of the XE's are off the market now, then we can cross that worry off our minds. Finding another more relevant comparison method is our next challenge. Nevertheless, it's not just bias that degrades the metrics here. The EFLM desirable specifications are out of reach, and even the minimum specifications are no easy hurdle to clear. The new CLIA 2024 goals look to be more forgiving to the instrument.