Tools, Technologies and Training for Healthcare Laboratories

Sigma-metrics of a Sysmex Coagulation Analyzer cs5100

Around 15 Years ago, in the Basic Planning for Quality Manual, we did some analysis of coagulation tests. It's been too long since we've examined the performance of these assays. So here we take a look at a recent publication about a Sysmex analyzer.

Sigma-metrics of Sysmex Coagulation Analyzer CS5100

May 2016
Sten Westgard, MS

[June 2016: Correction the bias for one level of AT was initially listed in the table as 2.0%, when in fact it was 0.2%. The graphics were not affected, only the table. We regret the error and apologize for any confusion that it caused.]

[Note: This QC application is an extension of the lesson From Method Validation to Six Sigma: Translating Method Performance Claims into Sigma Metrics. This article assumes that you have read that lesson first, and that you are also familiar with the concepts of QC Design, Method Validation, and Six Sigma. If you aren't, follow the link provided.]

Validation of the Sysmex CS5100 coagulation analyzer and comparison to the Stago STA-R analyzer for routine coagulation parameters T. Geens, F. Vertessen, R. Malfait, K. Deiteren and M.B. Maes, International journal of laboratory hematology 37(3) · June 2015.

The Imprecision and Bias Data

"The within-day imprecision was determined by measuring commercially available lyophilized control plasma samples both in the normal and in the pathological ranges for 20 times consecutively in a single run. The between-day imprecision was evaluated by analyzing the same plasma control samples twice a day, with at least 4 h difference between both measurements, for 10 days. Each day a fresh control sample was reconstituted which was stable for at least 8 h."

Of course what matters to us is the between-day imprecision estimates.

 "Bias was calculated by comparing the mean values of the control materials with the assigned values of the manufacturer as reported in the inserts." but it was also calculated by the comparison of methods experiment: "The results of the different coagulation parameters obtained on the CS5100 system were compared with those analyzed on the STA-R Evolution instrument. For each coagulation assay, at least 60 samples, covering a large measuring range, were analyzed on both instruments. The correlation between the two instruments was evaluated with paired sample t-test (two-sided; P < 0.05), Bland and Altman difference plots, Passing-Bablok regression analysis and concordance correlation
coefficient."

We're only going to select five parameters here for Sigma-metric analysis: activated partial thromboplastin time (APTT), antithrombin (AT), and fibrinogen (FBG), and prothrombin time (PT).

Assay Target Value
CV% Bias%
APTT(s) 24.4 1.4  2.9
  49.2 1.1  0.2
AT (%) 96 2.6  3.7
  37 2.7  13.8
 FBG (mg/dL) 233 4.5  2.1
  79 3.0  2.4
PT(INR) 1.05 1.9  1
  1.73 1.6  0
 PT(s) 11.3 2.0  0
  18.5 1.5  0.5

In the absence of context, it's hard to know if this is good performance. The AT bias at a low level jumps out (13.8%). But otherwise these are just a set of numbers without any clear sense of their acceptability. (Of course, those of you who are seasoned in coagulation may already have a verdict, but let's let us clinical chemists fumble our way through it)

Determine Quality Requirements at the decision levels

Now that we have our imprecision and bias data, we're almost ready to calculate our Sigma-metrics. We're just missing one critical component: the analytical quality requirement. In this example, we're going to take advantage of several different sets of quality requirements to judge the method. One limitation is that CLIA doesn't provide specifications for all of these parameters. Another limitation is that while the "Ricos goals" may be available, they are extremely stringent goals and perhaps beyond the ability of any current method on the market.

Assay Goal source TEa% Target Value
CV% Bias%
APTT(s) Ricos 6.7% 24.4 1.4  2.9
  minimum 6.7% 49.2 1.1  0.2
AT (%) Ricos 12.5% 96 2.6  3.7
  minimum 12.5% 37 2.7  13.8
 FBG (mg/dL) CLIA 20% 233 4.5  2.1
    20% 79 3.0  2.4
PT(INR) Spanish 24% 1.05 1.9  1
  minimum 24% 1.73 1.6  0
 PT(s) Spanish 17% 11.3 2.0  0
  minimum 17% 18.5 1.5  0.5

 

Calculate Sigma metrics

Sigma-metrics takes both imprecision and bias into account in a single equation. We're going to calculate Sigma-metrics using both "Ricos goals" and the CLIA goals.

Remember the equation for Sigma metric is (TEa - bias%) / CV.

Example calculation: for APTT, with a 6.7% quality requirement, given 1.4% imprecision and 2.9% bias:

(6.7 - 2.9) / 1.4 = 3.8 / 1.4 = 2.7 Sigma

The Sigma-metric verdict on this APTT assay is not good. But this is only one level, and maybe not perhaps the most critical level.

So here's the table with all the Sigma-metrics using the mixed set of Goals:

 

Assay Goal source TEa% Target Value
CV% Bias% Sigma-metric
APTT(s) Ricos 6.7% 24.4 1.4  2.9  2.7
  minimum 6.7% 49.2 1.1  0.2  5.9
AT (%) Ricos 12.5% 96 2.6  3.7  3.4
  minimum 12.5% 37 2.7  13.8  <0
 FBG (mg/dL) CLIA 20% 233 4.5  2.1  4.0
    20% 79 3.0  2.4  5.9
PT(INR) Spanish 24% 1.05 1.9  1  >6
  minimum 24% 1.73 1.6  0  >6
 PT(s) Spanish 17% 11.3 2.0  0  >6
  minimum 17% 18.5 1.5  0.5  >6

Overall, there are a lot of good metrics here. A few troubling numbers, but not too many of them.

Summary of Performance by Sigma-metrics Method Decision Chart and OPSpecs chart

We can make visual assessments of this performance using a Normalized Sigma-metric Method Decision Chart. First we'll look at the Normalized MEDx chart:

2016 Sysmex Coag CS5100 NMEDX

Now what about QC? How do we monitor and control these methods? For that, we need a Normalized OPSpecs chart:

2016 Sysmex Coag CS5100 NOPSpecs

Several of the methods are easily controllable with 2 controls and 3s control limits PT(s) and PT(INR). FBG probably needs just a small set of "Westgard Rules" and it can be controlled. A few of the points are not great. One of the levels for AT is pretty high; we need to consider which level has the most important clinical decisions. In one case, "Westgard Rules" are needed. In another case, we should be trying to make major improvements in performance. For APTT, one level is in the bull's-eye but the the other is not that great - again the critical decision level will have a big impact on what we decide to do for QC with this method.

Conclusion

The authors concluded "This evaluation confirmed that the CS5100 system is a suitable instrument for daily routine determination of the coagulation parameters APTT, PT, FBG... and AT. All parameters fulfill the criteria for imprecision, bias, and total error and a comparable or improved imprecision was observed compared with the STA-R Evolution."

Based on Sigma-metric analysis, we would agree. Many assays are world class. A few are in need of improvement, but only on one level.