Sigma Metric Analysis

Are there other choices than the big 5 diagnostic analyzers? Beyond Beckman, After Abbott, Roche, Siemens, and Vista, are there viable alternatives for chemisty analyzers? We take a look at a new automated clinical chemistry analyzer, evaluating both its "closed" and "open" reagent performance.

Sigma-metric Analysis of JEOL BioMajesty JCA/BM6010-/C Automated Clinical Chemistry Analyzer

Sten Westgard, MS
April 2018

• The Precision, Comparison and Sigma-metric data
• Summary of Performance according to CLIA and Ricos Goals by Sigma-metrics Normalized Method Decision charts
• QC Implications of Sysmex XN Performance by Normalized OPSpecs chart
• Conclusion

[Note: This QC application is an extension of the lesson From Method Validation to Six Sigma: Translating Method Performance Claims into Sigma Metrics. This article assumes that you have read that lesson first, and that you are also familiar with the concepts of QC Design, Method Validation, and Six Sigma metrics. If you aren't, follow the link provided.]

Are there only 5 choices when it comes to chemistry analyzers? Or are there other instruments entering the marketplace we should be considering? A recent paper gives us a chance to contemplate a new alternative.

Performance Evaluation of the JEOL BioMajesty JCA-BM6010/C Automated Clinical Chemistry Analyzer.Hyeong Nyeon Kim, Misuk Ji, Hee-Won Moon, Mina Hur, Yeo-Minn Yun, Lab Med Online Vol. 7, No3: 111-119, July 2017

The additional component of interest in this study is that they examined not only the closed reagents made by the manufacturer, but also the use of third-party reagents ("open" reagents). So we get a chance to see two different perspectives on performance on this instrument.

The Imprecision, Bias and Sigma-metric Data

"For evaluation of precision, we tests all the analytes in Liquid Assayed Multiqual (human serum based) chemistry control Level 1 and Level 2 (Bio-Rad Laboratories, Inc. Hercules, CA, USA) according to the Clinical and Laboratory Standards Institute (CLSI) guideline EP5-A2....After preliminary evaluation for five consecutive days, the two levels were tested in duplicate, with two runs per day over 20 days. Repeatability, between-run, between-day, and total precision were evaluated for all available analytes...."

"Method comparison was performed following CLSI guideline EP9-A2-IR....Specimens with variable measurement ranges were collected using leftover serum samples. In the first comparison of the 11 analytes, a set of 50 serum samples was tests using JCA-BM6010/C and the Roche-Hitachi Cobas 8000 c702 chemistry autoanalyzer (Cobas 8000, Roche diagnostics, Basel, Switzerland) in duplicate, using the reagenets provided by the respective manufacturers (JEOL vs Roche). In the second comparison, a set of 50 serum samples was tested for nine analytes (ALP, ALT, AST, calcium, creatinine, GGT, glucose, LDH and uric acid) in duplicate using teh JCA-BM6010/C analyzer with JEOL closed reagenets and other open reagents. The manufacturers of teh open reagenets were Sekisui (AST, ALT, GGT, calcium, and uric acid), Denka Seiken (Gluocose, LDH) and Roche (ALP, creatinine).... Each value was compared using Deming regression analysis and Blant-Altman plots."

Again, the design of the study is admirable, adhering to all the appropriate CLSI protocols. We can have confidence in the data produced, even if we're still not sure of the conclusion.

Two levels of performance were recorded. Bias was calculated using the Deming regression equations

 What may immediately jump out of this see of numbers are the regression slope and biases for Alkaline Phosphatase and the high biases for AST,

 Now that the instrument is using third-party reagents, the biases change, but some aren't any better than the "closed" system reagents. That alkaline phosphatase still has a huge regression slope and bias. LDH also has some significantly large biases, due to a very low regression slope.  The BM6010/C just isn't comparable to the Roche Cobas 8000 on these methods.

Nevertheless, what do all these numbers mean? In the absence of context, it's hard to know.

So let's calculate the Sigma-metrics.

Sigma-metric calculations for the BM6010/C

Remember the equation for Sigma metric is (TEa - bias) / CV:

For a 30% quality requirement, for Alkaline Phosphatase on the low level of BM6010/C, the equation is (30-243.4) / 1.5 = very much less than zero

For a 30% quality requirement, for Alkaline Phosphatase on the high level of BM6010/C, the equation is (30 - 240.5) / 2.4 = very much less than zero

Bluntly, this alk phos method is going to generate very different numbers than the Cobas 8000. It's either going to confuse a clinician or prompt them toward the wrong diagnosis.

The metrics are displayed along the right columns. (The last two columns give us an idea of where we need to graph the points on our Normalized Method Decision Charts.)

 So there's some good news, some great news, and some really, really bad news in this collection of metrics. Many of these analytes are good performers. But some are just outright terrible and we should investigate the study, the method, the reagent, and think seriously about using that parameter if we do decide to use the instrument.

Is it any better when the instrument is using third-party reagent?

 This time there are more Negative Sigma-metrics. Any time this happens is unusual, reflecting a situation when the bias completely exceeds the allowable total error.

Summary of Performance by Sigma-metrics Method Decision Chart using CLIA and Ricos Goals

We can make visual assessments of this performance using a Normalized Sigma-metric Method Decision Chart:

With the "open" reagents, there is an encouraging cluster of bull's-eyes, but the the few analytes with extreme bias are "off the charts".

Since there are more analytes involved in the closed reagents, there are more bull's-eyes but also more extremely biased analytes that are off the charts.

Summary of QC Design by Normalized OPSpecs chart - using CLIA and Ricos Goals

The benefit of the Sigma-metric approach is that labs can do more than assess their quality, they can act on it. Using OPSpecs charts, they can actually optimize their QC procedures for each test.

For the open reagents, all the green dots represent performance where control limits can be widened to 3.5 SD and only 2 controls are necessary. However, for the "off the charts" assays, all the "Westgard Rules" are going to be necessary, and many more control measurements should be made, perhaps 6 or even 9 per run.  QC frequency should be much higher for those analytes, too. Not once a day, but many times per day - unless that bias can somehow be fixed. If this large bias is resolved, the QC rules needed will be pretty simple. But if the bias is real, then either a recalibration needs to occur, or a re-education of the clinician to a new reference interval, or a kind of "fudge factor" will have to be applied when reporting out. None of these options is desirable.

For the closed reagents, you can see there is more bias present in most of the assays, and there are more troublesome assays that need serious attention.

Conclusion

The authors stated "When the analysis was conducted using JEOL closed reagents, the results of JCA-BM6010/C strongly correlated with those of the Cobas 8000 as well as with those obtained using other open reagents.... In addition, JCA-BM6010/C showed acceptable mean difference by excluding analytes showing notable differences. It is suspected that the reason for this discrepancy was the difference in teh analytical method used for each reagent by the Roche and JEOL systems (ALP, IFCC/JSCC; calcium, OCPC/Phosphonazo III; creatinine, Jaffe/Enzymatic). JCA-BM6010/C yielded slightly higher values than the Cobas 8000 for GGT. In the second comparison using the open reagents, JCA-BM6010/C demonstrated an acceptable mean difference by excluding analytes showing notable differences; these could be attributed to the difference in analytical meethod for each open reagent (ALP, Roche, IFCC; calcium, Sekisui, Arsenazo III; creatinine, Roche, Jaffe; LDH, Denka Seiken, P-L)."

It's an interesting phrase, to find that all methods are acceptable, when you exclude the ones that aren't. If these method problems were so significant, probably a better comparative method should have been found. However, given the practical reality that there are a lot of Roche Cobas 8000 instruments out there, it's very significant to find that this new analyzer doesn't compare well with those results. That's going to be a problem that can't be overcome just by shrugging your shoulders - this is something that will need to be addressed by any lab that uses it.

The authors conclude, "JCA-BM6010/C showed excellent performance in terms of precision, linearity, correlation, accuracy, and sample carryover according to CLSI guidelines. JCA-BM6010/C is an ultra-compact analyzer with a wide AMR and shows comparable performance with that of other instruments. Therefore, JCA-BM6010/C is suitable for small laboratories, especially those with a small staff. We conclude that JCA-BM6010/C would be a useful routine chemistry analyzer in medical laboratories."

We do not reach the same conclusion. Despite the many assays that show Six Sigma performance on this instrument, the significant biases found on a few of these analytes are such that we would disagree. This analyzer needs improvements before all of its assays could be considered comparable and acceptable. If used in a laboratory, several analytes should be turned off.