Sigma Metric Analysis

What happens when a hospital has multiple analyzers, "closed systems", "open systems" and "mixed systems"? When we use different analyzers, reagents, and calibrators, can we assume the results will always be comparable? NO. No, we can not.

Sigma-metrics comparison of a Vitros 5600, Hitachi 7170, and a Roche Cobas 8000

• The Precision and Comparison data
• Determine quality requirements at the critical decision level
• Calculate Sigma metrics
• Summary of Performance by Sigma-metrics chart
• Conclusion

January 2017
Sten Westgard, MS

[Note: This QC application is an extension of the lesson From Method Validation to Six Sigma: Translating Method Performance Claims into Sigma Metrics. This article assumes that you have read that lesson first, and that you are also familiar with the concepts of QC Design, Method Validation, and Six Sigma. If you aren't, follow the link provided.]

Clinical Laboratory published a Chinese study from Tianjin Medical University. The Chinese Laboratory Marketplace is unique because not only are the typical "closed sytems" of the major manufacturers present, but there are "more than 100 diagnostic reagent kit suppliers. In an effort to reduce testing costs many laboratories chose 'open system' or 'mixed system' which means that the reagent and analyzer are not matched....[S]ome laboratories use other calibrators rather than using the matching calibrators. Thus, results can have large variations between different diagnostics systems which cause big troubles for traceability and impairs the quality and reliability of our services."

Methods Comparison and Bias Estimation of Three Distinct Biochemistry Analytical Systems in One Clinical Laboratory Usign Patient Samples  Fu, H, Yan, S, Wang Y, Gu D, Li X, Yang G, Liu Y, Clin Lab 2016;62:187-194.

In this study, two different instruments were compared against a third instrument, which served as a de facto local reference method. The "reference" instrument was the Roche Cobas 8000. The two other instruments were the Vitros 5600 and the Hitachi 7170. For the Vitros, the reagents kits, calibrators, and QC materials were from the manufacturer. For the Hitachi, reagent kits and calibrators for the lipids were from Leademan (Beijing, China), while the lipid reagents were obtained from Sekisui and the calibrators from c. fas Lipids (Roche, Germany). Beckman LX20 was used for the reagents and calibrators for the electrolytes.

The Imprecision and Bias Data

"Commercial control samples were used for determination of between-day imprecision." The table lists the number of days as N=30.

A comparative study was performed using actual patient samples. "All 40 samples were run in duplicate by three diagnostic systems. Aliquots of each sample were analyzed sequentially in the first run and then reversed. Results of measurements were compared following CLSI guideline EP9-A2. The correlation and bias of each analyte between the reference system and test systems were evaluated and calculated by linear regression and Bland and Altman plot."

In other words, this study did a very good job of collecting both imprecision and bias data. The study provided different amounts of data for each instrument. We have more analytes with both imprecision and bias data on the Hitachi 7170 than we do for the Vitros 5600.

Remember we are using the regression equation to calculate the bias at the level where the imprecision was measured.

For example, with the Vitros 5600, at a level of 2.5, the regression equation is 0.988 and the y-intercept is - 0.95.

Thus the Vitros value = (0.988 * Cobas 8000 value) - 0.95

Vitros value = (0.988 * 2.5) - 0.95

    = 2.47 - 0.95

    = 1.52

The bias is therefore (1.52 - 2.5 ) / 2.5

    = -0.98 / 2.5

    = -39.2% bias for Albumin on the Vitros 5600. Representing a significant difference between the Cobas 8000 and the Vitros 5600 results.

[We will use the absolute value of the bias (that is, 39.2%) when we calculate Sigma-metrics.]

Yes, that is a whole lot of numbers! What do they all mean? In the absence of context, it's often hard to know. There are a few analytes where bias does seem rather high (Albumin, ALT, AST on both the Vitros 5600 and the Hitachi 7170). But we'll have to see whether the context of the allowable error is such that it allows that bias.

Determine Quality Requirements at the decision levels

Now that we have our imprecision and bias data, we're almost ready to calculate our Sigma-metrics. We're just missing one critical component: the analytical quality requirement. In this study, the majoritory of the analytes were covered by CLIA analytical performance specifications. While the study focused just on bias, since they included the imprecision data, we can determine the Sigma-metrics.

Note that the majority of the performance specifications we use here are CLIA goals, but where CLIA does not provide a goal, we have opted to select the desirable specifications for allowable total error from the biologic variation database ("Ricos goals"). Also note in cases where the CLIA allowable total error is expressed in units, that means that the % goal is different at each level - so those goals are listed in each row of the table, where necessary.

Now we have all the numbers we need for Sigma-metrics.

Calculate Sigma metrics

Sigma-metrics takes both imprecision and bias into account in a single equation. We're going to calculate Sigma-metrics using both "Ricos goals" and the CLIA goals.

Remember the equation for Sigma metric is (TE_a - bias%) / CV.

Example calculation: for the Vitros 5600, Albumin has a 10% quality requirement, given 7.4% imprecision and 17.15% bias:

(10 - 39.2) / 2.45 = -29.2 / 2.45 = negative

Once we get below zero, there is not a useful value to the Sigma-metric equation. We simply know that the verdict on this Albumin assay is harsh. Results are going to be significantly different between the two instruments.

Let's try to use the Sigma-metric equation in a slightly better case. The Hitachi 7170 Albumin. Again the TEa is 10%, but this time, bias is 16.1% and the imprecision is 3%.

(10 - 16.1) / 3 = -6.1 / 3 = negative

Whoops. Okay, the Hitachi value is also very different than the Cobas 8000 value. This is disconcerting, to think that there are three instruments in this health care system, and they are all going to give significantly different results.

Let's try to find an example where there is some good news. Take Alkaline Phosphatase on the Vitros 5600, where TEa is 30%, bias is 16.33% and imprecision is 1.74%.

(30 - 16.33) / 1.74 = 13.67 / 1.74 = 7.86  = >6 Sigma

So here we have an assay that is better than Six Sigma. In practical terms, it's not significant that there is a 7 Sigma or 7.8 Sigma. This assay has simply achieved the Six Sigma performance.

So here's the table with all the Sigma-metrics using the (again mostly CLIA) analytical performance goals:

Now here's the Hiatchi 7170

Just to re-emphasize two points: any Sigma-metric above 6 is simply treated the same way as Six Sigma.  Similarly, the Sigma-metrics that are "below zero" are not necessarily worse - but they do indicate that the bias is significant. Any time the bias exceeds the allowable total error, you will find a Sigma-metric of zero. This essentially means that the difference between the comparative method and the new method is so large that they are effectively aiming at different targets. The next question is now: which method is correct? A more important question comes up when all the different methods are in the same healthcare system: how does a clinician interpret results for a single patient if they've have their blood tested on these different instruments?

Overall, there are a lot of troubling metrics here, particularly for the Vitros 5600. The Hitachi 7170 has several assays performing at world class, but a few that are also in low Sigma or negative territory.

Summary of Performance by Sigma-metrics Method Decision Chart and OPSpecs chart

We can make visual assessments of this performance using a Normalized Sigma-metric Method Decision Chart:

It's very clear that the Hitachi 7170 is more comparable with the Cobas 8000 than the Vitros 5600.

Now what about QC? How do we monitor and control these methods? For that, we would usually need a Normalized OPSpecs chart. However, because the differences between methods are so significant, we should probably stop and consider our options.

For many of these Vitro methods, we need "maximum QC": a lot of "Westgard Rules", a lot of controls, and probably we should consider more frequent QC and maybe even watching average of normals. Very few assays are in the bull's-eye here, which means this will take more time, effort, resources, and money to operate this instrument. Most critically, we need some way to reconcile the significant differences between the instrument results.

Compared to the Vitros 5600, the Hitachi 7170 looks much better. Many assays in the bull's-eye, many assays that are going to be easily controlled with 2.5s, 3s, or even 3.5s control limits and just 2 controls per run. This is an opportunity to make some efficiency gains. Now we still have three methods in the red - these need "maximum QC", so it's not as if we're free and clear here.

The authors also commented on these significant biases: "[C]onsidering the significant unparallel results between dry and wet chemical analytical systems, reference intervals should be validated and transferred from Cobas 8000 to Vitros 5600 based on CLSI documents C28-A3.... Furthermore, users should clearly define the application fields to avoid simultaneously use [sic.] of both dry and wet analytical systems for the same analyte at the same site."

These suggested practices are both difficult and problematic. Remember these biases were determined on patient samples. This is REAL BIAS. So these differences are not just something that is showing up in the controls but not impacting patients. These biases are real and if clinicians are getting results from multiple instruments, those differences are present and will affect the interpretation of patient status.

Conclusion

The authors stated "Significant differences for several analytes between distinct analyzers were found, especially between dry chemical analyzer and conventional wet chemical analyzer, whose most relative predicted bias can reach 30% [actually more!], which is worthy of our concern. When one analyte was detected on more than one device, a strict method comparison study should be performed regularly." 

Based on Sigma-metric analysis, we strongly agree with these conclusions. It looks like comparability is not achievable between the wet and the dry. In our era of "standardization and harmonization", this is a stark outlier. The only way to get standardized and harmonized, it appears, would be to get rid of one of the discrepant instruments. And again, for the metrology crowd that believes bias should not be allowed to exist, there is a major reality check here. Bias exists and exists significantly in this hospital and laboratory. In order to express MU for our tests, we need to eliminate bias - does this mean that the metrologists want to eliminate the dry chemistry analyzers from this laboratory, or even the entire field of laboratory medicine?