Sigma Metric Analysis

Another chance to get a second look at instrument performance. Over a year ago, we looked at cancer marker performance on the Vista. Now we examine a 2012 study that compared the method performance of the new Vista methods vs. the older Immulite methods. How do the two generations of Siemens instrumentation compare? Do those results match the other study?

• The Precision and Comparison data
• Determine quality requirements at the measured decision levels
• Calculate Sigma metrics
• Summary of Performance by Sigma-metrics chart and OPSpecs Chart
• Conclusion

JUNE 2013
Sten Westgard, MS

[Note: This QC application is an extension of the lesson From Method Validation to Six Sigma: Translating Method Performance Claims into Sigma Metrics. This article assumes that you have read that lesson first, and that you are also familiar with the concepts of QC Design, Method Validation, and Six Sigma. If you aren't, follow the link provided.]

This application looks at a paper from a 2012 issue of Clinical Biochemistry which examined the performance cancer markers on the Dimension Vista system:

Berndt Zur , Stefan Holdenrieder , Eike Albers , Gisela Walgenbach-Brünagel
& Birgit Stoffel-Wagner (2012): METHOD COMPARISON FOR CA 15-3, CA 19-9, AND CA 125
DETERMINATION USING THE NEW LOCI TECHNIQUE OF DIMENSION VISTA 1500 AND IMMULITE 2000 XPI,
Journal of Immunoassay and Immunochemistry, 33:4, 435-445

This studies were conducted at University of Bonn in Germany. The Vista uses patented LOCI (luminescent oxygen channeling immunoassay) methodology while the Immulite uses classic luminescence technology in its methodology. 

The Imprecision and Bias Data

For determination of precision,

"three serum pools according to CLSI guidelines (Clinical and Laboratory Standards Institute, formerly NCCLS; document CLSI-EP5-A) at clinically relevant decision levels.[14] These included pools at the very low value range, the median, and the highly pathological range within the reference range...For the measurements, which were carried out twice daily on 20 days in double determinations, aliquots were each time newly defrosted, vortexed, and then measured."

As for bias ( the difference between the Vista methods and the Immulite methods),

"For the method comparison, parallel measurements were performed in 738 serum samples (133 CA 15-3, 395 CA 19-9, and 210 CA 125) received by the Institute of Clinical Chemistry and Clinical Pharmacology, which operates the central laboratory of the University Hospital Bonn, Germany, as part of routine diagnostics in patients admitted and treated at the University Hospital Bonn..".

Thus, we have pretty good estimates of bias and imprecision. Furthermore, the study calculated Passing-Bablock regression slopes for different ranges of the test. Thus, for the lower end of the range, there is one regression slope and y-intercept, while for a higher or normal part of the range, there is a different regression slope and y-intercept.

So we have a lot of numbers, right? We have three levels of controls, so we have imprecision estimates for all of those levels. Using the regression equations, we can estimate the bias at each level.

Looking at the raw numbers, you may find it difficult to judge the method performance. From experience, you might be able to tell when a particular method CV is high or low. But the numbers themselves don't tell the story.

If we want an objective assessment, we can set analytical goals - specify quality requirements - and use those to calculate the Sigma-metric. 

Determine Quality Requirements

Now that we have our imprecision and bias data, we're almost ready to calculate our Sigma-metrics. But we're missing one key thing: the analytical quality requirement. How good to cancer markers need to be?

In the US, traditionally labs look first to CLIA for guidance. Next, they might look at the biologic variation database (sometimes called the Ricos goals) to see desirable specifications for total allowable error. They might also look at the German Rilibak.

 As you can see, CLIA does not provide any guidance. Remember that the PT guidelines came out in the 1990s and haven't been updated since then. Meanwhile, the biologic variation database is updated every two years. The German Rilibak goals do not exist for every analyte, but it seems like around 25% is the general goal for cancer markers.

Since the Biologic goals are the most complete and most evidence-based quality requirements, we will use those for our Sigma-metric calculations.

Calculate Sigma metrics

Now the pieces are in place. Remember the equation for Sigma metric is (TEa - bias) / CV.

Example calculation: for CA 125, at the level 210.3 U//mL, the Biologic goal is 35.4%, 3.29% imprecision, 2.59% bias:

(35.4 - 2.59) / 3.29 = 32.81 / 3.29 = 9.97

Now, for CA 19-9, at the level of 7.62 U/mL, again using 39% as the quality requirement, with 29.87% bias, 7.84% imprecision:

(39 - 29.87) / 7.84 = 9.13 / 7.84 = 1.16

Recall that in industries outside healthcare, 3.0 Sigma is the minimum performance for routine use. 6.0 Sigma and higher is considered world class performance.We'll simplify the table below and calculate all the Sigma-metrics.

Now things are starting to look clearer. We have some good news and some real questions. For most of these assays, the imprecision is fairly good, but there is a troublesome difference between the Immulite methodology and the Vista LOCI methodology. This results in a bias that exceeds the total allowable error, and thus overwhelms the Sigma-metric calculation. When we use the graphic tools, that will become apparent.

Summary of Performance by Sigma-metrics Method Decision Chart and QC Design by OPSpecs chart

If the numbers are too much to digest, we can put this into a graphic format with a Six Sigma Method Decision Chart. Here's the chart for CLIA specifications for allowable total error

Here's where the graphic display helps reveal issues with performance. The mid and high levels of CA 125 II method are in the bull's eye. If it could could eliminate or reduce bias at the very low level, the whole assay would be there. With CA 19-9, its precision is pretty good. With lower bias, the method performance might even reach the Six Sigma zone. As for CA 15-3 performance, there's just a big difference between Vista and Immulite.

Now the question becomes, what would a laboratory do if this instrument was in routine operation? If they were switching from Immulite to Vista, or had both instruments in operation in the same facility, what QC would be necessary to assure the level of quality required for use of the tests? In this case, we use the same data, but plot the methods on an OPSpecs (Operating Specifications) chart.

If we could reduce bias, we could control both CA 125 and CA 19-9 with a very simple QC procedure such as1:3s with N=3. If we could eliminate bias for CA 15-3, we could probably provide enough quality assurance with a full set of "Westgard Rules" but still just N=3.

The study authors noted the bias issue:

"we focused on comparison of assays from the Vista system with the Immulite 2000 system,
as this comparison is relevant for the method changeover taking place in our laboratory, which serves as the central laboratory of the University Hospital Bonn. Despite an acceptable to good correlation of the methods, we identified a considerable slope for CA 15-3 and CA 19-9. In fact, tumor marker values measured by the Vista method were up to 30% lower for these markers. For CA 19-9, this was particularly the case in the lower measuring range. These findings are clinically relevant, as this phenomenon is not reflected by the reference ranges, which were almost equal for CA 15-3 (according to the manufacturers: Immulite 38U/mL, Vista 35U/mL) or
not comparable for CA 19-9 (Immulite 18.4U./L 95th percentile, Vista 37U/mL 99th percentile). Interestingly, the slope was higher at higher CA 19-9 levels, while in some patient samples, the obtained results were highly discrepant. As there was no obvious difference in transaminase or creatinine values in these patients; possible clinical causes of these discrepancies will have to be investigated in future studies. An analytical cause of these discrepancies may be found in the use of different monoclonal antibodies for CA 19-9 detection possibly binding to different epitopes or displaying different affinities to the antigen. Despite a certain scatter of the values and a trend to lower values in the Vista assay in the high value ranges observed for CA 125, marker results were most comparable when measured with both methods."

We assume that the new LOCI technology is more "true" than the previous Immulite methdology. Thus, this bias is telling us how much closer to the right answer we're are moving.

Conclusion

One of the big challenges evaluating tumor markers is, How good does performance need to be? It's interesting to note that CLIA provides no guidance, but the Biologic goals provide a complete set of goals. Usually the Rilibak goals are considered broad, but in this case, they were mostly smaller than what the biologic goals specified.

Precision performance, for the most part, for these tumor markers is good. But as the study authors conclude, "systematic changes and individual discrepancies reveal an urgent need for parallel measurements in case of a method changeover in a laboratory." In other words, if patients are going to be compared across these methods, special care will be needed. Once the transfer is made over to the new Vista LOCI method, possibly a new reference range may need to be established. In that way, the bias will in a sense be eliminated, because all the expectations for patient results wlil be shifted as well.

What we can take from this analysis and the previous analysis is this: we always need more studies and more data. Just because one study shows good results doesn't mean the next study will also show that. Different comparative methods will definitely yield different results. In a system where continuous improvement is supposed to be a goal, we need to make sure we are continually assessing the quality performance of our methods.