Tools, Technologies and Training for Healthcare Laboratories

Cancer markers on the Vista

A 2011 study performed analysis on the performance of cancer markers on the Siemens Vista. We use the data and calculate Sigma-metrics. How good do cancer marker assays need to be? Are new methods meeting those goals?

APRIL 2012
Sten Westgard, MS

[Note: This QC application is an extension of the lesson From Method Validation to Six Sigma: Translating Method Performance Claims into Sigma Metrics. This article assumes that you have read that lesson first, and that you are also familiar with the concepts of QC Design, Method Validation, and Six Sigma. If you aren't, follow the link provided.]

This application looks at a paper from a 2011 issue of Clinical Biochemistry which examined the performance cancer markers on the Dimension Vista system:

Analytical performance and clinical concordance of the cancer biomarkers CA 15-3, CA 19-9, CA 125 II, Carcinoembryonic Antigen, and Alpha-Fetoprotein on the Dimension Vista System, Robert H. Christenson, Denise R. Cervelli, Jamie Sterner, Lauren M. Bachmann, Heather Rebuck, Jeffery Gray, Walter E. Kelly, Clinical Biochemistry 44(2011) 1128-1136..

This studies were conducted at University of Maryland School of Medicine (Baltimore MD), Virginia Commonwealth University (Richmond, VA) and at the labs of Siemens Healthcare Diagnostics (Glasgow, DE). The controls used were from Bio-Rad.

The Imprecision and Bias Data

The imprecision data used in the study was collected according to CLSI EP5-A2. Three levels of control were run in duplicate, twice daily for 20 days (for a total N of 80). Total imprecision was estimated (see Table 1, labelled as Within-lab reproducibility). Thus, we have estimates of imprecision at three different levels for each of the assays. An additional imprecision estimate was made from a serum pool. However, we will focus on the imprecision measured by the controls.

As for bias,

"Measurements using serum samples for all five tumor markets were compared to an established FDA-cleared method and analyzed by Passing-Bablok regression analysis, residual plots and least squares linear regression analysis. CA 15-3, CA 19-9 and CA 125 II measurements on the Vista were compared to the ADVIA Centaur... and the CEA assay was compared to the Beckman Access.... the AFP method was compared to the Abbott AxSym.".

Thus, we have pretty good estimates of bias and imprecision

Method Level
Total
CV%
Slope Y-int
Bias
(units)
Bias %
CA 15-3, U/mL 24.5 3.9% 0.91 2.92 0.715 2.92%
66.6 3.8%       -3.074 4.62%
164.1 3.8% -11.849 7.22%
CA 19-9, U/mL 58.5 4.0% 1.07 -5.38  -1.285  2.2%
170.6 2.6%        6.562  3.85%
420.7 2.3  24.069  5.72%
CA 125 II, U/mL 24.9 2.7% 1.32  0.88  8.848  35.53%
61.9 3.2%     20.688  33.42%
165.4 3.4%  53.808  32.53%
AFP, ng/mL 12.1 1.9% 0.92  -0.45  -1.418  11.72%
83.9 2.3%        -7.162  8.54%
55.8 2.1%  -4.914  8.81%
CEA, ng/mL 2.0 2.4% 1.04  0.04  0.12  6%
15.9 1.5%     0.676  4.25%
36.6 1.5%  1.504  4.11%

So we have a lot of numbers, right? We have three levels of controls, so we have imprecision estimates for all of those levels. Using the regression equation, we can estimate the bias at each.

Looking at the raw numbers, you may find it difficult to judge the method performance. From experience, you might be able to tell when a particular method CV is high or low. But the numbers themselves don't tell the story.

If we want an objective assessment, we can set analytical goals - specify quality requirements - and use those to calculate the Sigma-metric. 

Determine Quality Requirements

Now that we have our imprecision and bias data, we're almost ready to calculate our Sigma-metrics. But we're missing one key thing: the analytical quality requirement. How good to cancer markers need to be?

In the US, traditionally labs look first to CLIA for guidance. Next, they might look at the biologic variation database (sometimes called the Ricos goals) to see desirable specifications for total allowable error. They might also look at the German Rilibak.

Method CLIA Goal Biologic Goal Rilibak Goal
CA 15-3 none  20.8% none
CA 19-9 none  39.0%  27.0%
CA 125 II none  35.4%  none
AFP none  21.8%  24.0%
CEA none  24.7%  24.0%

 As you can see, CLIA does not provide any guidance. Remember that the PT guidelines came out in the 1990s and haven't been updated since then. Meanwhile, the biologic variation database is updated every two years. The German Rilibak goals do not exist for every analyte, but it seems like around 25% is the general goal for cancer markers.

Since the Biologic goals are the most complete and most evidence-based quality requirements, we will use those for our Sigma-metric calculations.

Calculate Sigma metrics

Now the pieces are in place. Remember the equation for Sigma metric is (TEa - bias) / CV.

Example calculation: for CA 15-3, at the level 24.5 U/mL, the Biologic goal is 20.8%, 3.9% imprecision, 2.92% bias:

(20.8 - 2.92) / 3.9 = 17.88 / 3.9 = 4.6

Now, at the level of 66.6 U/mL, again using 20.8% as the quality requirement, at the level with 4.62% bias, 3.8% imprecision:

(20.8 - 4.62) / 3.8 = 16.188 / 3.8 = 4.3

Recall that in industries outside healthcare, 3.0 Sigma is the minimum performance for routine use. 6.0 Sigma and higher is considered world class performance.We'll simplify the table below and calculate all the Sigma-metrics.

Method Level
Total
CV%
Bias %
Quality
Requirement
Sigma-
metric
CA 15-3, U/mL 24.5 3.9% 2.92% 20.8% 4.6
66.6 3.8% 4.62% 4.3
164.1 3.8% 7.22% 3.9
CA 19-9, U/mL 58.5 4.0%  2.2% 39.0% 9.2
170.6 2.6%  3.85% 13.5
420.7 2.3  5.72% 14.5
CA 125 II, U/mL 24.9 2.7%  35.53% 35.4% 0
61.9 3.2%  33.42% 0.6
165.4 3.4%  32.53% 0.8
AFP, ng/mL 12.1 1.9%  11.72% 21.8% 5.3
83.9 2.3%  8.54% 5.8
55.8 2.1%  8.81% 6.2
CEA, ng/mL 2.0 2.4%  6% 24.7% 7.8
15.9 1.5%  4.25% 13.6
36.6 1.5%  4.11% 13.7

 Now things are starting to look clearer. We have a lot of good news here. Given the quality requirements, most of these methods pass the Six Sigma mark. There is one troublesome assay, CA 125 II, and if you look closely, the big problem is the bias (when we use the graphic tools, that will become apparent). One method, CA 15-3, has "merely" good performance. But the overall sense of the methods here is good performance.

Summary of Performance by Sigma-metrics Method Decision Chart and QC Design by OPSpecs chart

If the numbers are too much to digest, we can put this into a graphic format with a Six Sigma Method Decision Chart. Here's the chart for CLIA specifications for allowable total error

 2012-2011-VistaCancerMarkers-NormMedx

Here's where the graphic display helps reveal issues with performance. If the CA 125 II method could eliminate or reduce bias, its precision is pretty good. With lower bias, the method performance might even reach the Six Sigma zone. And we can see that only a small improvement in CA 15-3 performance would raise its Sigma-metric performance (of course, it's starting from a fairly good point already).

Now the question becomes, what would the laboratory do if this instrument was in routine operation? What QC would be necessary to assure the level of quality required for use of the tests? In this case, we use the same data, but plot the methods on an OPSpecs (Operating Specifications) chart.

2012-2011-VistaCancerMarkers-N3OPSpecs

For most of the analytes, we won't have to expend a lot of effort in implementing QC. 3 controls, wide control limits, low false rejection. Without making an improvement to the CA 125 II method bias, we can't really QC it into acceptability. But it's possible that a bias adjustment or recalibration could be made to bring that method into acceptability and routine control. With CA 15-3, a robust set of "Westgard Rules" may still keep it under control.

One mitigating factor for the bias problem with CA 125 II is noted in the study: "It is noteworthy that due to variability among tumor marker measurements, the same method should be used for individual patients. When changing methods, thorough studies to re-baseline individual patients should be conducted in collaboration with clinicians due to possible methodological differences."

In other words, if comparisons are not done between methods, then the bias becomes a non-factor. The performance is really only precision-driven at that point. In that case, the CA 125 II method is fine. Indeed, almost all the methods are very good if bias is eliminated from the calculations.

Conclusion

One of the big challenges evaluating tumor markers is, How good does performance need to be? It's interesting to note that CLIA provides no guidance, but the Biologic goals provide a complete set of goals. Usually the Rilibak goals are considered broad, but in this case, they were mostly smaller than what the biologic goals specified.

Analytical performance, for the most part, for these tumor markers is good. The entire study takes into account more factors than just the analytical performance, including such critical issues as patient concordance, detection limit, reference intervals, specimen types, etc. The study concludes: "measurements of CA 15-3, CA 19-9, CA 125 II, AFP and CEA with LOCI technology on the Vista analyzer demonstrated acceptable imprecision, low LoB and LoD, a wide AMR and an acceptable lot-to-lot stability."

 

847.714.3904 [Cell]