Tools, Technologies and Training for Healthcare Laboratories

Cancer Markers on the UniCel DxI

A look at the UniCel DxI 800 Immunoassay Analyzer and its performance for five tumor markers.

Cancer Markers on the UniCel DxI Immunoassay Analyzer

  • The Precision and Comparison data
  • Determine quality requirements at the measured decision levels
  • Calculate Sigma metrics
  • Summary of Performance by Sigma-metrics chart and OPSpecs Chart
  • Conclusion
  • AUGUST 2013
    Sten Westgard, MS

    [Note: This QC application is an extension of the lesson From Method Validation to Six Sigma: Translating Method Performance Claims into Sigma Metrics. This article assumes that you have read that lesson first, and that you are also familiar with the concepts of QC Design, Method Validation, and Six Sigma. If you aren't, follow the link provided.]

    This application looks at a paper from a 2012 issue of Yonsei Medical Journal which examined the performance of cancer markers on the UniCel DxI 800 Immunoassay Analyzer:

    Evaluation of the UniCel DxI 800 Immunoassay Analyzer in Measuring Five Tumor Markers, Younhee Park, Yongjung Park, Jungyong Park, and Hyon-Suk Kim, Yonsei Med J 2012 May 1; 53(3): 557-564

    Please note: all brands are commercial trademarks of their respective companies. For example UniCel is a trademark of Beckman Coulter, Centaur is a trademark of Siemens, Vitros ECi is a trademark of Ortho, etc.

    The Imprecision and Bias Data

    For determination of precision,

    "Imprecision of the assays by DxI was assessed based on guidelines from the Clinical and Laboratory Standards Institute (CLSI) document EP4-A2, using commercially available quality control materials of three levels (MAS T-Marker; Medical Analysis Systems, Camarillo, CA, USA) and pooled sera for the respective markers. Two daily runs of duplicate testing were conducted per day for 20 days, with a minimum of 2 hours between runs."

    As for bias (the difference between the DxI methods and the ADVIA Centaur or Vitros ECi methods),

    "Correlation between the levels of tumor markers as measured by DxI and Centaur or ECi were evaluated in 200 specimens for respective tumor markers (total 2000 tests with 1000 samples). All sera, which were requested for tumor marker testing, were collected and assayed for the respective tumor markers with DxI and other comparative instruments on the same day. The samples with measured concentrations over the analytical measurement range were re-tested after dilution according to the manufacturers’ instructions."

    Thus, we have pretty good estimates of bias and imprecision. [The 4th row for each marker represents the performance of the pooled serum control.] Furthermore, the study used Passing-Bablock to calculate the regression slopes and y-intercepts.

    Method Level
    Total
    CV%
    Slope Y-int
    Bias
    (units)
    Bias %
    CEA, U/mL 1.25 4.5% 0.91   -0.24   -.353 28.2%
    10.52 3.8% -1.187 11.28%
    48.13 4.4% -4.572 9.5%
    3.39 3.7% -0.545 16.08%
    AFP, U/mL 4.7 5.4% 0.766   -0.624   -1.724 36.68%
    22.53 5.0% -5.896 26.17%
    100.7 5.5% -24.040 24.02%
    88.91 4.3% -21.429 24.1%
    CA 15-3, U/mL 5.02 4.9% 0.507   1.052   -1.423 28.34%
    20 4.2% -8.808 44.04%
    33.38 4.4% -15.404 46.15%
    8.49 4.3% -3.134 36.91%
    CA 125, U/mL 15.6 3.6% 1.118  -1.689 0.152 0.97%
    34 2.8%  2.323 6.83%
    117.72 3.0% 12.202 10.37%
    23.48 3.1%  1.082 4.61%
    CA 19-9, U/mL 8.53 4.4% 0.606    1.512    -1.849 21.67%
    32.33 3.9% -11.226 34.72%
    146.71 3.7% -56.292 38.37%
    14.85 4.1%  -4.339 29.22%

    So we have a lot of numbers, right? We have three levels of controls, plus one pooled patient serum level, so we have imprecision estimates for all of those levels. Using the regression equation, we can estimate the bias at each level.

    Looking at the raw numbers, you may find it difficult to judge the method performance. From experience, you might be able to tell when a particular method CV is high or low. But the numbers themselves don't tell the story. For instance, the imprecision looks good, but the bias numbers are probably concerning to you.

    If we want an objective assessment, we can set analytical goals - specify quality requirements - and use those to calculate the Sigma-metric. 

    Determine Quality Requirements

    Now that we have our imprecision and bias data, we're almost ready to calculate our Sigma-metrics. But we're missing one key thing: the analytical quality requirement. How good to cancer markers need to be?

    In the US, traditionally labs look first to CLIA for guidance. Next, they might look at the biologic variation database (sometimes called the Ricos goals) to see desirable specifications for total allowable error. They might also look at the German Rilibak.

    Method CLIA Goal Biologic Goal Rilibak Goal
    CEA none 24.7% 24.0%
    AFP none 21.19% 24.0%
    CA 15-3 none  20.8% none
    CA 125 none  35.4%  27.0%
    CA 19-9 none  39.0%  none

     As you can see, CLIA does not provide any guidance. Remember that the PT guidelines came out in the 1990s and haven't been updated since then. Meanwhile, the biologic variation database is updated every two years. The German Rilibak goals do not exist for every analyte, but it seems like around 25% is the general goal for cancer markers.

    Since the Biologic goals are the most complete and most evidence-based quality requirements, we will use those for our Sigma-metric calculations.

    Calculate Sigma metrics

    Now the pieces are in place. Remember the equation for Sigma metric is (TEa - bias) / CV.

    Example calculation: for CA 125, at the level 34 U//mL, the Biologic goal is 35.4%, 2.8% imprecision, 6.83% bias:

    (35.4 - 6.83) / 2.8 = 28.57 / 2.8 = 10.2

    Recall that in industries outside healthcare, 3.0 Sigma on the short-term scale is the minimum performance for routine use. 6.0 Sigma and higher is considered world class performance.We'll simplify the table below and calculate all the Sigma-metrics.

    Method Level
    Total
    CV%
    Bias %
    Quality
    Requirement
    Sigma-metric
    CEA, U/mL 1.25 4.5% 28.2% 24.7% none
    10.52 3.8% 11.28% 3.53
    48.13 4.4% 9.5% 3.45
    3.39 3.7% 16.08% 2.33
    AFP, U/mL 4.7 5.4% 36.68% 21.19% none
    22.53 5.0% 26.17% none
    100.7 5.5% 24.02% none
    88.91 4.3% 24.1% none
    CA 15-3, U/mL 5.02 4.9% 28.34% 20.8% none
    20 4.2% 44.04% none
    33.38 4.4% 46.15% none
    8.49 4.3% 36.91% none
    CA 125, U/mL 15.6 3.6% 0.97% 35.4 9.56
    34 2.8% 6.83% 10.20
    117.72 3.0% 10.37% 8.34
    23.48 3.1% 4.61% 9.93
    CA 19-9, U/mL 8.53 4.4% 21.67% 39.0% 3.94
    32.33 3.9% 34.72% 1.10
    146.71 3.7% 38.37% 0.17
    14.85 4.1% 29.22% 2.39

    Now things are starting to look clearer. We have some good news and some real questions. For most of these assays, the imprecision is fairly good, but there is a troublesome difference between the DxI methodology and the Centaur or ECi methodology. This results in a bias that exceeds the total allowable error, and thus overwhelms the Sigma-metric calculation. When we use the graphic tools, that will become even more apparent.

    When we see "none" as the Sigma-metric, that simply means the bias calculated completely exceeds the total allowable error. The difference is so large there isn't really a metric to calculate. The method cannot hit the target - it's aiming somewhere else.

    Summary of Performance by Sigma-metrics Method Decision Chart and QC Design by OPSpecs chart

    If the numbers are too hard to digest, we can put this into a graphic format with a Six Sigma Method Decision Chart. Here's the chart for CLIA specifications for allowable total error

     2013-2012-DxI-800-CancerMarkers-NormMedx

    Here's where the graphic display helps reveal issues with performance. All the levels of CA 125 II method are definitely in the bull's eye. But all the other methods have bias problems, which are really more of a difference in methodology. The new DxI methods are just not comparable to the Centaur and ECi methods.  If we take bias out of the picture, you could drop all those operating points down to the x-axis. And If bias was thus eliminated or reduced, CA19-9 would be in the bull's eye, CEA could be a good to excellent method, and CA 15-3  and AFP could be marginal to good.

    Now the question becomes, what would a laboratory do if this instrument was in routine operation? If they were switching from Immulite to Vista, or had both instruments in operation in the same facility, what QC would be necessary to assure the level of quality required for use of the tests? In this case, we use the same data, but plot the methods on an OPSpecs (Operating Specifications) chart.

    2013-2012-DxI-800-CancerMarkers-NOPSPECS-N3

    First off, CA 125 is quite easy to control: 1:3s or 1:3.5s rules would provide more than adequate detection. For the other methods, if we ignore, reduce or eliminate bias, we could control most methods with a "Westgard Rules" multirule QC procedure designed around 3 controls.

    The study authors noted the bias issue. When methods did not have good correlation, the analytic concordance between the DxI methods and the Centaur/ECi methods dropped to 89.5% for CA 19-9 and 84.5% for CA 15-3:

    "The extensive differences between analyzer systems...could be caused by different antibodies utilized by the assaying systems or by unique circumstances of the samples....[T]he large mean differences between the results of AFP and CA 19-9 measured by DxI and the other systems seem to be due to the high antigen levels of the samples tested in our study....Therefore, even though the results, on average, agreed fairly well across the assays, when replacing tumor marker assays for clinical use, parallel tests by old and new methods are recommended to establish a new baseline in the management of patients."

    Essentially, in order for a laboratory to make this instrument transition, all patients whose care spans the switch from the old methodology to the new methodology must be re-baselined. The old results will have to be ignored and the care must be judged solely on the new values coming from the new method. We don't want clinicians looking back at the old results and comparing them with the new results, because a methodology difference might be so large it will appear like a clinical change in the patient.

    Conclusion

    One of the big challenges evaluating tumor markers is, How good does performance need to be? It's interesting to note that CLIA provides no guidance, but the Biologic goals provide a complete set of goals. Usually the Rilibak goals are considered too broad, but in this case, they were about the same as the biologic goals.

    Precision performance, for the most part, for these tumor markers is good. But as the study authors conclude, "In spite of efforts to harmonize the results from different analyzers developed by different manufacturers, discrepant results remain among analytical methods. These differences may result from the application of different antibodies by different assays and suppliers. Additional efforts to standardize tumor marker assays are greatly necessitated, and the establishment of reliable reference materials and methods are also needed. The substantial differences between methods also indicate that the redetermination of baselines and cutoff levels is necessary when replacing analyzers and methods for measuring tumor marker assays."

    The Sigma-metrics make it clear that there are big differences between methods. Until we make improvements in standardization of tumor marker assays, we will get metrics that tell us bias is a major problem and that special handling (re-baselining, re-drawing the cut off levels) will be required with shifts in instrumentation and methodology.


     

    847.714.3904 [Cell]