Tools, Technologies and Training for Healthcare Laboratories

Six Hcy methods

Elevated concentrations of homocysteine (Hcy) are clinically associated with disorders and cardiovascular disease. A recent paper compared six different methods for this important analyte. But in the absence of a CLIA quality requirement, how can we compare performance?

Sigma metrics of Six Homocysteine Methods

January 2009
Sten Westgard, MS with advice from David Plaut, MS


[Note: This QC application is an extension of the lesson From Method Validation to Six Sigma: Translating Method Performance Claims into Sigma Metrics. This article assumes that you have read that lesson first, and that you are also familiar with the concepts of QC Design, Method Validation, and Six Sigma. If you aren't, follow the link provided.]

This article looks at a recent paper, "Performance Characteristics of Six Homocysteine Assays" by Sonia L. La'ulu, Mindy L Rawlins, Christine M. Pfeiffer, Mindy Zhang, and William L Roberts, American Journal of Clinical Pathology 2008; 130:969-975.

Elevated concentrations of homocysteine (Hcy) are clinically associated with many disorders, but the most common use of the test is for cardiovascular disease. Hcy is also a sensitive marker for folate and cobalamin deficiency, and can be used to diagnose homocystinuria.

One of the obstacles to assessing Sigma performance for homocysteine is that it is an "unregulated" analyte. That is, CLIA does not provide a allowable error for proficiency testing. Furthermore, there is no widely accepted critical threshold for clinical interpretation.


The Precision and Comparison data

One of the immediate problems confronting this application is determining the proper imprecision value. While values for imprecision may be given for many levels, in practice, the Sigma calculation and subsequent QC Design should be centered on the most critical (most important) level. As noted earlier, there is no widespread agreement on a single cutoff, level, or threshold for clinical interpretation.

Working with David Plaut, MS, we performed a review of the literature on the use and interpretation of homocysteine values. This review produced a rough consensus that the range of 10-15 umol/L is mentioned frequently as an important range for clinical interpretation. We will take 15 umol/L as the critical decision level. Thus, for all the imprecision estimates, we will use the ones closest to that level.

Imprecision Estimates:

Imprecision was determined using commercially available quality control material at three concentration levels (only the level closest to 15 umol/L is displayed. Controls were run twice a day, for 5 days, in replicates of 2.

Instrument Mean Total Imprecision
ADVIA Centaur 19.2 4.3
Abbott ARCHITECT i2000 17.1 2.1
AxSYM 16.0 3.4
Catch (Roche Modular P) 16.7 3.4
Diazyme (Roche Modular P) 14.9 2.5
IMMULITE 2000 13.8 8.3

Bias estimates (added to the right):

A method comparison study was done using high-performance liquid chromatography (HPLC) with fluorescence detection as the reference method. 101 specimens were used with a range of 4.4 to 52.7 umol/L (measured by the HPLC). Deming regression was used.

Comparative Method N Slope Y-Int r
ADVIA Centaur 101 0.91 3.05 0.95
Abbott ARCHITECT i2000 101 0.93 0.64 0.99
AxSYM 101 0.92 0.46 0.98
Catch (Roche Modular P) 101 0.93 1.85 0.98
Diazyme (Roche Modular P) 101 0.91 3.03 0.97
IMMULITE 20000 101 0.90 2.86 0.98

Ideally, the bias is calculated at the critical decision levels for test. As we stated before, we will calculate the bias at the critical level of 15.0 ummol/L

Remember that the correlation value (r) is not the important statistic. It does provide us with this useful information, however: if the correlation is higher than 0.975, then simple linear regression is adequate to provide us with good estimates of the systematic error. When correlation is lower than 0.975, it is recommended that a better regression technique should be used, such as a Deming regression or Passing-Bablock regression. However, since Deming regression was used from the start, this is not an important issue here.

Calculate bias at the decision level

Now we take the comparison of methods data and set the equation to the level covered in the imprecision study. Solving those equations will give us a bias estimate.

Here are the steps for calculating bias:

((slope*level) + YIntercept) - level) / level = % bias

((0.91*15.0) +3.05) - 15.0) / 15.0 = ((13.65 +3.05) - 15.0) / 15.0

(16.7 - 15.0) / 15.0 = 1.7 / 15.0 = 0.1133 * 100 = 11.33%

The bias in the table below is expressed as a percentage of the level 15.0 umol/L, where the absolute value of the bias in units is taken.

Comparative Method Slope Y-Int % Bias
ADVIA Centaur 0.91 3.05 11.3%
Abbott ARCHITECT i2000 0.93 0.64 2.73%
AxSYM 0.92 0.46 4.93%
Catch (Roche Modular P) 0.93 1.85 5.33%
Diazyme (Roche Modular P) 0.91 3.03 11.2%
IMMULITE 20000 0.90 2.86 9.07%

Determine the quality requirement at the critical decision level

Now that we have both bias and CV estimates, we are almost ready to calculate the Sigma metrics for these analytes. The last (but not least) thing we need is the quality requirement for each method. Unfortunately, CLIA does not provide a quality requirements for homocysteine. Hcy is one of those "unregulated" analytes.

While there are few official guidelines for quality requirements, Carmen Ricos and colleagues do cover homocysteine in their database of biologic variation. Based on published data on variation in biologic variation, Ricos et al. calculated 17.7% as the total allowable error for homocysteine.

Calculate Sigma metrics

Now we have all the pieces in place.

Remember the equation for Sigma metric is (TEa - bias) / CV:

For a 17.7% quality requirement, with the Advia Centaur, (17.7 - 11.3) / 4.3 = 1.48

The metrics are displayed along the right columns.

Comparative Method CV% Bias% Sigma Metric
ADVIA Centaur 4.3 11.3 1.48
Abbott ARCHITECT i2000 2.1 2.73 7.13
AxSYM 3.4 4.93 3.75
Catch (Roche Modular P) 3.4 5.33 3.64
Diazyme (Roche Modular P) 2.5 11.2 2.60
IMMULITE 20000 8.3 9.07 1.04

A pretty wide range of performance here. The ARCHITECT method pops up, while the rest are clustered around the 3 Sigma range.

Summary of Performance by Sigma-metrics chart

Here's a normalized Method Decision chart, using Six Sigma metrics lines to delineate the performance of the methods. This is the same data from the table above presented graphically.

qcapp54hcynormmedx2008

If you think of the Method Decision chart as a target, with the origin (0,0) as the bulls' eye, there are a few methods that are close to the center of the target, but a few that seem to be missing the target entirely.


QC Design using OPSpecs chart

Not only can we use tools to graphically depict the performance of the method - we can also use those tools to help determine the best QC procedure to use with that method. Using EZ Rules 3, we can express the method performance on an OPSpecs (Operating Specifications) chart :

qcapp54OPSpecs

With OPSpecs charts, the lines above the operating point mean that those QC procedures are acceptable for use with that method. For example, with the ARCHITECT i2000, all of the lines are above the operating point, so any of the rules will suffice. A 1:3.5s rule with only 2 controls will still provide adequate error detection. If you look in the key at right, you'll see that there is essentially no danger of false rejection (repeats) with this control procedure.

For other methods, the QC requirements for proper quality assurance are more demanding. Take the AxSYM method (data shown here on the EZ Rules 3 program screen):

qcapp54ez3screen_2

In this case, Automatic QC Selection has determined that a set of "Westgard Rules" with N of 4 is needed to provided the error detection necessary for the quality required by the test. An N of 4 means that either 4 controls are in a run, or the usual 2 controls are read more than once.

We can look further into this scenario using the Sigma-metrics chart:

qcapp54sigmachart

This chart shows that the "Westgard Rules" combination will provide 84% error detection for the performance of AxSYM method. That 84% means that most errors will be detected in the first run. (The average run length before detection is 1.19.)


Conclusions

Usually, we don't venture into "non-regulated" territory with analytes. It's easier to examine methods where the CLIA rules are specific. However, this example shows that - with the help of a literature search and helpful, smart colleagues - even the non-regulated analytes like homocysteine can be examined.

Also, this example shows that there is a wide range of performance for this test. Some instruments perform at world class levels. Others require extra QC effort to make up for poor performance. Still other instruments are unacceptable; there aren't enough controls to provide the error detection necessary to meet the quality required by the test. Only severe measures like duplicate testing have a chance to bring those methods back to acceptable levels.