Tools, Technologies and Training for Healthcare Laboratories

Seven Vitamin D methods

In a 2008 issue of Annals of Clinical Biochemistry, a study looked a seven different 25-hydroxyvitamin D methods, compared with liquid chromatography-tandem mass spectometry. Using quality requirements developed by and derived from Stockl, Sluss, and Thienpont, we examine this methods with Sigma-metric analysis.

November 2009
Sten Westgard, MS

[Note: This QC application is an extension of the lesson From Method Validation to Six Sigma: Translating Method Performance Claims into Sigma Metrics. This article assumes that you have read that lesson first, and that you are also familiar with the concepts of QC Design, Method Validation, and Six Sigma. If you aren't, follow the link provided.]

This application looks at a paper from a 2008 issue Annals of Clinical Biochemistry which examined seven different 25-hydroxyvitamin D methods, compared to a reference method of liquid chromatography-tandem mass spectrometry. For reasons that will become clear, we're not going to use the brand names of the 25-OH D methods.

We'll refer to these methods as:

  • HPLC (High-performance liquid chromatography)
  • RIA (Radioimmunoassay)
  • EIA (Enzyme immunoassay)
  • CPBA (Competitive chemiluminescence protein-binding assay)
  • CLIA 1 (Competitive chemiluminescence immunoassay)
  • CLIA 2 (Competitive chemiluminescence immunoassay)
  • ECLIA (Electrochemiluminescence immunoassay)

The Precision and Comparison Data

The paper gives between-run imprecision data.

Precision Data
Method Level (nmol/L)) Between-run CV%
HPLC 73 6.5%
250 2.3%
RIA 58 8.1%
135 7.3%
EIA 73 6.4%
133 8.7%
CPBA 50 9.9%
125 6.4%
CLIA 1 38 14.8%
133 13.2%
CLIA 2 38 10.2%
133 8.4%
ECLIA 48 4.7%
178 5.1%

Note that Stockl, Sluss, and Thienpont recommend a 10% maximum CV for routine laboratory measurements, with a further recommendation that imprecision performance should actually better than 5%.

Analytical inaccuracy was assessed using nearly 300 randomly selected patient samples. Passing-Bablok regression analysis was performed. The reference method used was liquid chromatography-tandem mass spectrometry (LC-MS/MS.

 

Instrument Slope Y-intercept
HPLC 1.00 -0.64
RIA 0.64 9.35
EIA 0.62 7.28
CPBA 0.85 -1.33
CLIA 1 0.75 1.04
CLIA 2 0.83 4.29
ECLIA 0.94 -3.45

We aren't given any correlation numbers here, so we must make an assumption that correlation is fine. In any case, the correlation number only tells us if linear regression is sufficient to calculate the bias. If the correlation was below 97% or so, Deming or Passing-Bablok regression would be a better method to assess inaccuracy. Since we're already using Passing-Bablok regression, we're fine.

Based on the slope and y-intercept alone, it's hard to know which methods are more accurate. Certainly HPLC looks good with "perfect" slope. The other methods have very different slopes and intercepts.

Now we need to align our inaccuracy data with our imprecision data. The easiest thing to do is to calculate the bias at the levels where the imprecision was calculated, using the regression equation.

Just to review this calculation, here's a layman's explanation of the equation:

NewLevelNewMethod = ( slope * OldLevelOldMethod ) +Y-intercept

Then we take the difference between the New and Old level, and convert that into a percentage value of the Old level.

Example Calculation: Given HPLC at a level of 13 nmol/L, comparing to LC-MS/MS:

NewLevelHPLC = (1.00 * 73) - 0.64

NewLevelHPLC = 73 - 0.64

NewLevelHPLC = 72.36

Difference = 73 - 72.36 = 0.64

Bias% = 0.64 / 73 = 0.88%

 

Precision and Accuracy
Method Level (nmol/L)) Between-run CV% Bias%
HPLC 73 6.5% 0.88%
250 2.3% 0.26%
RIA 58 8.1% 19.9%
135 7.3% 29.1%
EIA 73 6.4% 28.0%
133 8.7% 32.5%
CPBA 50 9.9% 17.7%
125 6.4% 16.1%
CLIA 1 38 14.8% 22.3%
133 13.2% 24.2
CLIA 2 38 10.2% 5.7%
133 8.4% 13.8%
ECLIA 48 4.7% 13.2%
178 5.1% 7.94%
You can begin to see some concerns here. There are some large biases - differences between the test method and the reference method at the levels measured in this study.

Determine Quality Requirements at the decision level

Now that we have our imprecision and inaccuracy data, we're almost ready to calculate our Sigma-metrics. But we're missing one key thing: the analytical quality requirement.

This is where Stockl, Sluss, and Thienpont come in. They set recommendations for both maximum CV and Bias, as well as routine performance CV and Bias, needed for normal laboratory measurement.

Method
Level
Meets Maximum
CV% Goal 10%
Meets Maximum
Bias% Goal 5%
Meets Optimal
CV% Goal 5%
Meets Optimal
Bias% Goal 2.5%
HPLC 73 Y Y N Y
250 Y Y Y Y
RIA 58 Y N N N
135 Y N N N
EIA 73 Y N N N
133 Y N N N
CPBA 50 Y N N N
125 Y N N N
CLIA 1 38 N N N N
133 N N N N
CLIA 2 38 N N N N
133 Y N N N
ECLIA 48 Y N Y N
178 Y N Y N

As you can clearly see, there are a lot of No's there. That means performance did not meet either the maximum goal or the optimal goal for performance.

But we can make this performance assessment even more quantitative with Sigma-metrics. We can give each method a calculated rank on a universal process scale of 1 to 6.

One of the things we need is a quality requirement for total allowable error. In this case, we can use Stockl, Sluss and Thienpont's recommendations to calculate a TEa. In a separate article, we calculated this at 25%.

Calculate Sigma metrics

Now the pieces are in place.

Remember the equation for Sigma metric is (TEa - bias) / CV.

Example calculation: for our 21.5% quality requirement, with HPLC, at the level of 73 nmol/L,   6.5% imprecision, 0.88% bias:

(25 - 0.88) / 6.5 = 24.12 / 6.5 = 3.71

Sigma-metrics
Method Level (nmol/L)) Btwr CV% Bias% Sigma-metric
HPLC 73 6.5% 0.88% 3.71
250 2.3% 0.26% 10.75
RIA 58 8.1% 19.9% 0.62
135 7.3% 29.1% negative
EIA 73 6.4% 28.0% negative
133 8.7% 32.5% negative
CPBA 50 9.9% 17.7% 0.73
125 6.4% 16.1% 1.40
CLIA 1 38 14.8% 22.3% 0.18
133 13.2% 24.2 0.06
CLIA 2 38 10.2% 5.7% 1.90
133 8.4% 13.8% 1.33
ECLIA 48 4.7% 13.2% 2.51
178 5.1% 7.94% 3.35

Sobering results. With the exception of the HPLC method, most of these methods are not meeting the quality required by the quality requirement generated by . In cases where the Sigma-metric is "negative", that really means that bias is so large that it exceed the quality requirement. There is actually no way a method with a 32% bias can hit a 25% quality target.

Summary of Performance by Sigma-metrics Method Decision Chart

The graph below shows that both bias and imprecision are a problem with these methods:

Method Decision Chart, vitamin D methods

Conclusion

As Stockl, Sluss and Thienpont pointed out in their study, the performance of 25(OH) vitamin D methods is not adequate to the clinical use of the test. Major improvements need to be made by manufacturers.

Recall that in industries outside healthcare, 3.0 Sigma is the minimum performance for routine use. Clearly, in healthcare, we've been holding vitamin D methods to a lower standard of performance.

One might note that if we ignored bias, a few methods would be more acceptable. (You can make the same Sigma-metric calculation as before, using only the CV data and let Bias=0). The EIA and CPBA methods, with zero bias, would begin to approach 3 Sigma. The ECIA method, with zero bias, would have Sigma-metrics in the range of 4 to 5, which would be considered good. But in cases where we ignore bias, essentially we admit that our results cannot be compared to any other results and, furthermore, we can't accept any other results coming into our system at face value. To make the leap to an assumption of zero bias, it will require us to rely only on the results (possibly multiple times) within our own system to establish the patient's clinical status.

What can be done with methods that can't reach the goals set by Stockl, Sluss, and Thienpont? Options include running multiple specimens, or making replicate measurements, or even adjusting the clinical use and interpretation of the test results. If vitamin D tests are not good enough for tight cutoffs and decision-making, clinicians need to know.