Tools, Technologies and Training for Healthcare Laboratories

Sigma-metrics for HBV

Can we calculate Sigma-metrics for Molecular Diagnostic methods? We came across an early example of applying Sigma-metrics to the evaluation of an HBV method. What happens when we look at performance on the logarithmic scale?

Sigma-metrics of an HBV method

April 2016
Sten Westgard, MS

bmv3edsmall [Note: This QC application is an extension of the lesson From Method Validation to Six Sigma: Translating Method Performance Claims into Sigma Metrics. This article assumes that you have read that lesson first, and that you are also familiar with the concepts of QC Design, Method Validation, and Six Sigma. If you aren't, follow the link provided.] sixsigma2edsmall

Sigma-metric calculations are well-established in "traditional" testing fields such as chemistry, hematology, urinalysis, etc. What about Molecular Diagnostics? Can we review viral loads on the Sigma scale? Last year in a paper we discussed this possibilty for HIV and HCV. But evidently we weren't the first to walk down this path. An earlier paper exists where Sigma-metrics have been calculated for an HBV method

Application of quality control planning methods for the improvement of a quantitative molecular assay, Reza Shahsiah, Fatemeh Nili, Farid Azmoudeh Ardalana, Fatemeh Pourgholia, Mohammad Ali Borumand, Journal of Virological Methods 193 (2013) 683– 686

The Units and Decision Levels

HBV is typically measured on the log scale. This represents a novel challenge for Sigma-metrics, which are typically not applied to anything except the usual numeric scale. Since the viral loads can vary so dramatically, from hundreds of thousands of copies down to single digits, the log scale is more convenient. But once we have narrowed our focus to a few key decision levels, we have the option to convert the log values back to real numbers.

The paper helps establish the clinical scenario and from there identify an allowable total error

"In other words, the efficacy of antiviral treatment is defined as the ability of an antiviral drug to cause at least  1 log(IU/mL) reduction in HBV DNA from the baseline.... Therefore, a total error of less than 1 log(IU/mL) does not affect the clinical decision-making. Because other parameters,including minor changes in serum, may also affect the HBV DNA level, the arbitrary value of 0.5 log(IU/mL) was considered as
the maximum total allowable error (TAE) of the assay in this study. Based on this assumption, the values of systematic and random
error were measured at 4.2 log(IU/mL) and 3.2 log(IU/mL), which are near clinical decision points."

So we know that the TEa is going to be 0.5 log, which converts to a goal of 216.23%. This is a pretty big goal, particularly for those of us who are used to the requirements of chemistry and hematology.

The log decision points can be converted into normal numbers:

Low level
2.43 = 269
2.93 = 851.1
Difference between 2.43 and 2.93 is 0.5 log but also 582.
A TEa of 582 at a level of 269 = 216.23%

High level
4 = 10,000
4.5 = 31,622.8
Difference between 4.5 and .0 is 0.5 log but also 21,622.78
A TEa of 21,622.78 at a level of 10,000 = 216.23%

The TEa's should match when they are expressed as % of normal units, because the allowable total eror matches in log units.

The Imprecision and Bias Data

For imprecision, "Two plasma samples with HBV-DNA levels of about 2 log(IU/mL) and 4 log(IU/mL) were prepared by the pooling of HBV DNA positive and negative human plasma. Two-hundred microliters aliquots were prepared at each level and were frozen at −20 C. Over a period
of 15 months, a total of 20 samples, of each level were extracted separately, as described above, and quantified by a QIAGEN Artus HBV
RG PCR assay. The total standard deviation (SDt) of the values, which were transformed logarithmically at each level, was considered as
an estimation of total random error."

For bias, "A lyophilized secondary WHO standard was purchased from the National Institute for Biological Standards and Control (Hertfordshire,
United Kingdom) at a level of 106 IU/mL. The standard was reconstituted with 0.5 mL distilled water, as per the manufacturer’s instructions, by means of a class A pipette. Using a class A pipette, dilutions were prepared by adding 100 L of the reconstituted standard to 5 mL HBsAg-negative plasma, in order to achieve an HBV DNA concentration of 19,608 IU/mL. Subsequently, 500 L of the diluted standard was added to 5 mL HBsAg-negative plasma, to achieve an HBV DNA concentration of 1783 IU/mL. Finally, 200 L aliquots were prepared at each level and were frozen at −20 C.

"For each day over a five day period, duplicates were extracted of the two diluted standards separately, and quantified by a QIAGEN
Artus HBV RG PCR assay. The difference of the mean of the values of each level from the target value was calculated. The logarithmic transformation of this value was considered as an estimation of systematic error. "

 Imprecision Log
Units Log SD Units SD Units CV%
Low Level 2.43 269 0.17 128.95 47.9%
 High Level 7.71 1.7% 0.17  4791.08 47.9%

Notice, we haven't calculated the bias yet. But we can see that the large size of the TEa is going to be offset by the large size of the imprecision.

 Bias Log
Units Units Bias%
Low Level 0.22 106.97 39.7%
 High Level 0.33 5322.65 53.2%

Now we should start to feel a little worried, because the biases are high as well as the imprecision. Those big TEa values might not be big enough.

We should note that viral load assays tend to be used in absence of bias. That is, the bias of a method doesn't matter quite as much because the patient is monitored by a single method and the changes over time are more important than any single specific cutoff. So for some clinical scenarios, we can consider the Sigma-metrics with an effective bias of zero.

Calculate Sigma metrics

Sigma-metrics takes both imprecision and bias into account in a single equation. We're going to calculate Sigma-metrics using both "Ricos goals" and the CLIA goals.

Remember the equation for Sigma metric is (TEa - bias%) / CV.

For the HBV Low Level:
with a 216.23% quality requirement, given 47.9% imprecision and 39.7% bias:

(216.23 - 39.7) / 47.9 = 176.53 / 47.9 = 3.68 Sigma

For the HBV High Level:
with a 216.23% quality requirement, given 47.9% imprecision and 53.2% bias:

(216.23 - 53.2) / 47.9 = 163.03 / 47.9 = 3.40 Sigma

If we zero out the bias, here are the Sigma-metrics:

For the HBV Low Level:
216.23 / 47.9 = 4.5 Sigma

For the HBV High Level:
216.23 / 47.9 = 4.5 Sigma

Overall, there are a lot of good metrics here.

Summary of Performance by Sigma-metrics Method Decision Chart and OPSpecs chart

We can make visual assessments of this performance using a Normalized Sigma-metric Method Decision Chart.

HBV Normalized Method Decision Chart

 

Now what about QC? How do we monitor and control these methods? For that, we need a Normalized OPSpecs chart:

2016 QiagenHBV NOPSpecs 2016 4 4

If we take into account the bias, we need to invest in some heavy "Westgard Rules" that build up 8 to 10 measurements. That means a look-back of 3 to 4 runs if we're only running 2 controls per run.

If we expect to use the viral load results in isolation (no comparisons made with other testing methods), we only need a set of "Westgard Rules" that don't include the 8:x or 10:x rule and only require one look-back run.

Conclusion

The authors concluded "In conclusion, the employment of long-term controls, and simple QC rules are recommended for detecting critical changes in the measurement system and for preventing erroneous lab results that would otherwise cause misdiagnosis and mismanagement."

Our conclusion is not quite as rosy - the "Westgard Rules" are needed, so it's not that simple to QC this method. But it's well within the capability of a laboratory to implement without excessive additional expense.