Tools, Technologies and Training for Healthcare Laboratories

A direct enzymatic assay for %HbA1c

We take a look at a new direct enzymatic method for %HbA1c.

November 2008

[Note: This QC application is an extension of the lesson From Method Validation to Six Sigma: Translating Method Performance Claims into Sigma Metrics. This article assumes that you have read that lesson first, and that you are also familiar with the concepts of QC Design, Method Validation, and Six Sigma. If you aren't, follow the link provided.]

This article looks at a new "Direct enzymatic assay for %HbA1c for human whole blood samples" by Limin Liu, Stefanie Hood, Yuping Wang, Robert Berverkov, Chao Dou, Abhijit Datta, Chong Yuan, Clinical Biochemistry 41 (2008) 576-583.

Remember, for HbA1c, the critical decision level is 7.0%. That's the crucial cutoff for determinations, diagnosis, and treatment choices. It's also becoming one of the key threshholds for pay-for-performance programs in diabetic management.

The Precision and Comparison data

Imprecision Estimates:

Two patient samples, one normal (5.7%) and one abnormal (10.3%) , were used to determine total imprecision using the study design from the CLSI EP5A protocol. With 80 data points, total imprecision for both normal and abnormal levels was calculated at 1.8%.

Sample
Mean
Total Imprecision
Normal Control
5.7
1.8%
Abnormal Control
10.3
1.8%

Given the stable estimate of 1.8%, we will assume that the method will also have 1.8% imprecision at a level of 7.0% HbA1c.

Bias estimates (added to the right):

A method comparison study was done with a set of whole blood samples. The Diazyme Direct Enzymatic HbA1c assay was compared to the Tosoh G7 HPLC method and the Roche Tina-quant II immunoassay method.

Comparative Method
N
Slope
Y-Int
r2
Tosoh G7 HPLC
66
0.973
0.111
0.98
Roche Tinaquant II immunoassay
66
1.10
-0.421
0.97

Ideally, the bias is calculated at the critical decision levels for test. Here, we will calculate the bias at the critical level of 7.0%

Remember that the correlation value (r2) is not the important statistic. It dies provide us with this useful information, however: if the correlation is higher that ) 0.975, then simple linear regression is adequate to provide us with good estimates of the systematic error. When correlation is lower than 0.975, it is recommended that a better regression technique should be used, such as a Deming regression or Passing-Bablock regression.

The authors of this study do not explain whether simple linear regression or Deming regression or Passing-Bablock was used to calculate the slope and y-intercept.

We will assume that the slope and y-intercept values that are provided are appropriate.

Calculate bias at the decision level

Now we take the comparison of methods data and set the equation to the level covered in the imprecision study. Solving those equations will give us a bias estimate.

Here are the steps for calculating bias:

((slope*level) + YIntercept) - level) / level = % bias

((0.973*7.0) +0.111) - 7.0) / 7.0 = ((6.811 +0.111) - 7.0) / 7.0

(6.922 - 7.0) / 7.0 = -0.078 / 7.0 = -0.0111 * 100 = 1.84%

The bias is expressed as a percentage of the level 7.0%, as an absolute value.

Comparative Method
N
Slope
Y-Int
bias%
Tosoh G7 HPLC
66
0.973
0.111
1.11%
Roche Tinaquant II immunoassay
66
1.10
-0.421
3.99%

Determine the quality requirements at the critical decision level

Now that we have both bias and CV estimates, we are almost ready to calculate the Sigma metrics for these analytes. The last (but not least) thing we need is the quality requirement for each method. CLIA provides quality requirements for over 80 analytes, but in this case, it does not provide a quality requirement.

Source
Quality Requirement
CLIA PT
No quality requirement given
CAP PT 2007
Target value ± 15%
CAP PT 2008
Target value ± 12%
NACB 2007 Draft
"interassay CV<5%
(ideally <3%)"
Clinical Decision Interval
Target value ± 14%
(biologically based)

The details of these sources and quality requirements are discussed in Dr. Westgard's essay. The important thing to note here is that there is a pretty big difference between the requirements. Note also that the NACB guidelines do not specifically state any analytical quality requirement - at best, you can infer the the quality requirement based on their specifications for instrument performance. Finally, remember that while the Clinical Decision Interval quality requirement is almost the biggest number, using that number requires that the process take into account the known within-subject biological variation, which eats up a large amount of the error budget.

Calculate Sigma metrics

Now we have all the pieces in place. What makes this different this time is that we're going to calculate two metrics, one for the bias estimate from the HPLC method, and one from the immunoassay method. Then, we can apply different quality requirements to the calculations. This will produce a dizzying variety of metrics.

Remember the equation for Sigma metric is (TEa - bias) / CV:

For a 12% quality requirement, with the HPLC method, (12 - 1.11) / 1.8 = 6.05

The metrics are displayed along the right columns.

Direct enzymatic assay compared to HPLC method
Source
TEa%
Biologic Variation %
CV%
Bias%
Sigma metric
CLIA PT
??
n/a
1.8%
1.11%
???
CAP PT 2007
15.0%
n/a
1.8%
1.11%
7.72
CAP PT 2008
12.0%
n/a
1.8%
1.11%
6.05
NACB 2007 Draft
??
n/a
better than "ideal" specification
1.11%
???
Clinical Decision Interval
14.0%
4.1%
1.8%
1.11%
4.71
Direct enzymatic assay compared to immunoassay method
Source
TEa%
Biologic Variation %
CV%
Bias%
Sigma metric
CLIA PT
??
n/a
1.8%
3.99%
???
CAP PT 2007
15.0%
n/a
1.8%
3.99%
6.1
CAP PT 2008
12.0%
n/a
1.8%
3.99%
4.45
NACB 2007 Draft
??
n/a
better than "ideal" specification
3.99%
???
Clinical Decision Interval
14.0%
4.1%
1.8%
3.99%
3.11

Most of what we see here is good news. Regardless of which method, HPLC or immunoassay, method performance according to CAP standards is good or world class. If we hold the test method to the higher standard of the clinical decision interval (a more "evidence-based" approach), then we should consider ways of improving performance.

Summary of Performance by Sigma-metrics chart and OPSpecs chart

Not only can we use tools to graphically depict the performance of the method - we can also use those tools to help determine the best QC procedure to use with that method. Using EZ Rules 3, we can express the method performance on an OPSpecs (Operating Specifications) chart :

alt

Note that this depiction uses the tighter CAP requirement of 12%, and using the performance specifications of 1.8% CV with 1.1% bias vs. HPLC method. The CV and bias become the x, y coordinates of the method's Operating Point. The rule of thumb for an OPSpecs chart is that you want to be as close to the origin of the graph as possible, and if your operating point is below and to the left of the lines on the chart, the QC procedures (shown in the key at the right of the graph) that are represented by those lines are acceptable. In this case, any of the QC procedures shown on the graph would provide adequate error detection - even 3s or 3.5s control limits with 2 controls per run.

Here's a different depiction of performance - a Sigma-metrics chart:

alt

In this case, the performance is literally "off the charts" - usually a bold, straight vertical line is drawn in the graph area - and where it intersects the power curves is where the QC procedure performance can be determined. The Sigma-metrics chart provides more specific numbers to the error detection and false rejection (see the key at right) as well as the specific Sigma-metric of the method.

If you want to visually assess the larger bias against the immunoassay method, you can add another QC Design to the same EZ Rules 3 file. Or you can just look at the existing OPSpecs chart and mentally move the y-coordinate of the operating point up to 3.99. The rise in bias definitely eliminates most of the QC procedures displayed on the chart.

To assess the performance at the 15% quality requirement, you could set up another EZ Rules 3 file, or add another QC Design to the same file, or you could plot all the data points on a Normalized OPSpecs chart.

One last look - apply the clinical QC design model to the data and you get this chart:

alt

Using 1.8 for CV and 1.1 for bias, you've still got many different QC procedures to choose from. The operating point is below 6 different QC procedures - there is even one QC procedure that would use just two controls. This means that the method performs well even when the more demanding constraints of clinical use and biological variation are taken into account.

Conclusions

This application shows that at least some methods are advancing in their performance - and will be able to meet CAP's tighter performance requirements.

This also shows the importance of determining the best reference method for your comparison studies. The bias figures that result from different reference/comparison methods are significantly different. You need to use your best judgment in determing the appropriate way to determine bias for the method when it is in your laboratory.