Tools, Technologies and Training for Healthcare Laboratories

HbA1c on the cobas 6000

Analysis of the Roche cobas 6000, apply the different analytical quality requirements for Glycated hemoglobin (GHb) and evaluate its performance. What requirements should you pick? What rules should you use?


March 2008

[Note: This QC application is an extension of the lesson From Method Validation to Six Sigma: Translating Method Performance Claims into Sigma Metrics. This article assumes that you have read that lesson first, and that you are also familiar with the concepts of QC Design, Method Validation, and Six Sigma. If you aren't, follow the link provided.]

Continuing in our series of Sigma analyses of Glycohemoglobin, we're going to take a real-world instrument, the cobas 6000, and impose upon it various proposed quality requirements. As discussed in an earlier essay, despite years of earnest efforts, the 2007 draft NACB guidelines for Glycohemoglobin hasn't provided specific analytic quality requirements.

The source of data for this application is a poster from the 2007 AACC conference: (D-18) Performance Evaluation of HbA1c(%) on the cobas 6000 analyzer, P.V. Whelan, A. Bryson, Reid Hospital & Health Care Services, Richmond, IN

The Precision and Comparison data

The abstract states that the data was collected in the following manner:

"Imprecision was performed each day in triplicate over a period of 10 days. Method comparison of cobas 6000 analyzer HbA1c was performed against the Bio-Rad D10 HPLC method. Correlation studies were performed over 10 days."

The imprecision data is the equivalent of a short-term study, which means that the estimates will probably be optimistic. A long-term imprecision study would provide a more realistic estimate of imprecision. The "correlation" study, really a comparison of methods study, is approprate in terms of time span and data points.

All that we need to do now is select which estimates to use, supply the quality requirements and calculate the Sigma metrics.

Imprecision Estimates:

The Roche study calculates total imprecision estimates.

Level
Mean
Total Imprecision
1
5.885
1.70%
2
10.242
1.10%

Remember that the critical medical decision level that the NACB discusses is 7.0% GHb. That's the new threshold between normal and diabetic. This decision level lies right between the levels of the two controls. Lacking specific data on performance of this method at the 7.0% GHb level, we're going to have to use our professional judgment.

So, for the purposes of this application, we're going to use a different approach than usual: We're going to interpolate the imprecision between these values and use an estimate of 1.6% imprecision for performance the critical decision level of 7.0% GHb. A more conservative approach would use the 1.70% as the worst case scenario.

Comparison of Methods Data: Test Method (Cobas 6000) vs. Bio-Rad D10 HPLC (reference method)

N
Slope
Y-Int
r
133
0.90
0.61
0.9897

Remember that the correlation coefficient is not the key statistic here. The value of the correlation coefficient merely tells us that linear regression would be sufficient for these analytes (for those r values below 0.95, other forms of regression like Deming or Passing-Bablock are preferable, but in this case, are not available).

Calculate bias at the decision level

Now we take the comparison of methods data and set the equation to the decision level. Solving those equations will give us a bias estimate.

Here are the steps for calculating bias:

((slope*level) + YIntercept) - level) / level = % bias

((0.90*7.0) +0.61) - 7.0) / 7.0 = ((6.3 +0.61) - 7.0) / 7.0

(6.91 - 7.0) / 7.0 = -0.09 / 7.0 = -0.0128 * 100 = 1.28%

Level
Slope
Y-Int
Bias%
7.0
0.90
+0.61
1.28%

Determine the quality requirements at the critical decision level

Now that we have both bias and CV estimates, we are almost ready to calculate the Sigma metrics for these analytes. The last (but not least) thing we need is the quality requirement for each method. CLIA provides quality requirements for over 80 analytes, but for Glycated Hemoglobin, it does not.

Source
Quality Requirement
CLIA PT
No quality requirement given
CAP PT 2007
Target value ± 15%
Stricter Standards
Target value ± 10%
NACB 2007 Draft
"interassay CV<5%
(ideally <3%)"
Clinical Decision Interval
Target value ± 14%
(biologically based) Dint

The details of these sources and quality requirements are discussed in Dr. Westgard's essay. The important thing to note here is that there is a pretty big difference between the requirements. Note also that the NACB guidelines do not specifically state any analytical quality requirement - at best, you can infer the the quality requirement based on their specifications for instrument performance. Finally, remember that while the Clinical Decision Interval quality requirement is almost the biggest number, using that number requires that the process take into account the known within-subject biological variation, which eats up a large amount of the error budget.

Calculate Sigma metrics

Now we have all the pieces in place.

Remember the equation for Sigma metric is (TEa - bias) / CV:

For Glycohemoglobin, using the CAP PT 2007 requirement (15.0 - 1.84) / 2.0 = 6.58

Source
TEa%
Biologic Variation %
CV%
Bias%
Sigma metric
CLIA PT
??
n/a
1.6%
1.28%
???
CAP PT 2007
15.0%
n/a
1.6%
1.28%
8.57
Stricter Standard
10.0%
n/a
1.6%
1.28%
5.45
NACB 2007 Draft
??
n/a
better than "ideal" specification
1.28%
???
Clinical Decision Interval
14.0% Dint
4.1%
1.6%
1.28%
5.06

Because neither CLIA nor the NACB set a quantifiable quality requirement, it's not possible to calculate a Sigma metric for those scenarios. CLIA provides no information, NACB doesn't provide enough.

Given the current CAP PT requirement of 15%, this cobas method provides world class quality. Even if we tighten the quality requirement a bit (some people at CAP admit that 15% is too large), the Sigma metric of the cobas method is pretty good.

If we aim for "evidence-based" quality design, however, the demands on performance are more challenging. By using a Clinical Decision Interval (Dint), we must also account for within-subject biological variation, so even though we've got a 14% quality requirement, individual biologic variation is consuming 4.1% of that budget. Still, while this framework produces the lowest Sigma, it's pretty good.

Evaluation of Performance by OPSpecs chart, Sigma-metrics graph, and EZ Rules 3

Using EZ Rules 3, you can use QC Design to evaluation the impact of different quality requirements on testing performance. By using Automatic QC Selection, you can see the ideal rules and controls needed to provide appropriate QC.

Here's the OPSpecs chart for the 15% CAP requirement :

alt

The operating point is well below any of the possible candidate procedures. Any of those rules will do, but the best procedure would be to use 3.5s limits and only two controls. If you examine the Sigma-metrics chart (not pictured here), you'll note that the recommended rule selection has essentially no false rejection - you'll only get a flag when there is a genuine problem.

The moral of the story: if you achieve (or exceed) Six Sigma performance, you can do any kind of QC you want.

Now let's look at the case for a tighter requirement of 10%. This tighter requirement may be more appropriate, since CLIA states that glucose results should also meet at 10% quality requirement. Some people at CAP believe that 10% is the proper requirement as well, since that is more clinically realistic about the interpretation of the test results.

alt

Automatic QC Selection by EZ Rules 3 results in a recommendation of 3s control limits with 2 controls. That's tighter than the previous scenario, but not much different. The QC procedure still provides essentialy no false rejection. When you examine the Sigma-metrics chart (below), you also see that error detection is about 95%.

alt

Finally, let's look at the scenario that incorporates the biologic variation and clinical decision interval, which makes use of clinical quality requirement:

alt

Automatic QC Selection by EZ Rules 3 results in a recommendation of 2.5s control limits with 2 controls. Again, the increasing components of variation results in a tighter budget, which means the resulting control rules must also be tighter. Nevertheless, the performance of the method still keeps the control procedure down to 2 controls and a single rule. If 2.5s is not a possible choice on the QC system, a simple multirule of 13s and 22s would also provide equivalent error detection and actually less false rejection.

alt

Conclusion: What quality requirement should you use? What QC should you implement?

Despite the confusion about quality requirements, the choices here are pretty simple. Given the excellent performance of the method, it's not so important what quality requirement you apply. You will end up with a control procedure that uses 2 controls, with limits somewhere between 2.5 and 3.5s.

Source
TEa%
Sigma metric Recommended QC
CLIA PT
?
??
???
CAP PT 2007
15.0%
8.57
13.5s with N=2
Stricter standard
10.0%
5.45
13s with N=2
NACB 2007 Draft
?
??
???
Clinical Decision Interval
14.0% Dint
5.06
12.5s with N=2

Remember, the cobas imprecision data may be a bit optimistic - which may argue toward taking the most conservative approach in choosing QC. A better estimation of the imprecision may bring down the Sigma metrics, which in turn would mean that more QC (either tighter limits or more controls) is needed.

But, if this were your laboratory, and this was the information that you had, what would you choose? The 10% analytical quality requirement is a good compromise of all the possibilities, and the resulting recommendation of 3s limits and 2 controls is a very simple QC procedure to implement. Certainly, we would want to keep in mind the real evidence-based performance requirements - and the need to reduce imprecision and bias, if possible.