QC Design

Various expert groups have suggested different quality requirements for Glycated Hemoglobin. But what's happening on actual instruments? Taking an Abbott Architect c8000, we apply the different analytical quality requirements and evaluate its performance - and the suitability of the quality requirements.

• The Precision and Comparison data
• Calculate bias at the decision level
• Determine quality requirements at the critical decision level
• Calculate Sigma metrics
• Evaluation of performance by OPSpecs chart, Sigma-metrics graph, and EZ Rules 3
• Conclusion: what quality requirement to use?

January 2008

[Note: This QC application is an extension of the lesson From Method Validation to Six Sigma: Translating Method Performance Claims into Sigma Metrics. This article assumes that you have read that lesson first, and that you are also familiar with the concepts of QC Design, Method Validation, and Six Sigma. If you aren't, follow the links provided.]

A recent essay highlights the difficulties in defining the proper quality requirement for Glycohemoglobin. Despite years of earnest efforts, the 2007 draft NACB guidelines for Glycohemoglobin hasn't provided specific analytic quality requirements. In this application, we're going to take a real-world instrument, the Architect c8000, and impose upon it various proposed quality requirements.

The source of data for this application is a poster from the 2004 AACC conference: (C-56) Evaluation of the Seradyn MULTIGENT™ HbA1c Assay on the Abbott AEROSET® and the ARCHITECT® c8000™ Systems, E. Sasse¹, C. Bartell¹, J. Heffner¹, M. Durkin¹, J. Schneider¹, N. Bowman², R. Verdun², L. Padilla², B. Shull², L. Arabshahi²

¹VA Milwaukee, Milwaukee, WI; ²Seradyn, Inc., Indianapolis, IN

The Precision and Comparison data

According to the abstract, this evaluation study adhered closely to CLSI (formerly NCCLS) protocols, calculating total precision using EP5-A ( a 20 day precision study using two controls), and method comparison using EP9-A2 with the Tosoh G7 as the reference method, using 117 whole blood samples across the range of 4.1 to 14.7 % GHb. All that we need to do is select which estimates to use, supply the quality requirements and calculate the Sigma metrics.

Imprecision Estimates:

The Abbott study calculates within-run, between-run, between-day, and total imprecision estimates - a very robust approach.

Note that the critical medical decision level that the NACB discusses is 7.0% GHb. That's the new threshold between normal and diabetic. This decision level lies right between the levels of the two controls. Lacking specific data on performance of this method at the 7.0% GHb level, we're going to have to use our professional judgment.

So, for the purposes of this application, we're going to use an estimate of 2.0% imprecision at the critical decision level of 7.0% GHb.

Comparison of Methods Data: Test Method (c8000) vs. Tosoh G7 (reference method)

Remember that the correlation coefficient is not the key statistic here. The value of the correlation coefficient merely tells us that linear regression would be sufficient for these analytes (for those r values below 0.95, other forms of regression like Deming or Passing-Bablock are preferable, but in this case, are not available).

Calculate bias at the decision level

Now we take the comparison of methods data and set the equation to the level covered in the imprecision study. Solving those equations will give us a bias estimate. For this illustration, we're going to select the performance at the abnormal control level.

Here are the steps for calculating bias:

((slope*level) + YIntercept) - level) / level = % bias

((0.9982*7.0) +0.1414) - 7.0) / 7.0 = ((6.9874 +0.1414) - 7.0) / 7.0

(7.1288 - 7.0) / 7.0 = 0.1288 / 7.0 = 0.0184 * 100 = 1.84%

Determine the quality requirements at the critical decision level

Now that we have both bias and CV estimates, we are almost ready to calculate the Sigma metrics for these analytes. The last (but not least) thing we need is the quality requirement for each method. CLIA provides quality requirements for over 80 analytes, but in this case, it does not provide a quality requirement.

The details of these sources and quality requirements are discussed in Dr. Westgard's essay. The important thing to note here is that there is a pretty big difference between the requirements. Note also that the NACB guidelines do not specifically state any analytical quality requirement - at best, you can infer the the quality requirement based on their specifications for instrument performance. Finally, remember that while the Clinical Decision Interval quality requirement is almost the biggest number, using that number requires that the process take into account the known within-subject biological variation, which eats up a large amount of the error budget.

Calculate Sigma metrics

Now we have all the pieces in place.

Remember the equation for Sigma metric is (TEa - bias) / CV:

For Glycohemoglobin, using the CAP PT 2007 requirement (15.0 - 1.84) / 2.0 = 6.58

Because neither CLIA nor the NACB set a quantifiable quality requirement, it's not possible to calculate a Sigma metric for those scenarios. CLIA provides no information, NACB doesn't provide enough.

Given the current CAP PT requirement of 15%, this Architect method provides world class quality. Even if we tighten the quality requirement a bit (some people at CAP admit that 15% is too large), the Sigma metric of the Architect method is pretty good.

If we aim for "evidence-based" quality design, however, the demands on performance are more challenging. By using a clinical decision interval, we must also account for within-subject biological variation, so even though we've got a 14% quality requirement, individual biologic variation is chewing up 4.1% of that right off the top. The result is that we have a lower Sigma-metric than we would like.

Evaluation of Performance by OPSpecs chart, Sigma-metrics graph, and EZ Rules 3

Using EZ Rules 3, you can use QC Design to evaluation the impact of different quality requirements on testing performance. By using Automatic QC Selection, you can see the ideal rules and controls needed to provide appropriate QC.

Here's the OPSpecs chart for the 15% CAP requirement : 2 controls with 3.5s limits.

The Sigma-metric graph for the CAP 15% quality requirement (see below). Note that it provides us more specific details on the error detection and false rejection characteristics of the rule choice. Happily, the 3.5s limits with 2 controls will essentially eliminate all false rejections - you'll only get an out-of-control flag when this method is really out-of-control.

Now let's use the tighter 10% analytical quality requirement (see below). This makes a certain amount of sense, since CLIA states that glucose results should also meet at 10% quality requirement. Some people at CAP believe that 10% is the proper requirement as well. The rule choice here is still simple: 2.5s control limits with 4 controls.

Further analysis with EZ Rules 3 (see below) demonstrates that there are other rule choices that only use 2 controls that could provide nearly the same error detection capabilities. In the graph below, a "Westgard Rules" combination with 2 controls (note, though, that this must be monitored over 2 runs) is possible and would provide nearly equivalent error detection as the 12.5s rule with 4 controls.

Finally, let's look at a more evidence-based approach, using a clinical quality requirement. The rule choice here is significantly different: A "Westgard Rules" combination of rules with 4 controls.

The Sigma-metrics graph below shows that the recommended set of "Westgard Rules" will provide about 90% error detection.

Conclusion: What quality requirement should you use?

This application illustrates an unfortunate fact about the state of laboratory quality: as much as we believe there are high standards and performance, there is actually quite a bit of confusion. Some standards don't exist (CLIA), some standards don't provide enough information to be practical (NACB), and other standards may be too wide (CAP PT 2007). Finally, when you dive down into the clinical needs of the method, we may find that our methods - which may be world class by the current standards - are actually a bit less than we need for the best clinical care.

The Architect method is a good example of method performance. By current analytical standards, it's beyond world class. Even with a tightened requirement of 10%, it achieves a good Sigma-metric. But when measured against a clinical quality requirement, which is more "evidence-based," the method needs a little improvement. Still, given slight reductions in imprecision and bias, the Architect method could reach world class from a clinical perspective, too.

Undoubtedly, the regulatory and accreditation organizations believe they have done an adequate job in defining the quality required for glycohemoglobin, but they don't seem to communicate with each other. In other industries, standards like this would have to be reconciled. In the chaos of the healthcare laboratory, however, this problem gets put at the bottom of the list, if at all.

So if this were your laboratory, which standard would you choose? Probably the 10% analytical quality requirement is a good compromise of all the possibilities. Keep in mind the real evidence-based performance requirements - and the need to reduce imprecision and bias, if possible.