Tools, Technologies and Training for Healthcare Laboratories

Second Evaluation of Arkray HbA1c

A chance to get a second look at instrument performance. In 2011, CCLM had a study of performance of the Menarini/ARKRAY ADAMS A1c HA-8180V analyzer. In 2012, Lab Med Online has another study, which gives us another opportunity to see if the first findings hold up.

November 2012
Sten Westgard, MS

[Note: This QC application is an extension of the lesson From Method Validation to Six Sigma: Translating Method Performance Claims into Sigma Metrics. This article assumes that you have read that lesson first, and that you are also familiar with the concepts of QC Design, Method Validation, and Six Sigma. If you aren't, follow the link provided.]

In 2011, we calculated Sigma-metrics from a 2011 study on the Menarini/ARKRAY Adams A1c HA-8180V analyzer. This year we found another study of this instrument and wanted to see if the performance was the same. Here's the new study:

Evaluation of the Performance of ARKRAY ADAMS A1c HA-8180 HbA1c Analyzer. Jinsook Lim, Ji-Muung Kim, Sun Hoe Koo, Kye Chui Kwon, Lab Med Online, Vol. 2, No.3: 26-130, July 2012.

The Precision and Comparison Data

Analytical reproducibility was assessed using the CLIS EP-5 protocol. Between-run, Between-day and Total imprecision were calculated.
The estimate of interest for us is the total CV.

Instrument Level (%HbA1c)
Total CV%
HA-8180 5.31% 0.7%
9.53% 0.5%

Note that CLSI and current guidelines for Diabetes set the maximum allowable CV at 5%, with 3% as the desired performance. So performance is looking very good.

Bias / trueness was assessed using the CLSI EP-9 protocol. 150 samples were compared to the BioRad Variant II Turbo.

Instrument n
Regression Slope
Y-intercept Type
HA-8180 150 1.02 -0.03 NGSP units

We are going to use the regression equation to calculate the bias at both levels where imprecision was measured.  This helps to align our bias data with our imprecision data.

Just to review this calculation, here's a layman's explanation of the equation:

NewLevelNewMethod = (slope * OldLevelOldMethod ) +Y-intercept

Then we take the difference between the New and Old level, and convert that into a percentage value of the Old level.

Example Calculation: Given HA-8180V at level 5.7%, comparing to the NGSP assigned values:

NewLevelHA = (1.02 * 5.31) - 0.03

NewLevelHA = 5.42 - 0.03

NewLevelHA = 5.39

Difference = 5.39 - 5.31 = 0.08

Bias% = 0.08 / 5.31 = 1.5%

Now remember we have precision data for two levels - so we can use the regression equation and calculate bias for each level:

Instrument Level (NGSP)
Total CV% Bias%
HA-8180V 5.31% 0.7% 1.5%
9.53% 0.5% 1.7%

Determine Quality Requirements at the decision level

Now that we have our imprecision and bias data, we're almost ready to calculate our Sigma-metrics. We're just missing one key thing: the analytical quality requirement.

For HbA1c, the quality required by the test is a bit of a mystery. Despite the importance of this test, and the sheer volume of these tests being run, CLIA doesn't set a quality requirement.

Source
Quality Requirement
CLIA PT
No quality requirement given
Rilibak (Germany)
Target value ± 18%
CAP PT 2011 Target value ± 7%
Clinical Decision Interval
Target value ± 7.7%
(see below)
Ricos et al. biologic database, desirable specification
4.3%

The details of these sources and quality requirements are discussed in Dr. Westgard's essay. The important thing to note here is that there is a pretty big difference between the requirements.

If clinicians are now going to use the 6.5% level as a cutoff for diagnosis of diabetes, that level can be used to construct a clinical decision interval. Anecdotally, we have heard from multiple people that a change of greater  than 0.5% HbA1c is usually considered significant. That is, if you have a 6.5% HbA1c and it jumps to 7.0% or higher, your clinician will probably act upon that result. From that information, we can construct a clinical decision interval: 0.5/6.5 = about a 7.7% change. While this number is larger than the tightest goal (biologic database specification for desirable total error: 4.3%), if you use this type of quality requirement, you have to account for within-subject biologic variation.

A few years ago, we were talking about quality requirements of 10 to 12%, and clinical decision intervals of 14%. The requirements have gotten more demanding, and at the same time clinicians are tightening their interpretation of the test results. In this case, we're going to use several of the quality requirements and calculate a few different Sigma-metrics.

Calculate Sigma metrics

Now all the pieces are in place. Remember, this time we have two levels, so we're going to calculate two Sigma metrics. (And we'll make it more complicated by using multiple goals)

Remember the equation for Sigma metric is (TEa - bias) / CV.

Example calculation: for a 7% quality requirement, at the level of 5.7% HbA1c, given 0.7% imprecision, 0.6% bias:

(7 - 0.6) / 0.7 = 6.4 / 0.7 = 9.2

HA-8180V Sigma-metric Performance
Quality Requirement
Level Total CV% Bias% Sigma-metric
7% (CAP 2011) 5.31% 0.7% 1.5% 7.9
9.53% 0.5% 1.7% 10.6
4.3% (Ricos et al) 5.31% 0.7% 1.5% 4.0
9.53% 0.5% 1.7% 5.2
18% (Rilibak) 5.31% 0.7% 1.5% 23.6
9.53% 0.5% 1.7% 32.6
7.7% (Clinical Decision Interval)
(must account for 3.4% swsub)
5.31% 0.7% 1.5% 3.6
9.53% 0.5% 1.7% 4.3

 Recall that in industries outside healthcare, 3.0 Sigma is the minimum performance for routine use and 6.0 Sigma is considered world class quality. In this case, no matter what quality requirement we pick, the minimum level of quality is achieved. In fact, for all the analytical quality requirements, performance is at the very least excellent, if not world class.

Summary of Performance by Sigma-metrics Method Decision Chart and OPSpecs chart

Because we have different quality requirements, but we are still interested in assessing performance of the same method, we'll use normalized Method Decision Charts and Normalized OPSpecs charts. (We'll treat the case of the clinical decision interval separately, later down the page)

2012-ARKRAY-H8180-HbA1c Normalized Method Decision Chart

As our metrics indicated, this method hits the bull's-eye nearly every time, regardless of the analytical quality requirement.

Using QC Design tools (such as EZ Rules 3 ), we can express the method performance on an OPSpecs (Operating Specifications) chart and determine the changes we might make to our QC procedures for this method:

2012-ARKRAY-H8180-HbA1c Normalized OPSpecs Chart, for 2 controls

If we were to base our QC decisions on this data, for Rilibak and CAP requirements, we could use 2 controls and 3s or 3.5s control limits. The one issue is if we use Ricos et al biological quality requirements. On the low level, there is one operating point that would require us to consider using more robust rules, such as an extended "Westgard Rules" combination.

Now, turning our attention to the case of clinical decision intervals. In this case, if we're trying to ensure that a change in serial test results of 0.5% (a change from 6.5 to 7.0, for example) will reliably indicate a real change in the patient's clinical status (not just analytical noise), we need slightly better performance than what we've got.

If we calculate a Reference Change Value (based on Callum Fraser's equation), we use the imprecision data (0.7% and 0.4%) as well as biologic database estimate of within-subject biologic variation, 3.4%. That gives an RCV of about 9.6% at the level of 5.31% HbA1c (in other words, a 0.51 change) and an RCV of 9.5% at the level of 9.53% HbA1c (in other words, about a 0.9% change). So, for the case of a 6.5% HbA1c cutoff level, the RCV we would project would be about 0.6%. That is, you'd like to wait until you've seen a change from 6.5% to 7.1% before you have strong confidence that something has really changed in the patient. [Note: the figure to within-subject biologic variation is still being debated. While the Ricos et al. database specifies 3.4%, there are NGSP studies that claim the biologic variation is less than 1%. If the laboratory wants to accept the NGSP claim, that would have a big impact on the RCV.]

Using EZ Rules 3, again, we can take a deeper look at the clinical decision interval. If we wanted to be able to use an change interval of just 0.5% HbA1c, this is what performance looks like on a power function graph:

2012-ARKRAY-EZ Rules 3 screenshot

In order to QC a process to that specification, we'd need to use an extensive array of "Westgard Rules" and expand our control measurements from 2 to 4. That's either running additional controls or making additional measurements of the existing controls. Even then, we're not doing that well at detecting errors.

If we decide that the main problem is the bias - and assume we can in some way recalibrate the method, eliminate the bias, or use the test in such a way that the bias isn't important (i.e. baseline the patient on this method and only compare results from this method alone), we'd have a better assessment of the test. Here's the OPSpecs of that scenario:

2012-ARKRAY-Clinical Model, OPSpecs chart

In that case, a small set of "Westgard Rules" with N of 4 will provide adequate error detection for the method.

Conclusion

There's a lot of great news in this study. The HA-8180 analyzer can hit most of today's analytical targets in the bull's-eye. This is the second study we've covered and the performance closely matches the performance found in the other study. It's always heartening to see that one study wasn't a fluke - there really does seem to be a high level of quality in this method and instrument.

Because performance was so good, we again pushed even harder to see just how tight we can make our quality requirements, reference change values, and decision intervals using this method. When we use the most demanding clinical QC design models, and take into account the known within-subject biologic variation, performance isn't quite as great, but the instrument still delivers a good level of quality. Each laboratory needs to decide which quality requirement is relevant to their situation and patient population, and design their QC accordingly.