Tools, Technologies and Training for Healthcare Laboratories

Menarini/ARKRAY ADAMS A1c HA-8180V

A recent evaluation in CCLM of the Menarini/ARKRAY ADAMS A1c HA-8180V analyzer gives us another opportunity to see if current performance on the market can meet the tightening requirements for HbA1c.

June 2011
Sten Westgard, MS

[Note: This QC application is an extension of the lesson From Method Validation to Six Sigma: Translating Method Performance Claims into Sigma Metrics. This article assumes that you have read that lesson first, and that you are also familiar with the concepts of QC Design, Method Validation, and Six Sigma. If you aren't, follow the link provided.]

This application looks at a new study :

Evaluation of the Menarini/ ARKRAY ADAMS A1c HA-8180V analyser for HbA1c.Cas Weykamp, Erna Lenters-Westra, Hans van der Vuurst, Robbert Slingerland, Carla Siebelder, and Willeke Visser-Dekkers, Clin Chem Lab Med 2011; April 49(4):647-51.

The Precision and Comparison Data

Analytical reproducibility was assessed using the CLIS EP-5 protocol. On 20 working days, two levels were assayed in duplicate twice a day in analytical runs.Within-run, Between-run, Between-day and Total imprecision were calculated. The estimate of interest is the total CV.

Instrument Level (%HbA1c)
Total CV%
HA-8180V
5.7% 0.7%
11.2% 0.4%

Note that CLSI and current guidelines for Diabetes set the minimum precision performance at 5%, with 3% as the desired performance. So performance is looking very good.

Bias / trueness was assessed using the CLSI EP-9 protocol. "We used a set of 40 samples to which IFCC targets, as well as derived NGSP targets, have been assigned by two IFCC network labs. This allows evaluation of the trueness in absolute terms with respect to the IFCC and NGSP reference systems." Slopes and intercepts were calculated according to Deming and standard regression. This a very good comparison study, setting up a comparison against samples that have a "true" value assigned by reference-quality methods.

Instrument n Ordinary
Regression Slope
Y-intercept Type
HA-8180V 40 1.026
-0.181 NGSP units

The study actually calculated the specific bias at three different levels, but we are going to use the regression equation to calculate the bias for our purposes. In some cases, our calculated bias will exceed the measured bias from the study - we're assuming a worst-case scenario for bias.

To get a real idea of performance, we need to align our bias data with our imprecision data. The easiest thing to do is to calculate the bias at the levels where the imprecision was calculated, using the regression equation.

Just to review this calculation, here's a layman's explanation of the equation:

NewLevelNewMethod = (slope * OldLevelOldMethod ) +Y-intercept

Then we take the difference between the New and Old level, and convert that into a percentage value of the Old level.

Example Calculation: Given HA-8180V at level 5.7%, comparing to the NGSP assigned values:

NewLevelHA = (1.026 * 5.7) - 0.181

NewLevelHA = 5.85 - 0.181

NewLevelHA = 5.67

Difference = 5.67 - 5.7 = 0.03

Bias% = 0.03 / 5.7 = 0.6%

Now remember we have precision data for two levels - so we can use the regression equation and calculate bias for each level:

Instrument Level (NGSP)
Total CV% Bias%
HA-8180V
5.7% 0.7% 0.6%
11.2% 0.4% 1.0%

Determine Quality Requirements at the decision level

Now that we have our imprecision and bias data, we're almost ready to calculate our Sigma-metrics. We're just missing one key thing: the analytical quality requirement.

For HbA1c, the quality required by the test is a bit of a mystery. Despite the importance of this test, and the sheer volume of these tests being run, CLIA doesn't set a quality requirement.

Source
Quality Requirement
CLIA PT
No quality requirement given
Rilibak (Germany)
Target value ± 18%
CAP PT 2011 Target value ± 7%
Clinical Decision Interval
Target value ± 7.7%
(see below)
Ricos et al. biologic database, desirable specification
4.3%

The details of these sources and quality requirements are discussed in Dr. Westgard's essay. The important thing to note here is that there is a pretty big difference between the requirements.

If clinicians are now going to use the 6.5% level as a cutoff for diagnosis of diabetes, that level can be used to construct a clinical decision interval. Anecdotally, we have heard from multiple people that a change of greater  than 0.5% HbA1c is usually considered significant. That is, if you have a 6.5% HbA1c and it jumps to 7.0% or higher, your clinician will probably act upon that result. From that information, we can construct a clinical decision interval: 0.5/6.5 = about a 7.7% change. While this number is larger than the tightest goal (Ricos et al's 4.3%), if you use this type of quality requirement, you have to account for within-subject biologic variation.

A few years ago, we were talking about quality requirements of 10 to 12%, and clinical decision intervals of 14%. The requirements have gotten more demanding, and at the same time clinicians are tightening their interpretation of the test results. In this case, we're going to use several of the quality requirements and calculate a few different Sigma-metrics.

Calculate Sigma metrics

Now all the pieces are in place. Remember, this time we have two levels, so we're going to calculate two Sigma metrics. (And we'll make it more complicated by using multiple goals)

Remember the equation for Sigma metric is (TEa - bias) / CV.

Example calculation: for a 7% quality requirement, at the level of 5.7% HbA1c, given 0.7% imprecision, 0.6% bias:

(7 - 0.6) / 0.7 = 6.4 / 0.7 = 9.2

HA-8180V Sigma-metric Performance
Quality Requirement
Level Total CV% Bias% Sigma-metric
7% (CAP 2011)
5.1% 0.7% 0.6% 9.2
11.2% 0.4% 1.0% 15.0
4.3% (Ricos et al)
5.1% 0.7% 0.6% 5.3
11.2% 0.4% 1.0% 8.3
18% (Rilibak)
5.1% 0.7% 0.6% 24.9
11.2% 0.4% 1.0% 42.5
7.7% (Clinical Decision Interval)
(must account for 3.4% swsub)
5.1% 0.7% 0.6% 3.6
11.2% 0.4% 1.0% 4.3

Recall that in industries outside healthcare, 3.0 Sigma is the minimum performance for routine use and 6.0 Sigma is considered world class quality. In this case, no matter what quality requirement we pick, the minimum level of quality is achieved. In fact, for all the analytical quality requirements, performance is at the very least excellent, if not world class.

Summary of Performance by Sigma-metrics Method Decision Chart and OPSpecs chart

Because we have different quality requirements, but we are still interested in assessing performance of the same method, we'll use normalized Method Decision Charts and Normalized OPSpecs charts. (We'll treat the case of the clinical decision interval separately, later down the page)

2011-ARKRAY-NMedx

As our metrics indicated, this method hits the bull's-eye nearly every time, regardless of the analytical quality requirement.

Using EZ Rules 3, we can express the method performance on an OPSpecs (Operating Specifications) chart and determine the changes we might make to our QC procedures for this method:

2011-ARKRAY-n2NOPs

If we were to base our QC decisions on this data, we could use 2 controls and 3s or 3.5s control limits. The one issue is if we use Ricos et al biological quality requirements. On the low level, there is one operating point that would require us to use 2 controls and 3s limits (we couldn't widen them to 3.5s).

Now, turning our attention to the case of clinical decision intervals. In this case, if we're trying to ensure that a change in serial test results of 0.5% (a change from 6.5 to 7.0, for example) will reliably indicate a real change in the patient's clinical status (not just analytical noise), we need slightly better performance than what we've got.

If we calculate a Reference Change Value (based on Callum Fraser's equation), we use the imprecision data (0.7% and 0.4%) as well as Ricos et al's estimate of within-subject biologic variation, 3.4%. That gives an RCV of 9.6% at the level of 5.7% HbA1c and an RCV of 9.5% at the level of 11.2% HbA1c. So, for the case of a 6.5% HbA1c cutoff level, you'd like to see a change from 6.5% to 7.1% before you have confidence that something has really happened in the patient.

Using EZ Rules 3, again, we can take a deeper look at the clinical decision interval. If we wanted to be able to use an change interval of just 0.5% HbA1c, this is what performance looks like on a power function graph:

2011-ARKRAY-Dint7

In order to QC a process to that specification, we'd need to use an extensive array of "Westgard Rules" and expand our control measurements from 2 to 4. That's either running additional controls or making additional measurements of the existing controls.

If we allow our decision interval to increase slightly, it gets easier to QC the method:

2011-ARKRAY-Dint8

Finally, if we want to aim for the easiest QC procedure, we need widen that decision interval further:

2011-ARKRAY-Dint91

If we use a decision interval of 9.15%, we could content ourselves with 3s control limits, 2 control measurements, and no expected false rejection. But that would mean we would need to wait until a value of 6.5% HbA1c has increased to about 7.1% HbA1c. In other words, if the change interval is around 0.6% HbA1c, we can easily QC this method.

Conclusion

There's a lot of great news in this study. The HA-8180V analyzer can hit just about any of today's analytical targets in the bull's eye. In fact, because performance was so good, we pushed even harder to see just how tight we can make our quality requirements, reference change values, and decision intervals using this method. Even when we use the more demanding clinical QC design model, and take into account the known within-subject biologic variation, this instrument still delivers a high level of quality.