Tools, Technologies and Training for Healthcare Laboratories

Olympus AU2700 plus

In a 2010 study, the performance of an Olympus AU2700 plus was evaluated, using allowable maximum bias and imprecision values from the Ricos et al biologic variation database. Which got us to thinking, what would the metrics be if the Ricos specifications for allowable total error were used? For that matter, what if Rilibak or CLIA quality requirements were used?

March 2011
Sten Westgard, MS

[Note: This QC application is an extension of the lesson From Method Validation to Six Sigma: Translating Method Performance Claims into Sigma Metrics. This article assumes that you have read that lesson first, and that you are also familiar with the concepts of QC Design, Method Validation, and Six Sigma. If you aren't, follow the link provided.]

This application looks at a paper from a 2010 issue Biochemia Medica which examined the performance of the Olympus AU2700 plus:

Analytical evaluation of the clinical chemistry analyzer Olympus AU2700 plus, Jasna Juricek, Lovorka Derek, Adriana Unic, Tihana Serdar, Domagoj Marijancevic, Marcela Zickovic, Zeljko Romic, Biomedica Medica 2010;20(3):334-40.

The Imprecision and Bias Data

Analytical imprecision was assessed both within-run and between-run. The between-run estimates are of most interest here, since that estimate gives us the best picture of routine performance.

"Between-run imprecision was determined measuring the concentration of analytes in the control sera of different concentration ranges... and pool serum in duplicate during the period of 30 days."

Two estimates of bias were made, one using the values from the imprecision study, and another estimate from the linear regression (comparison) study. The comparison study was conducted with 50 sera samples on the AU2700 plus and the AU2700 instruments. Thus, we expect since the instruments are similar, this will give an optimistic estimate for bias. However, the advantage of the comparison study is that they determined a slope and y-intercept for each method, so we can use the regression equation to calculate bias at the decision levels where imprecision was estimated. In other words, we can focus our estimates of errors on specific decision levels.

Method Level Between-day CV% slope y-intercept Bias%
Glucose, mmol/L
5.3 2.29% 1.01 -0.146 1.75%
12.5
1.36% 0.17%
Creatinine, umol/L
116 2.37% 1.0 -2 1.72%
452 2.09% 0.44%
Urate, umol/L
298 1.82% 1 -2 0.67%
540 1.26% 0.37%
Bilirubin, umol/L
25.6 2.69% 1.027 0.047 2.88%
115 2.52% 2.74%
Cholesterol, mmol/L
4.08
1.77% 0.994 -0.06 2.07%
7.84 1.36% 1.37%
Triglycerides, mmol/L
1.87 2.69% 0.968 0.054 0.31%
3.73 1.81% 1.75%
AST, U/L
44 1.96% 1 0 0%
128 1.51% 0%
ALT, U/L
42 2.09% 1 1 2.38%
123 1.81% 0.81%
LD, U/L
151 3.39% 0.989 2.259 0.40%
537 1.59% 0.68%
Creatine Kinase, U/L
150 2.11% 1.022 0.064 2.24%
387 1.57% 2.22%
ALP, U/L
112 3.77% 1 1 0.89%
492 1.79% 0.20%
Potassium, mmol/L
3.8 4.8% 1 0 0%
6.4 5.6% 0%
Sodium, mmol/L
118 1.46% 1 1 0.85%
149 1.34% 0.67%
Chloride, mmol/L
90 1.5% 1 1 1.11%
112 0.97% 0.89%

A lot of numbers, right? We have two levels of control materials, so we have imprecision estimates for both of those levels. Using the regression equation, we can estimate the bias at each of those levels.

Just to review how we calculate the bias, here's a layman's explanation of the regression equation:

NewLevelNewMethod = ( )slope * OldLevelOldMethod ) +Y-intercept

Then we take the difference between the New and Old level, and convert that into a percentage value of the Old level.

Example Calculation: Given AU2700 Plus Glucose level 1 at 5.3 mmol/L, comparing to the AU2700 (no plus) method:

NewLevelPlus = (1.01 * 5.3) - 0.146

NewLevelPlus = 5.353 - 0.146

NewLevelPlus = 5.207

Difference between Plus and old method = 5.207 - 5.3 = 0.093

Bias% at 5.3 mmol/L = 0.093 / 5.3 = 1.75%

 

Looking at the raw numbers, you may find it difficult to judge the method performance. From experience, you might be able to tell when a particular method CV is high or low. But the numbers themselves don't tell the story.

If we want an objective assessment, we can add quality requirements and use those to calculate the Sigma-metric.

Determine Quality Requirements at the decision level

Now that we have our imprecision and bias data, we're almost ready to calculate our Sigma-metrics. But we're missing one key thing: the analytical quality requirement.

In this case, the study authors were using the estimates of maximum allowed imprecision and maximum allowed bias from the Ricos et al Biologic Within-Subject Variation Database. That database, however, also provides specifications for desirable allowable total error.

The problem with using 2 estimates for allowable imprecision and bias is that, like using 2 credit cards tied to one account, you have to be careful how you allocate the spending. You can't push the limit on both maximum imprecision and maximum bias - that will in turn blow past your credit limit (in this case, your allowable total error).

Since this study started by looking at Ricos biologic quality requirements we will use that as our source for total allowable error. However, as you will soon see, the Ricos database is very demanding for some analytes. So we are going to add two other sets of quality requirements as well: the (relatively new) Rilibak specifications for allowable interlaboratory variation, as well as the (much older) CLIA criteria for proficiency testing.

Note: by using three sets of quality requirements, we're going to come up with three different Sigma-metrics for each level. So this table is going to be a bit complicated. First, we'll list the different quality requirements, with a red highlight for the strictest specification:

Method Ricos TEa%
Rilibak TEa% CLIA TEa%
Glucose, mmol/L
6.9 15 10
Creatinine, umol/L
8.2 20 15
Urate, umol/L
12.4 13 17
Bilirubin, umol/L
31.1 22 20
Cholesterol, mmol/L
8.5 13 10
Triglycerides, mmol/L
27.9 16 25
AST, U/L
15.2 21 20
ALT, U/L
32.1 21 20
LD, U/L
11.4 18 20
Creatine Kinase, U/L
30.3 20 30
ALP, U/L
11.7 21 30
Potassium, mmol/L
5.8 8 13.16
7.81*
Sodium, mmol/L
0.9 5 3.39*
2.68*
Chloride, mmol/L
1.5 8 5
5

* For Potassium (+/- 0.5 mmol/L) and Sodium (+/- 4 mmol/L), CLIA sets a units-based quality requirement, therefore the specific quality requirement varies depending on the level where the performance is being assessed.

Notice that some of these quality requirements vary dramatically - Ricos sets a goal of less than 1% for Sodium while Rilibak allows 5%, for example. For 10 out of the 14 tests here, Ricos quality requirements are the most demanding. In the other cases, Rilibak and CLIA each have two tests where they set the most demanding requirement.

These discrepancies in goals will become clear when we move to Sigma-metric calculations.

Calculate Sigma metrics

Now the pieces are in place. Remember, this time we have two comparison methods, so we're going to calculate two sets of Sigma metrics - one using Central X for bias and one using Central Y.

Remember the equation for Sigma metric is (TEa - bias) / CV.

Example calculation: for glucose for the Ricos 6.9% quality requirement, at the level of 5.3 mmol/L, 2.29% imprecision, 1.75% bias:

(6.9 - 1.75) / 2.29 = 5.15 / 2.29 = 2.25

Now, with this same example, use the Rilibak quality requirement of 15% quality requirement, at the level of 5.3 mmol/L, 2.29% imprecision, 1.75% bias:

(15 - 1.75) / 2.29 = 13.25 / 2.29 = 5.78

 

And finally, with the CLIA quality requirement of 10% quality requirement, at the level of 5.3 mmol/L, 2.29% imprecision, 1.75% bias:

(10 - 1.75) / 2.29 = 8.25 / 2.29 = 3.60

Recall that in industries outside healthcare, 3.0 Sigma is the minimum performance for routine use. 6.0 Sigma and higher is considered world class performance. In the table below, we'll highlight the <3.0 Sigma in red, and highlight the >6.0 Sigma in green.

Also note that with the Central laboratory methods, if you calculated the Sigma metrics (assuming no bias), you'd get Sigma metrics of 5 and higher. The central laboratory methods meet the needs of patients, but the POC devices are not doing so well.

Method Level Btw-dy CV% Ricos Sigma Rilibak Sigma CLIA Sigma
Glucose, mmol/L
5.3 2.29% 2.25 5.78 3.60
12.5
1.36% 4.95 10.91 7.23
Creatinine, umol/L
116 2.37% 2.73 7.71 5.60
452 2.09% 3.71 9.36 6.97
Urate, umol/L
298 1.82% 6.44 6.77 8.97
540 1.26% 9.55 10.02 13.20
Bilirubin, umol/L
25.6 2.69% 10.49 7.11 6.36
115 2.52% 11.25 7.64 6.85
Cholesterol, mmol/L
4.08
1.77% 3.63 6.17 4.48
7.84 1.36% 5.25 8.55 6.35
Triglycerides, mmol/L
1.87 2.69% 14.29 8.13 12.79
3.73 1.81% 15.20 8.28 13.52
AST, U/L
44 1.96% 7.76 10.71 10.20
128 1.51% 10.07 13.91 13.25
ALT, U/L
42 2.09% 14.22 8.91 8.43
123 1.81% 17.29 11.15 10.60
LD, U/L
151 3.39% 3.25 5.19 5.78
537 1.59% 6.74 10.89 12.15
Creatine Kinase, U/L
150 2.11% 13.30 8.42 13.16
387 1.57% 17.89 11.33 17.70
ALP, U/L
112 3.77% 2.87 5.33 7.72
492 1.79% 6.42 11.62 16.65
Potassium, mmol/L
3.8 4.8% 1.21 1.67 2.74
6.4 5.6% 1.04 1.43 1.40
Sodium, mmol/L
118 1.46% 0.04 2.84 1.74
149 1.34% 0.17 3.23 1.50
Chloride, mmol/L
90 1.5% 0.26 4.59 2.59
112 0.97% 0.63 7.33 4.23

Again, the numbers can be overwhelming here. There are some cases where it's very easy to see, no matter the quality requirement, assay performance is world class (AST, ALT, for example). There are other methods where performance is not quite so good (Potassium, Sodium). And then there are a few methods where, according to the choice of quality requirement, the verdict is either good or bad (Chloride, Glucose)

Summary of Performance by Sigma-metrics Method Decision Chart

If the numbers are too much to digest, we can put this into a graphic format with a Six Sigma Method Decision Chart. Here's the chart for Ricos specifications for allowable total error

2011-OlympusAU2700-Ricos-NormMedx

Here is the chart for Rilibak quality specifications:

2011-OlympusRilibak-NormMedx

Finally, here is the chart for CLIA quality specifications:

2011-OlympusAU2700-CLIA-NormMedx

Conclusion

Interestingly, despite the wide differences in quality requirements, which in turn create different Sigma-metrics, the final verdict on these assays would not change very much. Whether or not you use Ricos, Rilibak, or CLIA quality requirements, most of the assays on this instrument are world class.

Unfortunately, where the assays are showing less than wonderful performance are some of the most important methods (Sodium, Potassium, Chloride, etc.). The disparity between quality requirements makes decisions about the performance of these assays a further challenge.