Tools, Technologies and Training for Healthcare Laboratories

Roche cobas c311

We look at a 2008 poster from the AACC/ASCLS convention and evaluate method validation data on the Roche cobas c311. See how the data can be converted into Sigma-metrics and ponder the implications.

July 2008

[Note: This QC application is an extension of the lesson From Method Validation to Six Sigma: Translating Method Performance Claims into Sigma Metrics. This article assumes that you have read that lesson first, and that you are also familiar with the concepts of QC Design, Method Validation, and Six Sigma. If you aren't, follow the link provided.]

This application comes from another set of posters from the 2008 AACC/ASCLS annual meeting. Every year this provides an excellent set of posters and abstracts on method performance.

In this example, we're going to take a look at a method validation study performed for the Roche coba c311 analyzer Clinical Chemistry System (B-22: S Welch, B McWhorter, K Stevens, M Winkelhake. Performance Evaluation of the Roche cobas c311 analyzer).

The Precision and Comparison data

According to the abstract, "precision was evaluated by running 2 levels of Bio-Rad Liquichek chemistry controls and human serum, plasm, urine and whole blood pools. Patient correlation samples were run vs. the [Roche] Modular Analytics system and the Bio-Rad Variant II Turbo (HbA1c)." We don't know the number of samples, number of days, or if these studies complied with other recommendations for method validation, such as the CLSI (formerly NCCLS) standards like EP10 or EP9. For this application, we will assume that the data presented is an accurate representation of routine performance.

While two levels of control were run, the abstract presents the results of one level only, which probably means that the authors consider that level to be the critical decision level. So all that we need to do now is supply the quality requirements and calculate the Sigma metrics.

Imprecision Estimates:

Both within-run and total precision estimates of %CV are provided. We use the total %CV estimate since that is more representative of performance over time.

Assay
Level
CV%
Na, mmol/L
121.43
0.8%
K, mmol/L
3.25
1.5%
Cl, mmol/L
82.67
1.3%
BUN, mg/dL
67.26
1.9%
Glucose, mg/dL
94.78
1.1%
CK, U/L
365.6
1.3%
Total Protein, g/dL
6.692
0.8%
HbA1c, %
6.089
1.6%

Comparison of Methods Data:

Assay
Slope
Y-Int
r
Na, mmol/L
0.971
5.02
0.9371
K, mmol/L
0.975
0.13
0.9956
Cl, mmol/L
0.989
-0.31
0.9766
BUN, mg/dL
1.015
-0.39
0.9980
Glucose, mg/dL
1.011
0.39
0.9997
CK, U/L
1.00
0
0.9997
Total Protein, g/dL
1.00
0.09
0.9925
HbA1c, %
0.974
0.4
0.9951

At this point, remember the following: the correlation coefficient is not the key statistic here. The values of the correlation coefficient merely tell us that linear regression is sufficient for these analytes (for those r values below 0.975, other forms of regression like Deming or Passing-Bablock are preferable). In this case, we are not told what kind of regression was performed, so we will assume that a more sophisticated technique was used for Sodium (with a correlation coefficient of 0.9371, ordinary linear regression is not sufficient).

Calculate bias at the critical decision level

Now we take the comparison of methods data and set those equations at one of the levels covered in the imprecision studies. Solving those equations will give us bias estimates. For this illustration, we have only one level, so that's where we will make our calculations.

Using Sodium as an example, let's see how to calculating bias:

((slope*level) + YIntercept) - level) / level = % bias

((0.971 * 121.43) + 5.02) - 121.43) / 121.43 = ((117.909 + 5.02) - 121.43 ) / 121.43

(122.929-121.43) / 121.43 = 1.499 / 121.43 = 0.0123 * 100 = 1.2%

Assay
Slope
Y-Int
level
Bias %
Na, mmol/L
0.971
5.02
121.43
1.23%
K, mmol/L
0.975
0.13
3.25
1.50%
Cl, mmol/L
0.989
-0.31
82.67
1.47%
BUN, mg/dL
1.015
-0.39
67.26
0.92%
Glucose, mg/dL
1.011
0.39
94.78
1.51%
CK, U/L
1.00
0
365.6
0%
Total Protein, g/dL
1.00
0.09
6.692
1.34%
HbA1c, %
0.974
0.4
6.089
3.97%

Determine the quality requirements at the critical decision level

Now that we have both bias and CV estimates, we are almost ready to calculate the Sigma metrics for these analytes. The last thing we need is the quality requirement for each method. CLIA provides most of the quality requirements we need, but in several cases, we need to transform those requirements into useful percentages.

Assay CLIA PT criterion notes
Final Quality Requirement
Na, mmol/L
Target value ± 4 mmol/L
At 121.43 mmol/L, (4/121.43) = 3.29%
3.29%
K, mmol/L
Target value ± 0.5 mmol/L
At 3.25 mmol/L, (0.5/3.25) = 15.38%
15.38%
Cl, mmol/L
Target value ± 5%
5.0%
BUN, mg/dL
Target value ± 2 mg/dL or ± 9% (greater)
At 67.26 mg/dL, (2/67.26) = 3.0%
9.0%
Glucose, mg/dL
Target value ± 6 mg/dL or ± 10% (greater)
At 94.78 mg/dL,
(6/94.78) = 6.3%
10.0%
CK, U/L
Target value ± 30%
30.0%
Total Protein, g/dL
Target value ± 10%
10.0%
HbA1c%
Target value ± 12%*
12.0%

Note that some of these requirements are extremely tight. Sodium and Potassium have a fixed target values, regardless of the level of the test, which cran creates very small "windows of opportunity." Chloride's requirement is also quite small. With HbA1c, remember that CLIA provides no official criterion for performance; various standards exist, ranging from 10 to 15%, with a middle figure chosen here.

Calculate Sigma metrics

Now we have all the pieces in place.

Remember the equation for Sigma metric is (TEa - bias) / CV:

For Sodium, (3.29 - 1.23) / 0.8 = 2.58

Assay
CV%
Bias % TEa% Sigma metric
Na, mmol/L
0.8%
3.29%
1.23%
2.58
K, mmol/L
1.5%
15.38%
1.50%
9.26
Cl, mmol/L
1.3%
5.0%
1.47%
2.71
BUN, mg/dL
1.9%
9.0%
0.92%
4.25
Glucose, mg/dL
1.1%
10.0%
1.51%
7.72
CK, U/L
1.3%
0%
30.0%
23.08
Total Protein, g/dL
0.8%
1.34%
10.0%
10.82
HbA1c, %
1.6%
3.97%
12.0%
5.02

While there are some troublesome methods (those with the tightest requirements fared worst), there is a fair amount of good news. 4 of the 8 methods performed at Six Sigma and higher.

Summary of Performance by Normalized Sigma-metrics chart and Normalized OPSpecs chart

Here's a graphic depiction of these analytes, normalized so they can be presented together on a single Sigma-metrics chart.

alt

One of the things this graph can tell us quickly is whether or not we can easily fix some of the problem methods. For example, if we could eliminate bias on Chloride or Sodium, would that be enough to improve performance to world class? You can assess that by drawing a straight line down from to the x-axis (simulating the elimination of bias but maintaining the current imprecision estimate). For Chloride and Sodium, unfortunately, reducing bias isn't enough. Imprecision must also be substantially reduced to reach desirable performance. On the other hand, if bias could be reduced for HbA1c, it is very possible it could reach world class performance.

Here's another view of the data, this time using a Normalized OPSpecs chart. This chart allows us to see what type of QC procedures we can use with these methods.

alt

As you can see, the operating points of the four world class methods are on "solid ground." For many of these analytes, there is a great deal of "wiggle room" (allowable variation) available without jeopardizing world class performance.

Note also that this Normalized OPSpecs chart displays some unusual rules (see the key at right). For the four world class methods, even one control with 3 SD limits would provide more than sufficient error detection of medically important errors. This is one of those (somewhat rare) cases where CLIA minimums are over-controlling those methods. Since CLIA requires at least 2 controls per run, the other solutions displayed here show control limits set at 3.5 and 4 times the standard deviation. Even the two "problem" methods, Urea and Creatine kinase, can still be adequately controlled with 2 controls and limits set at 3s.

For Sodium and Chloride, unfortunately, no QC procedure can provide adequate error detection. Something else needs to be done to bring these methods into control.

All of these QC procedures would essentially eliminate false rejection problems. If you set your limits that wide, you will only get a flag when there is a real problem. But remember, once you get that flag, you must do something about it (trouble-shoot the method), not just repeat the control.

Using EZ Rules 3, a QC Design program, you can experiment with different combinations of QC rules, numbers of controls, etc. One possible solution can be seen below, in this (slightly compressed) screen shot:

alt

You can see that the operating point is finally below the lines of the control procedures, and the rule selected is a 12.5s rule with only two controls. But the important caveats here are that only 50% AQA is being achieved (that is, instead of the 90% AQA ideal, which would detect almost all errors within the first run of the occurrence, this solution would take on average 2 runs to detect the error). More importantly, if you look toward the bottom left of the screen, you'll see tha the Replicate samples analyzed, nsamp has been set to 2. This means that you are running the sodium in duplicate. To work with this sodium method as it currently performs is going to require significant effort.

Conclusions

While this evaluation doesn't provide an unqualified success for the instrument, there are world class methods here. While sodium and chloride are presenting real challenges, that is not that unusual. Only a few instruments out there can work with sodium and chloride without performance problems - the fault might not be instrument performance but quality requirements that are too tight. Since only minimum QC is required for potassium, glucose, total protein, and creatinine kinase, extra resources can be devoted to the problem methods.