QC APPLICATION:
A ROCHE MODULAR ANALYZER
|
|
[Note: This QC application is an extension of the lesson From Method Validation to Six Sigma: Translating Method Performance Claims into Sigma Metrics. This article assumes that you have read that lesson first, and that you are also familiar with the concepts of QC Design, Method Validation, and Six Sigma. If you aren't, follow the links provided.]
Recently, one of our Westgard Workshop attendees provided us with data from one of their chemistry analyzers. In what may be perceived as a departure from the usual material at Westgard Web, this data has some very good news.
This data was supplied from a Roche Modular system, running 2 levels of control. The imprecision figures are pulled directly from the control data in the system. Unfortunately, no bias data was provided. But we will try to make use of this data to estimate bias issues.
The TEa column lists the Total Allowable Error, as specified by the CLIA proficiency testing criteria.
Test Observed
ImprecisionTEa Sigma metric Bicarbonate Direct Bilirubin Cholesterol Glucose Total Bilirubin Total Protein BUN Phosphorus Lipase Creatinine LDH SGOT Albumin HDL-Cholesterol Calcium SGPT Uric Acid Amylase Alkaline Phosphatase Triglycerides CPK
Obviously, the numbers here are not only good, they're great. Of course, we have to keep in mind that these Sigma metrics are calculated without any bias data. Therefore, they are artificially high. Nevertheless, only 2 of the 21 methods are unacceptable at this point.
Now that we have the optimistic numbers, let's try to get more realistic. As noted before, we do not have bias figures for this instrument. However, we can make estimates of the bias that could be tolerated if defined Sigma metrics are assumed. From a QC Design perspective, there is not difference between a Sigma metric of 6.0 and a higher Sigma metric of 22.0. We can illustrate this by showing some of these metrics by OPSpecs charts [Note: the source of these graphs and recommendations is the EZ Rules(r) 3 QC Design program]:
Here's a 6.6 Sigma method:
Now here's a 9.6 Sigma method:
And now here's a 22.1 Sigma method:
And what is the result of all these metrics? a recommendation for limits set at 3.5s with two controls per run, rejecting the run anytime a value is outside those limits. Beyond 6.0 Sigma, there is no practical difference in the control rule recommendation.
However, there is a practical effect of these metrics if they are only taking CV into account. A method with a Sigma metric of 22.1 based on CV alone will have a higher tolerable bias than a method with a Sigma metric of 6.0. For example, The CPK method above could tolerate up to 23.2% bias. Clearly, the higher Sigma metrics give you more safety margin while still achieving world class quality.
How do we calculate this tolerable bias? We manipulate the Sigma metric equation, plugging in 6.0 for the metric, and solve for bias:
Sigma = (TEa – bias)/CV.
6.0* CV = TEa -bias
bias + (6.0 *CV) = TEa
bias = TEa - (6.0*CV)
If we perform these calculations on all the methods, here are the results, not only for 6 Sigma but 5 Sigma as well:
Test Observed
ImprecisionTEa Sigma metric Tolerable Bias 5 Sigma 6 Sigma Bicarbonate Direct Bilirubin Cholesterol Glucose Total Bilirubin Total Protein BUN Phosphorus Lipase Creatinine LDH SGOT Albumin HDL-Cholesterol Calcium SGPT Uric Acid Amylase Alkaline Phosphatase Triglycerides CPK For those metrics that achieve 6 Sigma, the QC Design recommendation is a 13.5s control rule with 2 controls.
For those metrics that achieve 5 Sigma, the QC Design recommendation is a 12.5s control rule with 2 controls, or you could use a mini-"Westgard Rules" combination of 13s/22s with 2 controls.
What message can you take away from this data? That the performance of most of these methods is so good that there are rather large margins of safety for bias, despite not knowing what the bias is. This lab can still achieve world class quality with method bias issues.
Still the question remains, just what is the bias? Are the real world figures within the calculated tolerable bias figures?
Although the data provided did not include bias figures, it is possible to create a surrogate profile for bias. Given one piece of additional information about this laboratory - that it participates in the New York Wadsworth Proficiency Testing program, we can go to that website and access that data. While we cannot get individual data points on bias, we can retrieve the average bias of the instrument when compared to the entire proficiency testing group.Between 29 and 39 Roche Modular instruments participated in a recent (May 2006) testing event, which is close the same time this data was gathered, and we calculated the difference of the Roche Modular group mean from the entire testing group mean. That difference (expressed as a % of the group mean) is going to be our surrogate for bias.
Let us stress: this is not a good practice for a individual laboratory to do. We are mixing data here that really should not be mixed if we were going to make serious decisions about an individual laboratory. But this approach is as close as we can get to bias data for this study.
Below follows the calculations for bias (n/c stands for a method where no PT data was available):
Test Observed
ImprecisionTEa Sigma metric Tolerable Bias Surrogate 5 Sigma 6 Sigma Bias Bicarbonate n/c Direct Bilirubin n/c Cholesterol 1.87 Glucose 0.42 Total Bilirubin 11.8 Total Protein 4.23 BUN n/c Phosphorus 0.37 Lipase n/c Creatinine 1.97 LDH 13.87 SGOT n/c Albumin 1.53 HDL-Cholesterol n/c Calcium 2.9 SGPT n/c Uric Acid 2.16 Amylase 10.9 Alkaline Phosphatase 1.64 Triglycerides 1.3 CPK n/c Given the surrogate bias, we find that most of the analytes can be expected to continue to perform at the Six Sigma level. Where the bias number is italicized, that indicates a bias higher than the calculated tolerances, which would indicate a lower than 5 Sigma method, requiring more rules, possibly even more controls. In the eleven instances where methods achieved Six Sigma, there were only three cases where the surrogate bias knocked them down below Six Sigma.
This is an instance where there is a lot of good news. For this instrument, even in the absence of reliable bias data, it can be estimated that most of the analytes can use wide limits like 3.5s with only two controls and still tolerate a method bias.
For a laboratory that had this instrument, it would be advisable to minimize the QC efforts on most of the analytes, concentrating on just the two or three methods at the top of the table that have proven to be troublesome. Indeed, the performance of those methods were found to be outside the manufacturer's specifications (i.e. somehow broken) and this instrument was scheduled for replacement.
