Tools, Technologies and Training for Healthcare Laboratories

Sigma-metrics of a chemistry analyzer in Ghana

A 2014 article in the Nigerian Medical Journal assessed the Sigma-metrics of a chemistry analyzer. The results raise important questions. 

Sigma-metrics of a chemistry analyzer in Ghana

March 2015
Sten Westgard, MS

[Note: This QC application is an extension of the lesson From Method Validation to Six Sigma: Translating Method Performance Claims into Sigma Metrics. This article assumes that you have read that lesson first, and that you are also familiar with the concepts of QC Design, Method Validation, and Six Sigma. If you aren't, follow the link provided.]

This analysis looks at an analyzer less well-known to the US and Europe: a Mindray BC 6800 automated hematology analyzer:

Application of sigma metrics for the assessment of quality control in clinical chemistry laboratory in Ghana: A pilot study. Afrifa J, Gyekye SA, Owiredu WKBA, Ephraim RKD, Essien-Baidoo S, Amoah S, Simpong DL, Arthur AR, Nigerian Medical Journal, Vol 56 (1): 54-58.

The Imprecision and Bias Data

For imprecision, "Commercial control samples... one with normal and one with pathological values were analyzerd each day over a 20 day period."

 For bias, the study authors compared the laboratory mean against the "Designated mean." "Designated means (Dm) for selected analyters in commercial controls were provided by the reagent manufacturer." Basicaly, comparing the observed lab mean against the control target value. While this is not the most rigorous way to calculate bias, it is the most practical.

Assay Level
CV% Bias%
Glucose  1 7.75% 22.50%
2 4.82% 2.34%
Cholesterol 1 7.21% 3.4%
2 4.86% 4.80%
Triglyceride 1 12.12% 0.17%
2 8.05% 1.47%
HDL-C 1 25.04% 6.70%
2 28.27% 13.10%
Urea 1 14.34% 0.83%
2 10.66% 17.35%
Creatinine 1 25.07% 12.58%
2 14.56% 11.18%
Total Protein 1 7.30% 10.74%
2 5.01% 1.46%
AST 1 10.76% 16.92%
2 6.92% 10.76%
ALP 1 17.49% 10.90%
2 12.41% 16.54%
Potassium 1 7.53% 0.58%
2 6.48% 5.07%
Chloride 1 2.52% 0.29%
2 2.51% 3.09%
Sodium 1 2.40% 1.15%
2 3.83% 4.40%

 

Determine Quality Requirements

Now that we have our imprecision and bias data, we're almost ready to calculate our Sigma-metrics. We're just missing one key thing: the analytical quality requirement. In this example, we're going to look at use the CLIA goals for allowable total error (TEa). Note that here, in the study, the specific levels were not listed, so the study authors defined a single TEa, even though in some cases CLIA sets a unit-based goal and therefore the quality requirement would be different at each level.

we'll also judge the individual specifications for desirable imprecision and bias from the Ricos database. Often these two sets of goals (CLIA and Ricos) are set against each other. This time we're going to see if they agree.

Here's a comparative table of the different quality goals available:

Assay Level
TEa% CV% Bias%
Glucose  1 10.0% 7.75% 22.50%
2 4.82% 2.34%
Cholesterol 1 10.0% 7.21% 3.4%
2 4.86% 4.80%
Triglyceride 1 25.0% 12.12% 0.17%
2 8.05% 1.47%
HDL-C 1 30.0% 25.04% 6.70%
2 28.27% 13.10%
Urea 1 9.0% 14.34% 0.83%
2 10.66% 17.35%
Creatinine 1 15.0% 25.07% 12.58%
2 14.56% 11.18%
Total Protein 1 10.0% 7.30% 10.74%
2 5.01% 1.46%
AST 1 20.0% 10.76% 16.92%
2 6.92% 10.76%
ALP 1 30.0% 17.49% 10.90%
2 12.41% 16.54%
Potassium 1 6.0% 7.53% 0.58%
2 6.48% 5.07%
Chloride 1 5.0% 2.52% 0.29%
2 2.51% 3.09%
Sodium 1 5.0% 2.40% 1.15%
2 3.83% 4.40%

Judging on imprecision and bias separately, using the Ricos desirable maximum allowable specifications, we'll highlight in green where performance is acceptable, and highlight in red where performance is unacceptable.

Assay Level
Desirable CV% CV% Desirable Bias% Bias%
Glucose  1 2.8% 7.75% 2.34% 22.50%
2 4.82% 2.34%
Cholesterol 1 2.98% 7.21% 4.1% 3.4%
2 4.86% 4.80%
Triglycerides 1 9.95% 12.12% 9.57% 0.17%
2 8.05% 1.47%
HDL-C 1 3.65% 25.04% 5.61% 6.70%
2 28.27% 13.10%
Urea 1 6.05% 14.34% 5.57% 0.83%
2 10.66% 17.35%
Creatinine 1 2.98% 25.07% 3.96% 12.58%
2 14.56% 11.18%
Total Protein 1 1.38% 7.30% 1.36% 10.74%
2 5.01% 1.46%
AST 1 6.15% 10.76% 6.54% 16.92%
2 6.92% 10.76%
ALP 1 3.23% 17.49% 6.72% 10.90%
2 12.41% 16.54%
Potassium 1 2.3% 7.53% 1.81% 0.58%
2 6.48% 5.07%
Chloride 1 0.6% 2.52% 0.5% 0.29%
2 2.51% 3.09%
Sodium 1 0.3% 2.40% 0.23% 1.15%
2 3.83% 4.40%

The Ricos specifications are not ambiguous when it comes to the performance of these methods.

Calculate Sigma metrics

Sigma-metrics takes both imprecision and bias into account in a single equation. We're going to calculate Sigma-metrics using the CLIA goals.

Remember the equation for Sigma metric is (TEa - bias%) / CV.

Example calculation: for Glucose, the performance at level 2, with a 10.0% quality requirement, given 4.82% imprecision and 2.34% bias:

(10.0 - 2.34) / 4.82 = 7.66 / 4.82  = 1.589

Similar to the judgement we made earlier with separate components, this Sigma-metric verdict on this glucose assay is harsh: we would consider it unacceptable.

So here's the table with all the Sigma-metrics using CLIA Goals:

 

Assay Level
TEa% CV% Bias% Sigma-metric
Glucose  1 10.0% 7.75% 22.50% n/a
2 4.82% 2.34% 1.59
Cholesterol 1 10.0% 7.21% 3.4% 0.92
2 4.86% 4.80% 1.07
Triglycerides 1 25.0% 12.12% 0.17% 2.05
2 8.05% 1.47% 2.92
HDL-C 1 30.0% 25.04% 6.70% 0.93
2 28.27% 13.10% 0.59
Urea 1 9.0% 14.34% 0.83% 0.57
2 10.66% 17.35% n/a
Creatinine 1 15.0% 25.07% 12.58% 0.10
2 14.56% 11.18% 0.26
Total Protein 1 10.0% 7.30% 10.74% n/a
2 5.01% 1.46% 1.70
AST 1 20.0% 10.76% 16.92% 0.16
2 6.92% 10.76% 1.33
ALP 1 30.0% 17.49% 10.90% 1.09
2 12.41% 16.54% 1.08
Potassium 1 6.0% 7.53% 0.58% 0.72
2 6.48% 5.07% 0.14
Chloride 1 5.0% 2.52% 0.29% 1.87
2 2.51% 3.09% 0.76
Sodium 1 5.0% 2.40% 1.15% 1.60
2 3.83% 4.40% 0.16

The Sigma-metrics are equally unambiguous. Everything is below three sigma, which is generally considered the minimum level of acceptable quality.

The Ricos goals imprecision and bias and the CLIA TEa goals are in agreement here. We could apply the Ricos goals for TEa if we wanted, but the results would be the same. At all levels, this instrument is not reaching an acceptable level of performance.

Summary of Performance by Sigma-metrics Method Decision Chart

We can make visual assessments of this performance using a Normalized Sigma-metric Method Decision Chart. Here we're using the CLIA goals:

2015 Chemistry analyzer

Here we can see that there are no methods in the bull'-eye, and almost all of them are completely missing the target.

Normally, at this point we also use the OPSpecs chart to try and determine what QC is necessary. But in this case, when methods perform this poorly, the QC prescription is the same: all the "Westgard Rules" that can be practical and affordable.

To be honest most of the methods are simply not controllable. That is, even with the full "Westgard Rules" we probably won't be catching errors when they first occur - it will take a while before we pick them up. We would also need to double the number of controls in use for these troublesome methods (from 2 to 4 or from 3 to 6), raising the expense of running this instrument. We may even need to increase the frequency of running controls because of the poor performance. 

Conclusion

The simplest conclusion might be that this instrument, the BS 120, is not fit for purpose. But that may be too easy a conclusion to reach. We probably need to ask more questions first.

For instance, is the performance here due to a bad instrument, bad controls, bad logistics, improper training? We need to figure that out. If the instrument is the cause of poor performance, at least we can get a better one. But if the logistical chain is inadequate (and reagents, calibrators, etc. are compromised by the time they reach the lab), then a more difficult challenge must be overcome. For the manufacturer, they need to ask: can we improve this performance, either by building a better instrument, a better logistics chain, better reagents and calibrators? If no, perhaps getting out of that market would be advisable. For regulators, here's a tough question: if this laboratory and hospital has an otherwise stellar quality system, should they still be accredited? Or should they stop reporting results? ISO 15189 is not specific about what level of quality is acceptable - should this laboratory be allowed to be ISO certified? Is there a level of quality that's acceptable in Ghana but not acceptable in other parts of the world? Do the different certifications and regulatory standards have different meanings in Africa than they do in Europe or America?

We don't know the answer to amost of these questions.

The authors conclude "Unsatisfactory sigma levels were achieved for all parameters using both control levels, this shows instability and low consistency of results being delivered. This is an indication of poor quality control based on the sigma model applied in this study. There is therefore the need for detailed assessment of the analytical procedures and the strengthening of the laboratory control systems in order to achieve effective six sigma levels for our laboratory."

For once, we're in complete agreement with the study authors. Too often, performance studies jump to the conclusion that the methods are acceptable, regardless of the actual performance. Here the authors are courageous enough to accept the uncomfortable verdict. There are many facilities around the world that can learn a valuable lesson from this lab in Ghana.