Sigma Metric Analysis

Do major instruments perform the same all over the world? Or does reliability vary from region to region? We put this to the test, with data from a major laboratory in Ethiopia.

Sigma metric analysis of a Roche Cobas 6000 in Ethiopia

June 2021
Sten Westgard, MS

The key reference works that provide the foundation for this analysis are Basic Method Validation, 4th Edition and Six Sigma QC Design and Control, 2nd Edition.

We know about the gap between the developed and developing world healthcare, in many dimensions. But what's the difference on a Sigma metric level? If we take the same high performance, top of the line instrument that achieve great performance in an ideal laboratory, and we bring that to a more challenging environment, what happens?

Recently a major hospital in Ethiopia shared their performance data with us. They run 400-600 samples a day, running an average of 5 tests per sample.  They have a water purification system. They have an automated generator, with air conditioning in the lab, monitored by temperature check twice a day. The lab was accredited with ISO 15189 in 2017. 

The imprecision and bias data

The imprecision data is from Roche-provided controls. This is not ideal, and probably will generate optimistic results. The bias is calculated by comparison to the peer group in their EQA.

For the most part, the Total Allowable errors being used are from the CLIA regulations. Although for some analytes, like Phosphorous (CAP 10.7%) and Direct Bilirubin (Ricos 2014 44.5%), alternate sources for TEa are used. Note that the EQA analysis provides one bias, which is used for both control levels.

The specific control levels were not provided in this data set, so some TE:a goals were extrapolated (Potassium at 25%, Sodium at 3.5%). Other analytes that have a unit/percentage based goal, depending on the control level, were fixed at the percentage-based goal (Glucose, Urea, Total Bilirubin).

Sigma-metrics of Roche Cobas 6000

When we're using a comparison study to calculate bias, there's an extra step with the regression equation we take to calculate the bias.

Sigma-metric = ( TE:a - |Bias| ) / CV

For the first level of Albumin, this works out to

Sigma-metric =  (10.0 - 0.82 ) / 3.04

= 9.18 / 3.04

=  3.02

Here are the rest of the metrics:

 The good news is that 40% of the performance is at Six Sigma, or higher. And a majority of the performance on the instrument is over 5 Sigma.

The bad news is a third of the performance is 3 Sigma and lower. Over 30% of the performance is below 3 Sigma.

Only a few analytes have any issues with bias (Creatinine has the largest problem with bias, which is eating up 70% of the TE:a budget). The major challenge, or the reason why more analytes have low Sigma-metrics, is due to high imprecision. A few analytes are actually "off the chart" because of very high imprecision.

Since there are a lot of numbers in this table, it's easier to see the graph.

Visual display of Sigma-metric Performance of Roche Cobas 6000 in Ethiopia

There are a nice cluster of points in the "bull's eye", but a concerning number of analytes are missing the target.

QC Implications of Roche Cobas 6000 performance

So the good news: a good number of the analytes don't need the  "Westgard Rules." A simple 1:3s rule with 2 controls will be sufficient.

The bad news: for a significant number of analytes, the full "Westgard Rules" are not enough. More controls, more frequent running of controls, is warranted. Exactly the kind of additional QC burden you don't want to place on the shoulders of a developing laboratory.

And remember, this is the optimistic assessment. If independent, 3rd party controls were used to assess performance, we might see that the tests are performing worse.

It is not unusual to see challenging performance in the electrolytes for Roche cobas, nor is it uncommon to see challenging performance in albumin, creatinine and urea analytes even in the best laboratories. But the operating performance in this laboratory is lower that what we would typically see in a laboratory in Europe, the US, Australia, or select parts of SE Asia.

Conclusion

The laboratory quality divide is real. Simply installing advanced instruments in a laboratory doesn't make a laboratory high quality. The right infrastructure, training, support and staff education must be there, too. This laboratory in Ethiopia is exemplary in so many dimensions. It's difficult to see that despite their best efforts, they face significant challenges in keeping all their tests under control.